Method for storage consistency in deep learning model, computing subsystem, and computing platform

Abstract

The present application relates to the field of data processing, and relates in particular to a method for storage consistency in a deep learning model, a computing subsystem, and a computing platform. The method comprises: a first distributed node creating proposal information to be updated; using a dynamic parameter distribution model to acquire proposal distribution information matching the proposal information; sending the proposal information to second distributed nodes, such that the second distributed nodes determine whether or not to respond to the proposal information; receiving proposal response messages from the second distributed nodes; and if the number of the proposal response messages issued reaches a dynamic threshold, triggering all distributed nodes in a computing subsystem to update data pending update, so as to maintain data storage consistency across the computing subsystem with respect to said data. The method can solve data consistency issues experienced by deep learning models during distributed training, ensuring the effectiveness and accuracy of the training process and reducing system complexity and communication overhead during training.

Description

Deep learning model storage consistency method, computing subsystem and computing platform Technical Field

[0001] The present application relates to the field of data processing, and in particular to a deep learning model storage consistency method, computing subsystem, and computing platform. Background Art

[0002] At present, in order to improve the popularity of intelligent applications in various industries and fields, it is urgent to build an intelligent computing platform to assist the construction of intelligent supercomputing centers, provide a foundation for the construction of artificial intelligence platforms for scientific research, industry, and urban services, and further realize talent gathering, industrial upgrading, and development through intelligent computing platforms.

[0003] Deep learning, a branch of machine learning, uses multi-layered artificial neural networks to simulate the human learning process. Deep learning models typically require large amounts of data to train, making distributed training an effective solution. Distributed training distributes training tasks across multiple computing nodes for parallel execution, significantly improving training efficiency.

[0004] However, related technologies have proposed consistent storage technologies to address data inconsistency in distributed systems. This technology replicates data across multiple compute nodes and maintains a consistent state, ensuring that the data on each compute node is identical. However, existing consistent storage technologies can encounter performance bottlenecks when processing large amounts of data, impacting training efficiency. To address data consistency in distributed training, existing technologies primarily introduce coordinating nodes to manage data updates across all compute nodes. However, introducing a coordinating node increases system complexity and communication overhead, impacting system performance.

[0005] Therefore, it is urgent to propose a technical solution to solve the data consistency problem of deep learning models during distributed training, so as to ensure the effectiveness and accuracy of the training process and reduce the system complexity and communication overhead during the training process.

[0006] Summary of the Invention

[0007] This application provides a deep learning model storage consistency method, computing subsystem and computing platform to ensure data consistency of deep learning models during distributed training, ensure the effectiveness and accuracy of the training process, and reduce system complexity and communication overhead during training.

[0008] In a first aspect, the present application provides a deep learning model storage consistency method, which is applied to a computing subsystem of a deep learning model, wherein the computing subsystem includes at least multiple distributed nodes; the method includes:

[0009] The first distributed node creates proposal information to be updated; the proposal information includes at least: data to be updated in the deep learning model, the data to be updated including model parameters and / or training data;

[0010] Acquire proposal distribution information matching the proposal information using a dynamic parameter distribution model; wherein the proposal distribution information includes: a second distributed node for selecting the proposal information and a corresponding dynamic threshold, the dynamic threshold being the minimum number of nodes required for the computing subsystem to respond to the proposal information, the dynamic threshold being dynamically calculated by the dynamic parameter distribution model based on the proposal information and the computing subsystem;

[0011] sending the proposal information to the second distributed node, so that the second distributed node determines whether to respond to the proposal information;

[0012] receiving proposal response information from the second distributed node;

[0013] If the number of the proposal response messages sent reaches the dynamic threshold, all distributed nodes in the computing subsystem are triggered to update the data to be updated, so that the data to be updated maintains consistency of the stored data in the computing subsystem.

[0014] In a second aspect, an embodiment of the present application provides a computing subsystem, which is applied to a computing subsystem of a deep learning model, wherein the computing subsystem includes at least a plurality of distributed nodes; a first distributed node in the computing subsystem includes at least the following units:

[0015] an acquisition unit configured to create proposal information to be updated; the proposal information includes at least: data to be updated in the deep learning model, the data to be updated including model parameters and / or training data; and adopt a dynamic parameter distribution model to acquire proposal distribution information matching the proposal information; wherein the proposal distribution information includes: a second distributed node for electing the proposal information and a corresponding dynamic threshold, the dynamic threshold being the minimum number of nodes required for the computing subsystem to respond to the proposal information, the dynamic threshold being dynamically calculated by the dynamic parameter distribution model based on the proposal information and the computing subsystem;

[0016] a distribution unit configured to send the proposal information to the second distributed node, so that the second distributed node determines whether to respond to the proposal information;

[0017] The statistical unit is configured to receive the proposal response information of the second distributed node; if the number of the proposal response information issued reaches the dynamic threshold, all distributed nodes in the computing subsystem are triggered to update the data to be updated, so that the data to be updated maintains the consistency of the stored data in the computing subsystem.

[0018] In a third aspect, an embodiment of the present application provides a computing device, the computing device comprising:

[0019] at least one processor, memory, and input-output unit;

[0020] The memory is used to store a computer program, and the processor is used to call the computer program stored in the memory to execute the deep learning model storage consistency method of the first aspect.

[0021] In a fourth aspect, a computer-readable storage medium is provided, which includes instructions, which, when executed on a computer, enable the computer to execute the deep learning model storage consistency method of the first aspect.

[0022] The technical solution provided in the embodiments of the present application can be applied to a computing subsystem of a deep learning model, which includes at least multiple distributed nodes. First, a first distributed node creates proposal information to be updated. This proposal information includes at least the data to be updated in the deep learning model, including model parameters and / or training data. Second, a dynamic parameter distribution model is used to obtain proposal distribution information that matches the proposal information. The proposal distribution information includes a second distributed node for selecting the proposal information and a corresponding dynamic threshold. The dynamic threshold is the minimum number of nodes required for the computing subsystem to respond to the proposal information. The dynamic threshold is dynamically calculated by the dynamic parameter distribution model based on the proposal information and the computing subsystem. The proposal information is then sent to the second distributed node, allowing it to determine whether to respond to the proposal information. Next, proposal response information is received from the second distributed node. Finally, if the number of proposal response messages issued reaches the dynamic threshold, all distributed nodes in the computing subsystem are triggered to update the data to be updated, ensuring that the data to be updated maintains data consistency within the computing subsystem.

[0023] The technical solution of this application introduces a dynamic parameter distribution model and a dynamic threshold mechanism, making the system more adaptable and scalable, and providing an effective way to ensure the storage consistency of deep learning models in a distributed environment. This solution can solve the data consistency problem during the distributed training process of deep learning models, ensuring the effectiveness and accuracy of the training process and reducing system complexity and communication overhead during the training process. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0025] FIG1 is a schematic diagram of a flow chart of a method for storing consistency of a deep learning model according to an embodiment of the present application;

[0026] FIG2 is a schematic diagram showing the principle of a dynamic parameter distribution model according to an embodiment of the present application;

[0027] FIG3 is another schematic diagram of a dynamic parameter distribution model according to an embodiment of the present application;

[0028] FIG4 is a schematic diagram of the structure of a computing subsystem according to an embodiment of the present application;

[0029] FIG5 is a schematic structural diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0030] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0031] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application pertains. The terms used herein in the specification of this application are for the purpose of describing specific embodiments only and are not intended to limit this application.

[0032] At present, in order to improve the popularity of intelligent applications in various industries and fields, it is urgent to build an intelligent computing platform to assist the construction of intelligent supercomputing centers, provide a foundation for the construction of artificial intelligence platforms for scientific research, industry, and urban services, and further realize talent gathering, industrial upgrading, and development through intelligent computing platforms.

[0033] Deep learning, a branch of machine learning, uses multi-layered artificial neural networks to simulate the human learning process. Deep learning models typically require large amounts of data to train, making distributed training an effective solution. Distributed training distributes training tasks across multiple computing nodes for parallel execution, significantly improving training efficiency.

[0034] However, in related technologies, consistent storage technology has been proposed to address data inconsistency issues in distributed systems. This technology replicates data across multiple computing nodes and maintains a consistent state of the data, ensuring that the data on each computing node is identical. Existing consistent storage technology may encounter performance bottlenecks when processing large amounts of data, affecting training efficiency. In existing technologies, to address data consistency issues in distributed training, a coordination node is mainly introduced to manage data updates for all computing nodes. When a computing node modifies its local data, it sends a notification to the coordination node, which then sends instructions to other computing nodes to update their local data as well. This method ensures that the data on all computing nodes is identical, thereby ensuring the effectiveness and accuracy of the training process.

[0035] However, in existing technologies, the introduction of coordination nodes increases system complexity and communication overhead, impacting system performance. Secondly, existing consistent storage technologies can encounter performance bottlenecks when processing large amounts of data, as each data update requires replication and synchronization across all compute nodes, which consumes significant time and resources. Furthermore, existing technologies cannot guarantee data consistency in extreme situations (such as network outages or compute node failures), which can lead to training failures.

[0036] Therefore, without introducing coordination nodes, how to ensure the data consistency of deep learning models during distributed training, ensure the effectiveness and accuracy of the training process, and reduce the system complexity and communication overhead during training is a technical problem that needs to be solved urgently.

[0037] To solve at least one of the above technical problems, an embodiment of the present application provides a deep learning model storage consistency method, a computing subsystem, and a computing platform.

[0038] Specifically, a deep learning model storage consistency solution can be applied to a computing subsystem of a deep learning model, which includes at least multiple distributed nodes. First, a first distributed node creates proposal information to be updated. This proposal information includes at least the data to be updated in the deep learning model, including model parameters and / or training data. Second, a dynamic parameter distribution model is used to obtain proposal distribution information that matches the proposal information. The proposal distribution information includes a second distributed node for selecting the proposal information and a corresponding dynamic threshold. The dynamic threshold is the minimum number of nodes required for the computing subsystem to respond to the proposal information. The dynamic threshold is dynamically calculated by the dynamic parameter distribution model based on the proposal information and the computing subsystem. The proposal information is then sent to the second distributed node, allowing it to determine whether to respond to the proposal information. Next, proposal response information is received from the second distributed node. Finally, if the number of proposal response messages issued reaches the dynamic threshold, all distributed nodes in the computing subsystem are triggered to update the data to be updated, ensuring that the data to be updated maintains data consistency within the computing subsystem.

[0039] The deep learning model storage consistency solution first employs a dynamic parameter distribution model, which calculates dynamic thresholds in real time based on the current system state and the characteristics of the proposal information. This adaptability enables the system to flexibly adapt to varying workloads and network environments, thereby improving system performance and efficiency. Furthermore, the introduction of dynamic thresholds allows the system to automatically adjust the number of responding nodes as needed. This helps improve system scalability, enabling it to better handle growing computing demands without the need for manual parameter adjustments. Furthermore, by introducing dynamic thresholds during the proposal response phase, the system ensures a sufficient number of nodes respond when a consistency update is triggered. This helps maintain data consistency in a distributed deep learning model environment and prevents issues caused by insufficient node responses. Furthermore, the use of dynamic thresholds avoids unnecessary consistency updates, thereby reducing system communication overhead. Consistency updates are triggered only when a sufficient number of responses are received, reducing unnecessary data transmission and computational overhead. The dynamic thresholds can also be adjusted based on actual needs, providing system flexibility and configurability to accommodate diverse application scenarios and performance requirements.

[0040] Overall, this deep learning model storage consistency solution introduces a dynamic parameter distribution model and a dynamic threshold mechanism, making the system more adaptable and scalable, and providing an effective way to ensure the storage consistency of deep learning models in a distributed environment. This solution can solve the data consistency issue during the distributed training of deep learning models, ensuring the effectiveness and accuracy of the training process and reducing system complexity and communication overhead during the training process.

[0041] The deep learning model storage consistency solution provided in the embodiment of the present application can be executed by an electronic device, which can be a server, a server cluster, or a cloud server. The electronic device can also be a terminal device such as a mobile phone, a computer, a tablet computer, a wearable device, or a dedicated device (such as a dedicated terminal device with a computing subsystem). In an optional embodiment, a service program for executing the deep learning model storage consistency solution can be installed on the electronic device.

[0042] Figure 1 is a schematic diagram of a method for storing consistency of a deep learning model provided by an embodiment of the present application. The method is applied to a computing subsystem of a deep learning model, which computing subsystem includes at least multiple distributed nodes. As shown in Figure 1, the method includes the following steps:

[0043] 101. A first distributed node creates proposal information to be updated.

[0044] In the embodiments of the present application, the first distributed node refers to a node in a distributed system that plays a specific role in the system. In the deep learning model storage consistency solution, the first distributed node generally refers to the node responsible for initiating model update requests, generating proposal information, and sending the proposal information to other nodes. In other words, the first distributed node is the node in the system responsible for initializing and initiating consistency update operations.

[0045] In a distributed deep learning environment, the first distributed node can be a server, a master node, or any other node designated to manage and coordinate model updates. It collects information about model parameter changes, training data updates, and other related information, then consolidates this information into proposals for subsequent consistency updates. Overall, the first distributed node plays a crucial role in initiating model updates and maintaining consistency in a distributed deep learning system.

[0046] In an embodiment of the present application, the proposal information includes at least: data to be updated in the deep learning model. For example, the data to be updated includes model parameters and / or training data. For example, suppose a deep learning model is used for image classification tasks. During training, the model parameters need to be continuously updated to improve performance. The first distributed node collects a batch of new training data and decides to update the model parameters across the entire computing subsystem. The proposal information includes the model parameters to be updated and the training data. For example, the weights, biases, and image training data in the deep learning model. Specifically, the model parameters in the proposal information can be represented as a data structure containing weight matrices and bias vectors for each layer. These parameters are key components of the deep learning model's learning process. The training data in the proposal information can be represented as a dataset description containing image files, label information, or other input data. This dataset will be used to update the deep learning model to improve its performance on the relevant task. By combining these two types of data to be updated in the proposal information, the first distributed node can clearly specify the content that needs to be updated in the computing subsystem. This helps ensure that model update operations performed on different nodes in the distributed environment are consistent, thereby maintaining consistency across the entire system.

[0047] In this way, the proposal message contains key proposal information, describing the content that needs to be updated in the computing subsystem. In the above example, the model parameters and training data to be updated are key components of the deep learning model. This information will be used in subsequent steps to calculate the dynamic parameter distribution model and trigger the proposal response mechanism.

[0048] In 102, a dynamic parameter distribution model is used to obtain proposal distribution information that matches the proposal information.

[0049] The proposal distribution information includes: a second distributed node for electing proposal information and a corresponding dynamic threshold.

[0050] In this embodiment of the present application, a second distributed node refers to multiple nodes in a distributed system, relative to the first distributed node, that play different roles in a specific operation or task. In a deep learning model storage consistency solution, a second distributed node is a node selected to respond to a proposal message initiated by the first distributed node.

[0051] Specifically, the second distributed node is selected as the node that responds to the proposal information, which may be determined based on system load, network topology, node availability, and other factors. Its main task is to receive and process the proposal information from the first distributed node and participate in the update operation under the condition that the dynamic threshold is met to ensure the consistency and effectiveness of the system.

[0052] In a distributed deep learning environment, the second distributed nodes may be other servers, worker nodes, or any nodes with processing power and resources in the system. Their selection and participation are crucial to ensuring the consistency and accuracy of model parameters.

[0053] In this embodiment of the present application, the dynamic threshold is the minimum number of nodes required for the computing subsystem to respond to the proposal. The dynamic threshold is dynamically calculated by the dynamic parameter distribution model based on the proposal and the computing subsystem. It is understood that the dynamic threshold is a value calculated based on the proposal and the state of the computing subsystem according to the dynamic parameter distribution model to determine the minimum number of nodes required to respond to the proposal. The purpose of this threshold is to ensure that the system can reach a certain consensus during update operations and that sufficient nodes participate to maintain consistency.

[0054] For example, in a distributed deep learning environment, the first distributed node has created a proposal for an update, which includes the model parameters and training data to be updated. In this step, it is necessary to determine which distributed node will be responsible for responding to this proposal and set a dynamic threshold to ensure that a sufficient number of nodes respond to maintain system consistency.

[0055] In an optional example, the dynamic parameter distribution model will select the second distributed node based on the current system state and the characteristics of the proposed information. This selection process may involve factors such as the node load, network latency, and node availability. Assume that node B is elected as the second distributed node. It can be understood that the dynamic threshold is obtained based on the dynamic calculation of the proposal information and the computing subsystem. For example, the dynamic parameter distribution model may consider factors such as the number of nodes in the current system and the reliability of network communication to determine the size of the dynamic threshold. Assuming that the dynamic threshold is calculated to be 3 nodes, based on this, the proposed distribution information specifies node B as the second distributed node and sets the dynamic threshold to 3 nodes. This means that the subsequent consistency update operation will only be triggered when at least 3 nodes respond to this proposal information. This ensures that the update operation is sufficiently supported in the system, thereby maintaining data consistency.

[0056] By providing such proposal distribution information, the system can effectively select the response node while ensuring consistency and set a reasonable threshold to control the triggering of the update operation.

[0057] In another example, consider a distributed deep learning environment with five nodes, each with varying loads and network latency. A proposal message contains a critical model parameter update, requiring sufficient node participation to maintain consensus. Based on the system state, the dynamic parameter distribution model may detect that two nodes have high loads and one node has high network latency. These factors must be considered to determine the dynamic threshold. Analysis of the proposal message indicates that this update is critical to the overall model performance, requiring a higher level of consensus. The dynamic parameter distribution model calculates a dynamic threshold of four nodes, meaning that at least four nodes must respond to the proposal message to trigger the update. Based on analysis of the system state and the proposal message, the dynamic parameter distribution model uses a specific algorithm or rule to calculate the dynamic threshold. This calculation further improves system fault tolerance and ensures consistency even in the event of node failures.

[0058] Through such dynamic threshold calculation, the system can adjust the minimum number of nodes required according to actual conditions, thereby maintaining consistency under different environments and update requirements.

[0059] As an optional embodiment, it is assumed that the dynamic parameter distribution model includes at least: a current state monitoring layer, a decision engine, a dynamic threshold calculation layer, and a parameter adjustment layer. As an optional embodiment, the dynamic parameter distribution model may include the following components:

[0060] The current status monitoring layer is responsible for monitoring and collecting information about the system's current status, including but not limited to: The node count records the current number of nodes in the system so that the overall node status can be considered when calculating dynamic thresholds. Node load monitors the load of each node, such as CPU usage and memory usage. Network latency measures the communication delay between nodes to determine the efficiency of data transmission. Node availability monitors the health of each node to ensure that only healthy nodes are included in the calculation.

[0061] The decision engine, the core component of the dynamic parameter distribution model, is responsible for developing specific decision strategies based on system status and proposal information. Specifically, it formulates appropriate decision strategies based on the current system state, the importance of the proposal information, and its impact. It also adjusts the parameters required for dynamic threshold calculation, such as fault tolerance requirements and the minimum number of participating nodes, based on different scenarios.

[0062] The dynamic threshold calculation layer calculates the actual dynamic threshold based on the parameters and policies provided by the decision engine. Specifically, it uses specific algorithms or rules to calculate the dynamic threshold. For example, factors such as node load balancing, network topology, and fault tolerance requirements are taken into account. The threshold is dynamically adjusted based on the actual system operation to adapt to different environments and requirements.

[0063] The parameter adjustment layer is responsible for adjusting and configuring system parameters based on the results of dynamic threshold calculations. For example, it determines which nodes will be selected for update operations based on dynamic thresholds. It notifies the selected nodes and coordinates their operations to ensure smooth execution of the consistent update process. Specifically, it handles errors or anomalies that may arise during the update process to ensure system stability and reliability.

[0064] Through the components and functions at these levels, the dynamic parameter distribution model can flexibly calculate appropriate dynamic thresholds based on the system dynamics and the characteristics of the proposed information, and adjust system parameters accordingly, thereby ensuring the consistency, efficiency, and reliability of the distributed deep learning system.

[0065] In an optional example, based on the above structure, in 102, a dynamic parameter distribution model is adopted to obtain a second distributed node matching the proposed information and a corresponding dynamic threshold, as shown in FIG2 , which can be implemented as follows:

[0066] 201 : Acquire proposal information and a node connection structure associated with a first distributed node that issues the proposal information through a current state monitoring layer.

[0067] In this step, the system uses the current state monitoring layer to obtain the second distributed node that matches the proposed information and the node connection structure associated with the first distributed node. Specifically, the system obtains the node connection structure in the current distributed system to understand the relationships and communication topology between each node. The proposed information is received from the first distributed node, which contains key information such as model update requests and parameter changes. By analyzing the node connection structure, the system determines the second distributed node that matches the proposed information, i.e., the node that responds to the proposal of the first distributed node.

[0068] 202. The decision engine selects a proposal response decision strategy that matches the proposal information and the node connection structure from among the stored multiple proposal response decision strategies, and issues a dynamic calculation instruction to the dynamic threshold calculation layer based on the selected proposal response decision strategy.

[0069] In this step, the system uses a decision engine to select a strategy that matches the proposal information and node connection structure from among the multiple stored proposal response decision strategies. Specifically, the decision engine selects an appropriate proposal response decision strategy based on the system state and the characteristics of the proposal information. These strategies may be predefined in the system. The decision engine then issues a dynamic calculation instruction to the dynamic threshold calculation layer, containing the selected proposal response decision strategy and related parameters.

[0070] 203 , generating proposal distribution information in response to the dynamic computing instruction through the dynamic threshold computing layer based on the proposal response decision strategy, proposal information, and node connection structure in the dynamic computing instruction.

[0071] Dynamic calculation instructions are commands issued by the decision engine to instruct the dynamic threshold calculation layer to perform specific calculations to generate dynamic threshold values ​​appropriate for the current system state and proposed information. These instructions contain the proposed response strategy and related parameters selected by the decision engine, providing specific guidance for the dynamic threshold calculation layer to perform calculations.

[0072] The goal of the dynamic calculation instructions is to enable the dynamic threshold calculation layer to flexibly calculate appropriate dynamic threshold values ​​based on specific circumstances through efficient information transfer. This dynamism and flexibility enable the system to respond appropriately in different situations, better meeting the requirements of distributed deep learning systems.

[0073] In this step, the system, through the dynamic threshold calculation layer, responds to instructions issued by the decision engine and generates proposal distribution information based on the proposal response decision strategy, proposal information, and node connection structure. This may include the following operations: First, according to the specified proposal response decision strategy, the dynamic threshold calculation layer calculates a dynamic threshold value applicable to the current situation. Second, based on the calculated dynamic threshold value, combined with the proposal information and node connection structure, a proposal distribution information is generated, including the responding node and the dynamic threshold.

[0074] 204 , applying the proposed distribution information to the relevant distributed nodes in the computing subsystem through the parameter adjustment layer to start the distribution process of the proposed information.

[0075] In this step, the system applies the generated proposal distribution information to the relevant distributed nodes in the computing subsystem through the parameter adjustment layer, initiating the proposal information distribution process. The parameter adjustment layer selects the corresponding nodes based on the response nodes specified in the proposal distribution information and notifies them to participate in the model update. The selected nodes then begin the model update process based on the received proposal information, which may involve modifying model parameters and updating training data. The parameter adjustment layer may also be responsible for handling errors or anomalies that may arise during the distribution process to ensure system stability and reliability.

[0076] Through steps 201 to 204 , the dynamic parameter distribution model can flexibly select proposal response strategies, calculate dynamic thresholds, and effectively distribute update information to appropriate nodes based on the system status and characteristics of the proposal information, thereby maintaining system consistency and efficiency.

[0077] Optionally, the dynamic parameter distribution model also includes a historical data analysis layer. This layer is an optional component of the dynamic parameter distribution model and is used to analyze and utilize the system's past operational data to optimize the dynamic parameter distribution process. By introducing this layer, the dynamic parameter distribution model can more intelligently utilize past experience and data, improving the system's adaptability and performance, thereby better meeting the requirements of distributed deep learning systems.

[0078] Based on this, before adopting the dynamic parameter distribution model in 102 to obtain the second distributed node matching the proposal information and the corresponding dynamic threshold, the proposal response decision strategy set in the decision engine can also be updated based on the historical proposal information and / or historical response data through the historical data analysis layer.

[0079] The historical data analysis parameters include but are not limited to the following parameters:

[0080] The Historical Data Weight determines the weighting of new and old data in historical data analysis. A higher weight may cause the system to place more emphasis on recent data, while a lower weight gives a more balanced consideration to historical data.

[0081] Prediction Accuracy Weight: While considering historical data, the system may use the accuracy of historical predictions to adjust decision-making policy parameters. This weight is used to balance the influence of historical data and prediction accuracy.

[0082] The proposed response decision strategy at least includes: decision strategy parameters. Decision strategy parameters include but are not limited to the following parameters:

[0083] The decision threshold is used in dynamic threshold calculations to determine whether to accept or reject a proposal. When the proposal's score or relevance exceeds this threshold, the system may choose to accept the proposal and perform the corresponding action.

[0084] Fault Tolerance determines the system's tolerance for potential errors or abnormal situations. A higher Fault Tolerance may result in a more tolerant response, thus avoiding overly sensitive rejection of proposals.

[0085] The learning rate is used to control how quickly the system adapts to new information. A higher learning rate may cause the system to adjust thresholds more quickly, while a lower learning rate may make it more conservative in treating new information.

[0086] Update Frequency: This determines when the decision engine updates its proposed response strategies. A higher update frequency may make the system more flexible, but it may also lead to excessive decision engine updates.

[0087] In addition, other parameters can be introduced into historical data analysis, such as system resource parameters. Specifically, these parameters include, but are not limited to, information such as available system resources and node load. These parameters are used to consider the overall system status in decision-making strategies. These parameters may include available memory, network bandwidth, etc.

[0088] Together, these parameters comprise the proposal response decision strategy. By dynamically adjusting these parameters, the system can more flexibly adapt to varying workloads, environmental changes, and the characteristics of proposal information. By using these parameters in the historical data analysis layer, the system can more intelligently leverage past experience, improving the performance and adaptability of the dynamic parameter distribution model.

[0089] As an optional embodiment, in step 202, selecting a proposal response decision strategy that matches the proposal information and the node connection structure, as shown in FIG3 , can be implemented as follows:

[0090] 301, performing feature analysis on the proposal information to obtain corresponding proposal type, proposal demand level, and proposal priority;

[0091] 302, performing feature analysis on the node connection structure to obtain a communication relationship type, a topology type, a network bandwidth, and a network delay corresponding to the first distributed node;

[0092] 303, obtaining a target feature combination corresponding to the proposal type, proposal demand level, proposal priority, communication relationship type, topology type, network bandwidth, and network delay obtained through feature analysis;

[0093] 304 : Determine the proposed response decision strategy corresponding to the target feature combination as the proposed response decision strategy matched by the first distributed node.

[0094] For example, at 301, the system first performs a feature analysis on the received proposal information. This may include the type of proposal (e.g., model update request, resource request), the degree of need (e.g., urgency), and the priority of the proposal. By analyzing these features of the proposal information, the system can better understand the nature and importance of the proposal. Next, at 302, the system performs a feature analysis on the node connection structure associated with the proposal. This includes the communication relationship type (e.g., point-to-point, broadcast), topology type (e.g., star, mesh), network bandwidth, and network latency of the first distributed node. These features help the system understand the communication status and network conditions between nodes, allowing it to better decide how to respond to the proposal. Furthermore, at 303, based on the feature analysis of the proposal information and the node connection structure, the system determines a target feature combination: a combination of features related to the proposal type, need, priority, and node connection structure. This combination reflects the system's specific requirements and conditions for the proposal. Finally, at 304, the system determines an appropriate proposal response decision strategy based on the target feature combination. This strategy may be predefined or dynamically generated based on the results of the historical data analysis layer. It includes specific measures such as how to set dynamic thresholds and adjust parameters to maximize proposal satisfaction and ensure system stability and efficiency. This example analyzes the characteristics of proposal information and node connectivity to determine a specific response strategy for each proposal. This enables the system to more intelligently handle proposals of different types and priorities, and make real-time adjustments based on system status and network conditions.

[0095] In 103, the proposal information is sent to the second distributed node, so that the second distributed node determines whether to respond to the proposal information.

[0096] In step 103, the purpose of sending the proposal information to the second distributed node is to allow the node to determine whether to respond to the proposal information. In this case, a dynamic parameter distribution model is usually used to determine whether to respond to the proposal information.

[0097] For example, consider a distributed system consisting of multiple nodes, each with its own resources and capabilities. Node A receives a proposal message containing a task that requires other nodes to execute. Based on its own policy, node A decides to send this proposal message to connected nodes B, C, and D. Node B is a high-performance node with sufficient resources to execute the proposed task. After receiving the proposal message, node B calculates a dynamic parameter distribution model and determines whether to respond based on the characteristics of the proposal and the current system state. If node B determines it has sufficient resources and the importance of the proposal exceeds its set decision threshold, it will respond to the proposal and execute the task. Node C is a resource-limited node with low network bandwidth requirements. After receiving the proposal message, node C also performs a similar dynamic parameter distribution model calculation, taking into account its own resources and the characteristics of the proposal. If node C determines it can handle the task and has low network latency, it may respond to the proposal and execute the task. Node D is a highly loaded node and may have already received or is currently executing other tasks. After receiving the proposal message, node D also calculates a dynamic parameter distribution model. If node D believes that it is too loaded or the importance of the proposal does not meet its set decision threshold, it may choose not to respond to the proposal.

[0098] This example demonstrates how to leverage a dynamic parameter distribution model to make decisions at a second distributed node regarding whether to respond to a proposal. By considering the node's own resources, the characteristics of the proposal, and the current system state, each node can intelligently decide whether to take on a task, enabling efficient task distribution and execution.

[0099] In a distributed system, after receiving a proposal, the second distributed node needs to determine whether to respond to the proposal based on the system's policy. This decision may be based on a number of factors, including the content of the proposal, the current node's state, and the overall system load.

[0100] For example, consider a distributed system for resource allocation and task scheduling. A second distributed node receives a proposal requesting additional computing resources to perform an important task. In this scenario, the second distributed node needs to consider the following factors:

[0101] First, the current node load: Second, the distributed node will first check its own load. If the current node is already under high load, allocating more resources may affect the overall performance of the system.

[0102] Second, the importance and urgency of the proposal: The second distributed node will evaluate the importance and urgency of the proposal. If the proposal involves performing a critical task or handling an emergency, the second distributed node may be more inclined to respond to the proposal and allocate the required resources.

[0103] Third, the global state of the system: Nodes also need to consider the overall state of the system. If other nodes are also processing similar requests and system resources are limited, the second distributed node may need to weigh the priorities of different requests.

[0104] Fourth, policies and rules: Distributed systems may have predefined policies and rules that determine how to handle different types of proposals. These rules may be based on historical data, predictive models, or configurations by system administrators.

[0105] Based on these considerations, the second distributed node will comprehensively evaluate the proposal information and make a decision based on system policy. For example, if the current node load is low, the proposal involves performing an urgent task, and overall system resources are sufficient, the second distributed node may respond to the proposal and allocate the required resources. However, if the node is already overloaded or other factors affect the feasibility of resource allocation, the node may choose to reject the proposal or postpone its response until an appropriate time.

[0106] In 104, a proposal response message from a second distributed node is received.

[0107] In step 105 , if the number of proposal response messages sent reaches a dynamic threshold, all distributed nodes in the computing subsystem are triggered to update the data to be updated, so that the data to be updated maintains consistency of the stored data in the computing subsystem.

[0108] In a distributed system, suppose a second distributed node receives a proposal response message. This response message may be an acceptance or rejection of a previous proposal, or a confirmation or adjustment of resource allocation. When the second distributed node receives such a proposal response message, it needs to process it accordingly and may take some actions.

[0109] For example, suppose a distributed system is used to coordinate task execution across multiple nodes. The second distributed node sends a proposal to other nodes, requesting them to allocate more computing resources to accelerate a specific task. After receiving this proposal, the other nodes may send a proposal response message to the second distributed node. If the second distributed node receives a sufficient number of positive responses, that is, exceeds the dynamic threshold defined by the system, then it may trigger all distributed nodes in the computing subsystem to update the data to be updated to maintain data consistency. The purpose of this action is to ensure that all nodes in the system have the latest data status when executing tasks to avoid problems caused by data inconsistency.

[0110] For example, if multiple nodes in a distributed system are collaborating to complete a large data processing task, one node may propose to other nodes to allocate more resources. If the other nodes accept the proposal and agree, once the second distributed node receives a sufficient number of positive responses, it triggers all nodes to update the pending data. This means that each node will update its own data state, ensuring that all nodes operate based on the latest data during task execution, thereby maintaining data consistency and accurate task execution.

[0111] In another example, in a distributed computing system, when the number of received proposal responses reaches a dynamic threshold, all distributed nodes in the computing subsystem are triggered to update the data to be updated, ensuring data consistency within the computing subsystem. Consider a distributed machine learning system used for model training. In this system, each distributed node processes a portion of the data and decides whether to update its local data based on received proposal responses. Suppose the system has dozens of distributed nodes, each processing different batches of data for model training. When a node completes a round of training and is ready to update its local data, it sends a proposal response to other nodes, requesting them to also update their local data to ensure model consistency across the entire system. A dynamic threshold is set through the above steps. When the number of received proposal responses reaches this threshold, a data consistency update is triggered. Suppose the threshold is set to half the number of nodes plus one. When a node sends a proposal response, the other nodes receive it and confirm the data update. When the threshold is reached, for example, more than half the nodes in the system plus one have confirmed the data update, the system triggers a consistency update. At this point, all distributed nodes will perform data updates locally to ensure that each node's model parameters are consistent with those of other nodes. This synchronizes the model state parameters of the entire system, ensuring the accuracy and consistency of the calculation results.

[0112] Further optionally, after triggering all distributed nodes in the computing subsystem to update the data to be updated in 105, the operating status data of multiple distributed nodes can also be monitored, and the operating status data at least includes: node operating status, resource utilization, and network connectivity information. Furthermore, through the self-healing decision model, the operating status data is identified for abnormalities to obtain abnormal warning information indicating the node to be repaired. The abnormal warning information also includes: the abnormality type corresponding to the node to be repaired, and the scope of abnormal impact. Finally, based on the abnormality type and the scope of abnormal impact of the node to be repaired, the corresponding node repair operation is triggered, and real-time monitoring of the repair progress of the node to be repaired is started. Among them, the node repair operation includes at least: node abnormality handling, grayscale release, and rollback operation.

[0113] This description involves a system self-healing mechanism that primarily monitors, identifies, and repairs anomalies in distributed nodes. The following describes this mechanism using specific examples. First, the operational data of distributed nodes must be monitored. This step helps understand the node's current status. Furthermore, within this self-healing mechanism, a self-healing decision model can be used to analyze this monitored operational data and identify any anomalies. If an anomaly is detected, an anomaly warning is generated, including the type of anomaly node and the scope of impact. Based on this warning, the system triggers appropriate node repair operations. This may involve handling node anomalies, performing a phased release, and performing a rollback. Once the node repair operation is initiated, the system must monitor the repair progress in real time to ensure smooth progress.

[0114] In an example scenario, assume there is a distributed storage system consisting of multiple nodes that stores files uploaded by users. The system monitors the operation of the nodes at regular intervals.

[0115] During the monitoring phase, the system continuously monitors the operating status of nodes. For example, a node's disk utilization might suddenly spike, or a node's network connectivity might be abnormal. When the system detects abnormally high disk utilization on a node, the self-healing decision model identifies this anomaly and generates an abnormality warning message, indicating that this is an abnormal disk utilization situation affecting the files stored on that node. The system then triggers a node repair operation based on the abnormality warning message. Possible actions include stopping the node's file upload service, clearing disk space, and reallocating file storage locations. Once the repair operation is initiated, the system monitors its progress in real time. For example, it monitors the progress of disk cleanup to ensure that disk space has been freed up.

[0116] Through this self-healing mechanism, the system can automatically identify node anomalies and attempt to repair them, ensuring system stability and availability.

[0117] In practical applications, monitoring the operational data of distributed nodes involves multiple aspects of information, including but not limited to the node's operating status, resource utilization, and network connectivity. Some monitoring methods and data collection approaches include heartbeat monitoring. This involves periodically sending heartbeat packets to monitor the node's operating status, ensuring proper operation and enabling timely responses to abnormalities. Once a node stops sending heartbeats, it can be determined that the node has failed or is unavailable. For example, collecting information such as CPU usage, memory usage, and disk space utilization can help identify resource shortages or abnormal usage. Regular network connectivity testing can ensure smooth communication between nodes and promptly detect network failures or disconnections. For example, collecting node log information and event records and analyzing them can help identify abnormal node behavior or errors, such as abnormal program exits or service unavailability. For example, collecting node performance metrics such as response time and throughput can help analyze node performance and identify potential performance issues. For example, using a distributed tracing system, monitoring call chains and request flows between nodes ensures that all parts of the distributed system are functioning properly.

[0118] Furthermore, the collection of this monitoring data is typically accomplished through various monitoring systems and tools, such as Prometheus, Zabbix, and Nagios, as well as log analysis systems and APM (Application Performance Management) tools. These tools enable comprehensive monitoring and data collection of distributed node operational data, ensuring system stability and availability.

[0119] In this application, a dynamic parameter distribution model and a dynamic threshold mechanism are introduced to make the system more adaptable and scalable, and provide an effective way to ensure the storage consistency of deep learning models in a distributed environment. This solution can solve the data consistency problem during the distributed training of deep learning models, ensuring the effectiveness and accuracy of the training process and reducing system complexity and communication overhead during the training process.

[0120] In the above and following embodiments, there are actually multiple proposal response decision strategies. A proposal response decision strategy refers to how a node in a distributed system decides whether to respond to a proposal after receiving it. This strategy is based on system requirements and design considerations to ensure that the system can efficiently and accurately respond to proposals. By developing appropriate proposal response decision strategies, a distributed system can better process proposals, ensuring system stability, performance, and reliability.

[0121] In practical applications, the multiple proposal response decision strategies may further optionally include at least a node load balancing strategy. Specifically, a node load balancing strategy refers to a strategy used to balance the load across nodes in a distributed system. This strategy aims to ensure that each node is load-balanced when processing tasks, preventing overloading of certain nodes, which could lead to system performance degradation or uneven node resource utilization. For example, there are various node load balancing strategies. These strategies can be based on load balancing algorithms, using different algorithms to assign tasks to different nodes to achieve load balancing. Common load balancing algorithms include round-robin, least connections, weighted round-robin, and random, selecting the most suitable node based on node load when assigning tasks. Alternatively, a strategy can be based on node health, dynamically adjusting task allocation based on node health (such as CPU usage, memory usage, network bandwidth, etc.) to avoid assigning tasks to overloaded or unstable nodes. Alternatively, a strategy can be based on task type. Different types of tasks may have different requirements for node resources. Based on the nature and needs of the task, tasks can be assigned to the node most suitable for handling that type of task to achieve load balancing. Alternatively, a data distribution-based strategy could be used. If a task involves processing a large amount of data, tasks can be assigned to nodes with higher data locality based on the data distribution, reducing data transmission overhead and achieving load balancing. Alternatively, a historical load-based strategy could be used. This would use the node's historical load and processing capacity to predict future load conditions and adjust the task allocation strategy to achieve load balancing.

[0122] In this way, by rationally selecting and implementing node load balancing strategies, the overall system performance, reliability, and efficiency can be improved, ensuring that each node can effectively process tasks and avoiding problems caused by load imbalance.

[0123] The node load balancing strategy adjusts the calculation method of the dynamic threshold according to the node load situation, so that the probability of being selected as the second distributed node increases as the node load decreases.

[0124] The process of generating the dynamic threshold in the proposed distribution information in the dynamic threshold calculation layer is expressed as the following formula, namely:

[0125] Where D1 represents the dynamic threshold, and γ is the adjustment factor. The adjustment factor here is used to balance the impact between the relative connection load of the node and the adjustment item based on the historical dynamic threshold D1′. represents the relative connection load of the node, where c i represents the computing resources consumed by the related nodes connected to the i-th node, w i represents the weight coefficient corresponding to the i-th node, R represents the resource utilization of the related nodes connected to the i-th node, δ represents the deviation value adjustment factor, B represents the task execution delay term, β and (1-β) are balance adjustment factors, α is the adjustment coefficient of resource utilization R, and (1-α) is the adjustment coefficient of the historical dynamic threshold D1′.

[0126] In this formula, can be viewed as the relative connection load of a node, This means that nodes with fewer connected nodes and lower loads have a higher probability of being selected. Thus, the above formula can be used to dynamically adjust the threshold, making nodes with fewer connections and lower loads more likely to be selected as the second distributed node. It is important to note that the above adjustment factor can be determined based on a trade-off between node load balancing, proposal information, and node connection structure. Specifically, it can be adjusted based on actual needs.

[0127] Further optionally, the multiple proposal response decision strategies include at least a network communication delay strategy. Specifically, a network communication delay strategy refers to a strategy used to address network communication delay in a distributed system. Communication delay refers to the time it takes for a sender to send data and for a receiver to receive it. This is a critical consideration in distributed systems, as network communication speed can be affected by various factors, such as network congestion, bandwidth limitations, and transmission distance. For example, a prefetching strategy caches data on nodes that pre-need the data or pre-sends the data to nodes that are likely to use it. This reduces wait time when requesting data and improves data access efficiency. For example, a proximity-based strategy attempts to keep two communicating nodes as close together as possible, for example by selecting shorter paths in the network topology or by deploying nodes that frequently communicate with each other nearby. This reduces the distance data travels and reduces communication delay. For example, a multi-path strategy utilizes multiple communication paths simultaneously to transmit data, increasing bandwidth, reducing congestion, and thus reducing communication delay. For example, in a hierarchical strategy, the system divides the network into multiple layers, each with different communication methods and delays. This allows the appropriate communication layer to be selected based on actual conditions to minimize communication delays. For example, a load-aware strategy considers node loads and selects nodes with lower loads for communication, avoiding situations where excessive loads increase communication delays. For example, an adaptive strategy dynamically adjusts communication methods and paths based on real-time network conditions and system load to optimize communication efficiency and delays.

[0128] These network communication delay strategies can be applied individually or in combination, depending on specific system requirements and network environment, to improve system communication efficiency and performance. In distributed systems, effective network communication delay strategies can reduce system response time, improve user experience, and ensure system stability and reliability.

[0129] In this embodiment of the present application, the network communication delay strategy determines the threshold calculation method based on the communication delay between nodes, so that the probability of being selected as the second distributed node increases as the node communication delay decreases. In this case, the calculation of the dynamic threshold can take into account the communication delay between nodes.

[0130] The process of generating the dynamic threshold in the proposed distribution information in the dynamic threshold calculation layer is expressed as the following formula, namely:

[0131] In the above formula, D2 represents the dynamic threshold, γ is the adjustment factor, which is used to balance the impact between the node's network communication delay term and the adjustment term based on the historical dynamic threshold D2′, τ i represents the network communication delay term of the i-th node, R represents the resource utilization of the related nodes connected to the i-th node, δ represents the deviation value adjustment factor, B represents the task execution delay term, β and (1-β) are balance adjustment factors, α is the adjustment coefficient of resource utilization R, and (1-α) is the adjustment coefficient of the historical dynamic threshold D2′.

[0132] In this formula, τ i can be regarded as the network communication delay term of the i-th node, This means that nodes with lower network communication delays have a higher probability of being selected. Thus, the above formula can be adjusted dynamically through the threshold to make nodes with lower network communication delays more likely to be selected as the second distributed node. For example, the node with the lowest communication delay can be selected as the second distributed node. Similarly, the above adjustment factor can be determined based on a trade-off between node load balancing, proposal information, and node connection structure. Specifically, it can be adjusted based on actual needs.

[0133] Optionally, the multiple proposal response decision strategies include at least a node reliability strategy. A node reliability strategy refers to a series of policies and mechanisms used to ensure the reliability of each node in a distributed system. Node reliability is crucial in a distributed system because if a node fails, the stability and availability of the entire system are impacted.

[0134] The node reliability strategy dynamically adjusts the threshold according to the reliability of the node, so that the probability of being selected as the second distributed node increases with the improvement of the node reliability;

[0135] The process of generating the dynamic threshold in the proposed distribution information in the dynamic threshold calculation layer is expressed as the following formula, namely:

[0136] In the above formula, D3 represents the dynamic threshold, γ is the adjustment factor, which is used to balance the impact between the node's network communication delay term and the adjustment term based on the historical dynamic threshold D3′, ρ i represents the reliability evaluation index value of the i-th node, R represents the resource utilization of the related nodes connected to the i-th node, δ represents the deviation value adjustment factor, B represents the task execution delay term, β and (1-β) are balance adjustment factors, α is the adjustment coefficient of resource utilization R, and (1-α) is the adjustment coefficient of the historical dynamic threshold D3′.

[0137] In this formula, ρ i That is, the reliability evaluation index value of the i-th node, This means that nodes with higher reliability evaluation index values ​​are more likely to be selected. Thus, the above formula can be used to adjust the dynamic threshold to make nodes with higher reliability evaluation index values ​​more likely to be selected as the second distributed node. For example, the node with the highest reliability evaluation index value can be selected as the second distributed node. Similarly, the above adjustment factor can be determined based on a trade-off between node load balancing, proposal information, and node connection structure. Specifically, it can also be adjusted based on actual needs.

[0138] In addition, various proposal response decision strategies also include historical response strategies, which dynamically adjust thresholds based on historical response patterns, such that the probability of being selected as the second distributed node decreases as the frequency of a node responding to proposal information increases. This will not be discussed in detail here.

[0139] It is worth noting that the various proposed response decision strategies include at least: load balancing strategy, network communication delay strategy, node reliability strategy, historical response situation strategy, etc. These strategies can be combined and adjusted according to system requirements and environmental characteristics to achieve dynamic threshold calculation.

[0140] In another embodiment of the present application, a computing subsystem is provided, as shown in FIG4 . The computing subsystem is applied to a computing subsystem of a deep learning model, wherein the computing subsystem includes at least a plurality of distributed nodes. A first distributed node in the computing subsystem includes at least the following units:

[0141] an acquisition unit configured to create proposal information to be updated; the proposal information includes at least: data to be updated in the deep learning model, the data to be updated including model parameters and / or training data; and adopt a dynamic parameter distribution model to acquire proposal distribution information matching the proposal information; wherein the proposal distribution information includes: a second distributed node for electing the proposal information and a corresponding dynamic threshold, the dynamic threshold being the minimum number of nodes required for the computing subsystem to respond to the proposal information, the dynamic threshold being dynamically calculated by the dynamic parameter distribution model based on the proposal information and the computing subsystem;

[0142] a distribution unit configured to send the proposal information to the second distributed node, so that the second distributed node determines whether to respond to the proposal information;

[0143] The statistical unit is configured to receive the proposal response information of the second distributed node; if the number of the proposal response information issued reaches the dynamic threshold, all distributed nodes in the computing subsystem are triggered to update the data to be updated, so that the data to be updated maintains the consistency of the stored data in the computing subsystem.

[0144] Further optionally, the dynamic parameter distribution model includes at least: a current state monitoring layer, a decision engine, a dynamic threshold calculation layer, and a parameter adjustment layer;

[0145] The acquisition unit adopts a dynamic parameter distribution model to acquire a second distributed node matching the proposal information and a corresponding dynamic threshold, and is specifically configured to:

[0146] Acquire, through the current state monitoring layer, the proposal information and a node connection structure associated with the first distributed node that issues the proposal information;

[0147] Selecting, by a decision engine, a proposal response decision strategy that matches the proposal information and the node connection structure from a plurality of stored proposal response decision strategies, and issuing a dynamic calculation instruction to the dynamic threshold calculation layer based on the selected proposal response decision strategy;

[0148] generating, by a dynamic threshold calculation layer, the proposal distribution information in response to the dynamic calculation instruction and based on the proposal response decision strategy in the dynamic calculation instruction, the proposal information, and the node connection structure;

[0149] The proposed distribution information is applied to the relevant distributed nodes in the computing subsystem through the parameter adjustment layer to start the distribution process of the proposed information.

[0150] Further optionally, the dynamic parameter distribution model further includes: a historical data analysis layer;

[0151] The acquisition unit is further configured to: before adopting the dynamic parameter distribution model to obtain the second distributed node matching the proposal information and the corresponding dynamic threshold, update the proposal response decision strategy set in the decision engine based on the historical proposal information and / or historical response data through the historical data analysis layer; the proposal response decision strategy at least includes: decision strategy parameters.

[0152] Further optionally, the acquisition unit selects a proposal response decision strategy that matches the proposal information and the node connection structure, and is specifically configured to:

[0153] Performing feature analysis on the proposal information to obtain a corresponding proposal type, proposal demand level, and proposal priority; Performing feature analysis on the node connection structure to obtain a communication relationship type, topology type, network bandwidth, and network latency corresponding to the first distributed node;

[0154] Obtain target feature combinations corresponding to proposal type, proposal demand level, proposal priority, communication relationship type, topology type, network bandwidth, and network delay obtained through feature analysis;

[0155] The proposed response decision strategy corresponding to the target feature combination is determined as the proposed response decision strategy matched by the first distributed node.

[0156] Further optionally, the multiple proposal response decision strategies include at least: a node load balancing strategy;

[0157] The node load balancing strategy adjusts the calculation method of the dynamic threshold according to the node load, so that the probability of being selected as the second distributed node increases as the node load decreases;

[0158] The process of generating the proposed distribution information in the dynamic threshold calculation layer is expressed as the following formula:

[0159] Where D1 represents the dynamic threshold and γ is the adjustment factor. The adjustment factor in the above formula is used to balance the impact between the relative connection load of the node and the adjustment term based on the historical dynamic threshold D1′. represents the relative connection load of the node, where c i represents the computing resources consumed by the related nodes connected to the i-th node, w i represents the weight coefficient corresponding to the i-th node, R represents the resource utilization of the related nodes connected to the i-th node, δ represents the deviation value adjustment factor, B represents the task execution delay term, β and (1-β) are balance adjustment factors, α is the adjustment coefficient of resource utilization R, and (1-α) is the adjustment coefficient of the historical dynamic threshold D1′.

[0160] Further optionally, the plurality of proposal response decision strategies include at least: a network communication delay strategy;

[0161] The network communication delay strategy determines the threshold calculation method according to the communication delay between nodes, so that the probability of being selected as the second distributed node increases as the node communication delay decreases;

[0162] The process of generating the proposed distribution information in the dynamic threshold calculation layer is expressed as the following formula:

[0163] Where D2 represents the dynamic threshold, γ is the adjustment factor, and the adjustment factor in the above formula is used to balance the impact between the node's network communication delay term and the adjustment term based on the historical dynamic threshold D2′. i represents the network communication delay term of the i-th node, R represents the resource utilization of the related nodes connected to the i-th node, δ represents the deviation value adjustment factor, B represents the task execution delay term, β and (1-β) are balance adjustment factors, α is the adjustment coefficient of resource utilization R, and (1-α) is the adjustment coefficient of the historical dynamic threshold D2′.

[0164] Further optionally, the multiple proposal response decision strategies include at least: a node reliability strategy;

[0165] The node reliability strategy dynamically adjusts the threshold according to the reliability of the node, so that the probability of being selected as the second distributed node increases with the improvement of the node reliability;

[0166] The process of generating the proposed distribution information in the dynamic threshold calculation layer is expressed as the following formula:

[0167] Where D3 represents the dynamic threshold, γ is the adjustment factor, and the adjustment factor in the above formula is used to balance the impact between the node's network communication delay term and the adjustment term based on the historical dynamic threshold D3′. i represents the reliability evaluation index value of the i-th node, R represents the resource utilization of the related nodes connected to the i-th node, δ represents the deviation value adjustment factor, B represents the task execution delay term, β and (1-β) are balance adjustment factors, α is the adjustment coefficient of resource utilization R, and (1-α) is the adjustment coefficient of the historical dynamic threshold D3′.

[0168] Further optionally, after triggering all distributed nodes in the computing subsystem to update the data to be updated, the method further includes:

[0169] Monitoring the operational data of multiple distributed nodes, wherein the operational data includes at least: node operational status, resource utilization, and network connectivity information;

[0170] Using a self-healing decision model, the operation status data is subjected to abnormality identification to obtain abnormality warning information indicating a node to be repaired; the abnormality warning information also includes: the abnormality type corresponding to the node to be repaired, and the scope of the abnormality impact;

[0171] Based on the abnormality type and abnormal impact range of the node to be repaired, trigger the corresponding node repair operation and start real-time monitoring of the repair progress of the node to be repaired;

[0172] The node repair operation includes at least: node exception handling, grayscale release, and rollback operation.

[0173] In this application, a dynamic parameter distribution model and a dynamic threshold mechanism are introduced to make the system more adaptable and scalable, and provide an effective way to ensure the storage consistency of deep learning models in a distributed environment. This solution can solve the data consistency problem during the distributed training of deep learning models, ensuring the effectiveness and accuracy of the training process and reducing system complexity and communication overhead during the training process.

[0174] In another embodiment of the present application, an intelligent computing platform is provided, comprising: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0175] Memory for storing computer programs;

[0176] The processor is used to implement the deep learning model storage consistency method described in the method embodiment when executing the program stored in the memory.

[0177] The communication bus 1140 mentioned in the electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus 1140 can be divided into an address bus, a data bus, a control bus, etc.

[0178] For example, let's assume we need to build a large-scale, autonomous, and controllable intelligent computing platform based on specialized neural network chips. This platform will provide the hardware foundation for the research, development, and construction of intelligent computing platforms. This intelligent computing platform will also provide the hardware foundation for the construction of an intelligent supercomputing center. This center will serve as an artificial intelligence platform for scientific research, industry, and urban development, thereby attracting talent and developing industries.

[0179] Specifically, the intelligent computing platform primarily consists of five components: an intelligent hardware platform, an intelligent computing cloud operating system, application environment development, a big data platform, and an intelligent application PaaS platform. Based on intelligent computing theory, the intelligent hardware platform integrates deep learning chips, AI smart accelerator cards, and distributed servers to provide the foundational hardware support for the entire supercomputing platform and related derivative platforms. Its primary components include the intelligent computing subsystem, the network switching subsystem, the data storage subsystem, and the support and management subsystem.

[0180] An embodiment of the present application provides a deep learning model storage consistency method for constructing a low-energy computing unit.

[0181] For ease of representation, FIG5 shows only one thick line, but this does not mean that there is only one bus or one type of bus.

[0182] The communication interface 1120 is used for communication between the electronic device and other devices.

[0183] The memory 1130 may include a random access memory (RAM) or a non-volatile memory (non-volatile memory), such as at least one disk storage. Alternatively, the memory may be at least one storage device located away from the processor.

[0184] The above-mentioned processor 1110 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0185] Accordingly, an embodiment of the present application further provides a computer-readable storage medium storing a computer program, which, when executed, can implement the steps that can be performed by the electronic device in the above method embodiment.

Claims

1. A method for storing consistent deep learning models, characterized in that: In a computing subsystem applied to a deep learning model, the computing subsystem includes at least a plurality of distributed nodes; the method includes: The first distributed node creates proposal information to be updated; the proposal information includes at least: data to be updated in the deep learning model, the data to be updated including model parameters and / or training data; Acquire proposal distribution information matching the proposal information using a dynamic parameter distribution model; wherein the proposal distribution information includes: a second distributed node for selecting the proposal information and a corresponding dynamic threshold, the dynamic threshold being the minimum number of nodes required for the computing subsystem to respond to the proposal information, the dynamic threshold being dynamically calculated by the dynamic parameter distribution model based on the proposal information and the computing subsystem; sending the proposal information to the second distributed node, so that the second distributed node determines whether to respond to the proposal information; receiving proposal response information from the second distributed node; If the number of the proposal response messages sent reaches the dynamic threshold, all distributed nodes in the computing subsystem are triggered to update the data to be updated, so that the data to be updated maintains consistency of the stored data in the computing subsystem.

2. The method for storing a consistent deep learning model according to claim 1, wherein: The dynamic parameter distribution model includes at least: a current state monitoring layer, a decision engine, a dynamic threshold calculation layer, and a parameter adjustment layer; The adopting a dynamic parameter distribution model to obtain a second distributed node matching the proposal information and a corresponding dynamic threshold includes: Acquire, through the current state monitoring layer, the proposal information and a node connection structure associated with the first distributed node that issues the proposal information; Selecting, by a decision engine, a proposal response decision strategy that matches the proposal information and the node connection structure from a plurality of stored proposal response decision strategies, and issuing a dynamic calculation instruction to the dynamic threshold calculation layer based on the selected proposal response decision strategy; generating, by a dynamic threshold calculation layer, the proposal distribution information in response to the dynamic calculation instruction and based on the proposal response decision strategy in the dynamic calculation instruction, the proposal information, and the node connection structure; The proposed distribution information is applied to the relevant distributed nodes in the computing subsystem through the parameter adjustment layer to start the distribution process of the proposed information.

3. The method for storing a deep learning model in a consistent manner according to claim 2, wherein: The dynamic parameter distribution model also includes: a historical data analysis layer; Before adopting the dynamic parameter distribution model to obtain the second distributed node matching the proposal information and the corresponding dynamic threshold, the method further includes: The proposal response decision strategy set in the decision engine is updated based on historical proposal information and / or historical response data through the historical data analysis layer; the proposal response decision strategy at least includes: decision strategy parameters.

4. The method for storing a consistent deep learning model according to claim 2, wherein: The selecting a proposal response decision strategy that matches the proposal information and the node connection structure includes: Performing feature analysis on the proposal information to obtain corresponding proposal type, proposal demand level, and proposal priority; Performing feature analysis on the node connection structure to obtain a communication relationship type, a topology type, a network bandwidth, and a network delay corresponding to the first distributed node; Obtain target feature combinations corresponding to proposal type, proposal demand level, proposal priority, communication relationship type, topology type, network bandwidth, and network delay obtained through feature analysis; The proposed response decision strategy corresponding to the target feature combination is determined as the proposed response decision strategy matched by the first distributed node.

5. The method for storing a consistent deep learning model according to claim 4, wherein: The multiple proposal response decision strategies include at least: node load balancing strategy; The node load balancing strategy adjusts the calculation method of the dynamic threshold according to the node load, so that the probability of being selected as the second distributed node increases as the node load decreases; The process of generating the proposed distribution information in the dynamic threshold calculation layer is expressed as the following formula: Where D1 represents the dynamic threshold and γ is the adjustment factor. The adjustment factor in the above formula is used to balance the influence between the relative connection load of the node and the adjustment term based on the historical dynamic threshold D1′. represents the relative connection load of the node, where c i represents the computing resources consumed by the related nodes connected to the i-th node, w i represents the weight coefficient corresponding to the i-th node, R represents the resource utilization of the related nodes connected to the i-th node, δ represents the deviation value adjustment factor, B represents the task execution delay term, β and (1-β) are balance adjustment factors, α is the adjustment coefficient of resource utilization R, and (1-α) is the adjustment coefficient of the historical dynamic threshold D1′.

6. The method for storing a consistent deep learning model according to claim 4, wherein: The multiple proposed response decision strategies include at least: network communication delay strategy; The network communication delay strategy determines the threshold calculation method according to the communication delay between nodes, so that the probability of being selected as the second distributed node increases as the node communication delay decreases; The process of generating the proposed distribution information in the dynamic threshold calculation layer is expressed as the following formula: Where D2 represents the dynamic threshold, γ is the adjustment factor, and the adjustment factor in the above formula is used to balance the impact between the node's network communication delay term and the adjustment term based on the historical dynamic threshold D2′. i represents the network communication delay term of the i-th node, R represents the resource utilization of the related nodes connected to the i-th node, δ represents the deviation value adjustment factor, B represents the task execution delay term, β and (1-β) are balance adjustment factors, α is the adjustment coefficient of resource utilization R, and (1-α) is the adjustment coefficient of the historical dynamic threshold D2′.

7. The method for storing a consistent deep learning model according to claim 4, wherein: The multiple proposal response decision strategies include at least: node reliability strategy; The node reliability strategy dynamically adjusts the threshold according to the reliability of the node, so that the probability of being selected as the second distributed node increases with the improvement of the node reliability; The process of generating the proposed distribution information in the dynamic threshold calculation layer is expressed as the following formula: Where D3 represents the dynamic threshold, γ is the adjustment factor, and the adjustment factor in the above formula is used to balance the impact between the node's network communication delay term and the adjustment term based on the historical dynamic threshold D3′. i represents the reliability evaluation index value of the i-th node, R represents the resource utilization of the related nodes connected to the i-th node, δ represents the deviation value adjustment factor, B represents the task execution delay term, β and (1-β) are balance adjustment factors, α is the adjustment coefficient of resource utilization R, and (1-α) is the adjustment coefficient of the historical dynamic threshold D3′.

8. The method for storing consistent deep learning models according to claim 1, wherein: After triggering all distributed nodes in the computing subsystem to update the data to be updated, the method further includes: Monitoring the operational data of multiple distributed nodes, wherein the operational data includes at least: node operational status, resource utilization, and network connectivity information; Using a self-healing decision model, the operation status data is subjected to abnormality identification to obtain abnormality warning information indicating a node to be repaired; the abnormality warning information also includes: the abnormality type corresponding to the node to be repaired, and the scope of the abnormality impact; Based on the abnormality type and abnormal impact range of the node to be repaired, trigger the corresponding node repair operation and start real-time monitoring of the repair progress of the node to be repaired; The node repair operation includes at least: node exception handling, grayscale release, and rollback operation.

9. A computing subsystem, characterized in that: In a computing subsystem applied to a deep learning model, the computing subsystem includes at least a plurality of distributed nodes; a first distributed node in the computing subsystem includes at least the following units: an acquisition unit configured to create proposal information to be updated; The proposal information includes at least: data to be updated in the deep learning model, the data to be updated including model parameters and / or training data; a dynamic parameter distribution model is used to obtain proposal distribution information that matches the proposal information; wherein the proposal distribution information includes: a second distributed node for selecting the proposal information, and a corresponding dynamic threshold, the dynamic threshold being the minimum number of nodes required for the computing subsystem to respond to the proposal information, and the dynamic threshold being dynamically calculated by the dynamic parameter distribution model based on the proposal information and the computing subsystem; a distribution unit configured to send the proposal information to the second distributed node, so that the second distributed node determines whether to respond to the proposal information; The statistical unit is configured to receive the proposal response information of the second distributed node; if the number of the proposal response information issued reaches the dynamic threshold, all distributed nodes in the computing subsystem are triggered to update the data to be updated, so that the data to be updated maintains the consistency of the stored data in the computing subsystem.

10. An intelligent computing platform, characterized in that: The intelligent computing platform includes: at least one processor, memory, and input-output unit; The memory is used to store a computer program, and the processor is used to call the computer program stored in the memory to execute the deep learning model storage consistency method as described in any one of claims 1 to 8.

Images