Optimization method and system for uninterrupted upgrade based on large model training

Abstract

The application provides an optimization method for uninterrupted upgrade based on large model training, an independent intermediate connection layer is introduced between the training layer and the bottom basic data layer, and the external connection of the training layer is uniformly managed and maintained by the layer. When the bottom basic data layer changes, the intermediate connection layer will temporarily disconnect the link with the bottom layer, but still maintain the long connection state with the training layer. During this period, the request issued by the training layer will be temporarily stored in the local queue by the intermediate connection layer, and the response will be returned immediately, so that the training layer can continue to perform other computing tasks without blocking and waiting. After the bottom service is restored, the intermediate connection layer will re-establish communication and gradually process the backlog requests, while informing the training layer to perform subsequent operations. Through this mechanism, the training interruption caused by the bottom upgrade can be effectively avoided, and the overall training efficiency and system availability are significantly improved.

Description

Technical Field

[0001] This invention relates to the fields of computer software engineering and artificial intelligence applications, and in particular to an optimization method and system for non-disruptive upgrades based on large model training. Background Technology

[0002] In this new stage of industrial development characterized by the deep integration of digitalization and intelligentization, artificial intelligence (AI) technology has permeated various industries, including finance, manufacturing, healthcare, transportation, and the internet, becoming a core driving force for industrial upgrading, improved production efficiency, and the restructuring of service models. The depth and breadth of its technological applications and industrial implementation continue to expand. The core capabilities of AI systems are concentrated in key areas such as accurate analysis of complex business problems, in-depth analysis of multi-dimensional data, and intelligent responses to scenario-based needs. The realization and optimization of these capabilities highly rely on the technical support of large-scale AI models, such as large language models and multimodal large models. The performance, training efficiency, and stability of these large models directly determine the practical application effects of AI systems.

[0003] Because existing large-scale model training systems are highly dependent on the underlying data layer, and the coupling between training tasks and the data layer is high, any change to the data layer can easily lead to a sudden interruption of the training process. This not only results in the ineffective consumption of a large amount of computing and time resources in the early stages, but also causes problems such as data loss and model parameter corruption due to training interruption. To build artificial intelligence systems applicable to multiple industries and fields, it is necessary to train large models with massive amounts of data. As the amount of training data continues to grow, reaching billions or even tens of billions of data points, the time taken for each training process is significantly extended. During this process, if changes to the data layer are caused by system expansion, service upgrades, or underlying infrastructure anomalies, training tasks will be interrupted, seriously affecting efficiency and schedule. Therefore, how to achieve smooth upgrades of underlying services without interrupting the training process has become an urgent technical challenge to be solved. Summary of the Invention

[0004] To overcome the above problems, the purpose of this invention is to provide an optimization method and system for uninterrupted upgrades based on large model training, which can effectively avoid training interruptions caused by underlying upgrades and significantly improve overall training efficiency and system availability.

[0005] This invention is implemented using the following scheme: an optimization method for non-disruptive upgrade based on large model training, comprising the following steps: Step 1: Introduce an intermediate connection layer between the training layer and the underlying basic data layer in the traditional artificial intelligence system architecture. This intermediate connection layer will uniformly maintain the external communication connections of the training layer. Step 2: The intermediate connection layer and the underlying basic data layer interact with each other using short connections; when a training request arrives at the intermediate connection layer, the intermediate connection layer will dynamically create a short connection with the underlying basic data layer service to complete request forwarding and result return; Step 3: When the underlying basic data layer changes or is upgraded, the connection between the intermediate connection layer and the underlying basic data layer will be disconnected. However, the connection between the training layer and the intermediate connection layer will still be maintained. All request commands will be temporarily stored by the intermediate connection layer. At this time, the training layer does not need to wait for response information and can still perform other business operations. Step 4: After the underlying basic data layer has finished processing, the intermediate connection layer re-establishes the connection with the underlying basic data layer and then notifies the training layer again.

[0006] Furthermore, a long-term connection mechanism is used between the training layer and the intermediate connection layer to ensure the long-term stability of the communication link between the two parties.

[0007] Furthermore, step 3 further involves the following steps: After receiving the signal, the intermediate connection layer sets a connection disconnection flag in its local memory. Subsequent requests will check this flag before accessing the underlying basic data layer. If the flag is on, the request is forwarded to the underlying basic data layer normally. If the flag is off, the request is recorded in the local queue and a response is returned immediately, so that the training layer does not need to block and wait, and can continue to execute other tasks.

[0008] Furthermore, step 4 further comprises: after the underlying basic data layer completes the upgrade and restores the service, it will actively send a ready notification to the intermediate connection layer. The intermediate connection layer will then start a local queue processing mechanism to process the temporarily stored requests one by one in sequence. After each request is processed, a response is sent to the training layer through a long connection to ensure that the training task can be seamlessly connected and continue to be executed.

[0009] The present invention also provides an optimization system for uninterrupted upgrade based on large model training, including an intermediate processing module, a short connection module, a temporary storage processing module, and a reconnection processing module; The intermediate processing module introduces an intermediate connection layer between the training layer and the underlying basic data layer in the traditional artificial intelligence system architecture. This intermediate connection layer uniformly maintains the external communication connection of the training layer. The short connection module uses short connections to interact with the underlying basic data layer. When a training request arrives at the intermediate connection layer, the intermediate connection layer dynamically creates a short connection with the underlying basic data layer to forward the request and return the result. When the underlying basic data layer changes or is upgraded, the temporary storage processing module will disconnect the connection between the intermediate connection layer and the underlying basic data layer. However, the connection between the training layer and the intermediate connection layer will still be maintained. All request commands will be temporarily stored by the intermediate connection layer. At this time, the training layer does not need to wait for response information and can still perform other business operations. The reconnection processing module waits for the underlying basic data layer to finish processing, and then the intermediate connection layer re-establishes a connection with the underlying basic data layer before notifying the training layer again.

[0010] Furthermore, a long-term connection mechanism is used between the training layer and the intermediate connection layer to ensure the long-term stability of the communication link between the two parties.

[0011] Furthermore, the temporary storage processing module is implemented as follows: after receiving the signal, the intermediate connection layer sets a connection disconnection flag in its local memory. Subsequent requests will check this flag before accessing the underlying basic data layer: if the flag is on, it will be forwarded to the underlying basic data layer normally; if the flag is off, the request will be recorded in the local queue and a response will be returned immediately, so that the training layer does not need to block and wait, and can continue to execute other tasks.

[0012] Furthermore, the reconnection processing module is implemented as follows: after the underlying basic data layer completes the upgrade and restores the service, it will actively send a ready notification to the intermediate connection layer. The intermediate connection layer will then start a local queue processing mechanism to process the temporarily stored requests one by one in sequence. After each request is processed, a response is sent to the training layer through a long connection to ensure that the training task can be seamlessly connected and continue to be executed.

[0013] The beneficial effects of this invention are as follows: through a multi-layered connection management and request buffering mechanism involving the training layer, intermediate connection layer, and underlying basic data layer, the system can maintain the continuous execution of training tasks during changes in underlying services, effectively avoiding interruptions caused by upgrades. This method significantly improves the coherence and efficiency of large-scale model training and is suitable for high-concurrency, long-cycle AI training scenarios. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the method flow of the present invention.

[0015] Figure 2 This is a schematic flowchart of an embodiment of the present invention.

[0016] Figure 3 This is a schematic diagram of the system of the present invention. Detailed Implementation

[0017] The invention will now be further described with reference to the accompanying drawings.

[0018] Please see Figure 1 As shown, the present invention provides an optimization method for non-disruptive upgrade based on large model training, comprising the following steps: Step 1: Introduce an intermediate connection layer between the training layer and the underlying basic data layer in the traditional artificial intelligence system architecture. This intermediate connection layer will uniformly maintain the external communication connections of the training layer. Step 2: The intermediate connection layer and the underlying basic data layer interact with each other using short connections; when a training request arrives at the intermediate connection layer, the intermediate connection layer will dynamically create a short connection with the underlying basic data layer service to complete request forwarding and result return; Step 3: When the underlying basic data layer changes or is upgraded, the connection between the intermediate connection layer and the underlying basic data layer will be disconnected. However, the connection between the training layer and the intermediate connection layer will still be maintained. All request commands will be temporarily stored by the intermediate connection layer. At this time, the training layer does not need to wait for response information and can still perform other business operations. Step 4: After the underlying basic data layer has finished processing, the intermediate connection layer re-establishes the connection with the underlying basic data layer and then notifies the training layer again.

[0019] The following is a further explanation using a specific embodiment: An optimization method for non-disruptive upgrades based on large model training. Step 1: In traditional AI system architectures, there are typically two main layers: a training layer and a lower-level data layer. To achieve uninterrupted upgrades, this method introduces an intermediate connection layer between the two, which uniformly maintains the external communication connections of the training layer. A long-lived connection mechanism is used between the training layer and the intermediate connection layer to ensure a persistent and stable communication link, avoiding performance overhead caused by frequent connection establishment, while also guaranteeing the real-time performance and reliability of data transmission.

[0020] Step 2: The intermediate connection layer and the underlying basic data layer interact with each other using short connections. When a training request arrives at the connection layer, the intermediate connection layer dynamically creates a short connection with the underlying basic data layer service to forward the request and return the result. This design reduces the connection maintenance burden on the underlying basic data layer service and can flexibly adapt to dynamic changes in underlying resources, improving the overall scalability of the system.

[0021] Step 3: When the underlying data layer needs to be changed or upgraded, a notification will be sent to the intermediate connection layer in advance. Upon receiving the signal, the intermediate connection layer will set a connection disconnection flag in its local memory. Subsequent requests will check this flag before accessing the underlying layer: if the flag is on, it will be forwarded to the underlying layer normally; if the flag is off, the request will be recorded in a local queue and a response will be returned immediately, allowing the training layer to continue executing other tasks without being blocked.

[0022] Step 4: After the underlying data layer completes the upgrade and restores service, it will proactively send a ready notification to the intermediate connection layer. The intermediate connection layer then initiates a local queue processing mechanism to process the temporarily stored requests sequentially. After processing each request, a response is sent to the training layer via a persistent connection to ensure that the training task can be seamlessly connected and continue execution.

[0023] This invention, through the aforementioned multi-layered connection management and request buffering mechanism, enables the system to maintain the continuous execution of training tasks during changes to underlying services, effectively avoiding interruptions caused by upgrades. This method significantly improves the coherence and efficiency of large-scale model training and is suitable for high-concurrency, long-cycle AI training scenarios.

[0024] like Figure 2 As shown, for example: 1. Here, both the training layer and the intermediate connection layer use long connections for data interaction.

[0025] 2. The intermediate connection layer and the underlying basic data layer use short links based on common HTTP or RPC protocols for data interaction.

[0026] 3. The normal process is as follows: When the training layer needs to obtain a piece of data, it sends a message to the middle layer through a short connection, and the middle layer forwards this command and sends a message to the underlying basic data layer through the short connection.

[0027] When the underlying layer restarts, undergoes changes, or other operations that require service interruption or temporary inaccessibility, the training layer sends data to the intermediate connection layer when needed. The intermediate connection layer caches these messages in a local queue and monitors the accessibility of the underlying data layer. When access is available, the intermediate connection layer prioritizes data requests from the queue using a first-in, first-out (FIFO) strategy and sends the messages to the underlying layer via short connections. Once the underlying data is received, the intermediate connection layer uses the long-lived connection information recorded in the queue to find the corresponding long-lived connection and returns the required training data to the training layer through this channel. This method achieves uninterrupted training.

[0028] like Figure 3 As shown, the present invention provides an optimization system for uninterrupted upgrade based on large model training, including an intermediate processing module, a short connection module, a temporary storage processing module, and a reconnection processing module. The intermediate processing module introduces an intermediate connection layer between the training layer and the underlying basic data layer in the traditional artificial intelligence system architecture. This intermediate connection layer uniformly maintains the external communication connection of the training layer. The short connection module uses short connections to interact with the underlying basic data layer. When a training request arrives at the intermediate connection layer, the intermediate connection layer dynamically creates a short connection with the underlying basic data layer to forward the request and return the result. When the underlying basic data layer changes or is upgraded, the temporary storage processing module disconnects the intermediate connection layer from the underlying basic data layer. However, the connection between the training layer and the intermediate connection layer remains intact. All request commands are temporarily stored by the intermediate connection layer. At this time, the training layer does not need to wait for response information and can still perform other business operations. The temporary storage processing module is further implemented as follows: after receiving a signal, the intermediate connection layer sets a connection disconnection flag in its local memory. Subsequent requests will check this flag before accessing the underlying basic data layer: if the flag is on, it is forwarded normally to the underlying basic data layer; if the flag is off, the request is recorded in the local queue and a response is returned immediately, so that the training layer does not need to block and wait, and can continue to execute other tasks.

[0029] The reconnection processing module waits for the underlying basic data layer to finish processing, and then the intermediate connection layer re-establishes a connection with the underlying basic data layer before notifying the training layer again.

[0030] The reconnection processing module is further implemented as follows: after the underlying basic data layer completes the upgrade and restores the service, it will actively send a ready notification to the intermediate connection layer. The intermediate connection layer will then start a local queue processing mechanism to process the temporarily stored requests one by one in sequence. After each request is processed, a response is sent to the training layer through a long connection to ensure that the training task can be seamlessly connected and continue to be executed.

[0031] The training layer and the intermediate connection layer employ a long-lived connection mechanism to ensure a persistent and stable communication link between them. This approach allows the entire training process to proceed without interruption, thereby improving overall training efficiency to a certain extent.

[0032] The above description is only a preferred embodiment of the present invention. All equivalent changes and modifications made within the scope of the claims of the present invention should be included in the scope of the present invention.

Claims

1. An optimization method for non-disruptive upgrades based on large model training, characterized in that: Includes the following steps: Step 1: Introduce an intermediate connection layer between the training layer and the underlying basic data layer in the traditional artificial intelligence system architecture. This intermediate connection layer will uniformly maintain the external communication connections of the training layer. Step 2: The intermediate connection layer and the underlying basic data layer interact with each other using short connections; when a training request arrives at the intermediate connection layer, the intermediate connection layer will dynamically create a short connection with the underlying basic data layer service to complete request forwarding and result return; Step 3: When the underlying basic data layer changes or is upgraded, the connection between the intermediate connection layer and the underlying basic data layer will be disconnected. However, the connection between the training layer and the intermediate connection layer will still be maintained. All request commands will be temporarily stored by the intermediate connection layer. At this time, the training layer does not need to wait for response information and can still perform other business operations. Step 4: After the underlying basic data layer has finished processing, the intermediate connection layer re-establishes the connection with the underlying basic data layer and then notifies the training layer again.

2. The optimization method for non-disruptive upgrade based on large model training according to claim 1, characterized in that: The training layer and the intermediate connection layer employ a long-term connection mechanism to ensure the persistent and stable communication link between the two parties.

3. The optimization method for non-disruptive upgrade based on large model training according to claim 1, characterized in that: Step 3 further involves the following steps: After receiving the signal, the intermediate connection layer sets a connection disconnection flag in its local memory. Subsequent requests will check this flag before accessing the underlying basic data layer. If the flag is on, it will be forwarded to the underlying basic data layer normally. If the flag is off, the request will be recorded in the local queue and a response will be returned immediately, so that the training layer does not need to block and wait, and can continue to execute other tasks.

4. The optimization method for non-disruptive upgrade based on large model training according to claim 1, characterized in that: Step 4 further involves the following steps: After the underlying basic data layer completes the upgrade and restores the service, it will proactively send a ready notification to the intermediate connection layer. The intermediate connection layer will then start a local queue processing mechanism to process the temporarily stored requests one by one in sequence. After each request is processed, a response will be sent to the training layer through a long connection to ensure that the training task can be seamlessly connected and continue to be executed.

5. An optimization system for uninterrupted upgrades based on large model training, characterized in that: It includes an intermediate processing module, a short connection module, a temporary storage processing module, and a reconnection processing module; The intermediate processing module introduces an intermediate connection layer between the training layer and the underlying basic data layer in the traditional artificial intelligence system architecture. This intermediate connection layer uniformly maintains the external communication connection of the training layer. The short connection module uses short connections to interact with the underlying basic data layer. When a training request arrives at the intermediate connection layer, the intermediate connection layer dynamically creates a short connection with the underlying basic data layer to forward the request and return the result. When the underlying basic data layer changes or is upgraded, the temporary storage processing module will disconnect the connection between the intermediate connection layer and the underlying basic data layer. However, the connection between the training layer and the intermediate connection layer will still be maintained. All request commands will be temporarily stored by the intermediate connection layer. At this time, the training layer does not need to wait for response information and can still perform other business operations. The reconnection processing module waits for the underlying basic data layer to finish processing, and then the intermediate connection layer re-establishes a connection with the underlying basic data layer before notifying the training layer again.

6. The optimization system for non-disruptive upgrades based on large model training according to claim 5, characterized in that: The training layer and the intermediate connection layer employ a long-term connection mechanism to ensure the persistent and stable communication link between the two parties.

7. The optimization system for non-disruptive upgrade based on large model training according to claim 5, characterized in that: The temporary storage processing module is further implemented as follows: after receiving the signal, the intermediate connection layer sets a connection disconnection flag in its local memory. Subsequent requests will check the flag before accessing the underlying basic data layer: if the flag is on, it will be forwarded to the underlying basic data layer normally; if the flag is off, the request will be recorded in the local queue and a response will be returned immediately, so that the training layer does not need to block and wait, and can continue to execute other tasks.

8. The optimization system for non-disruptive upgrades based on large model training according to claim 5, characterized in that: The reconnection processing module is further implemented as follows: after the underlying basic data layer completes the upgrade and restores the service, it will actively send a ready notification to the intermediate connection layer. The intermediate connection layer will then start a local queue processing mechanism to process the temporarily stored requests one by one in sequence. After each request is processed, a response is sent to the training layer through a long connection to ensure that the training task can be seamlessly connected and continue to be executed.

Images