Inference service system, model parameter updating method, equipment and storage medium

Abstract

The invention discloses an inference service system, a model parameter updating method, equipment and a storage medium. Comprising the steps that a training server obtains part of first-class model parameters and all second-class model parameters from a parameter server for training, and updates the gradient of the part of the first-class model parameters obtained through training to the parameter server, performing parameter updating on all the second type of model parameters by using the gradient of all the second type of model parameters obtained by training, and sending all the updated second type of model parameters to a parameter server; the parameter server accumulates the gradients of the partial first-class model parameters so as to update the partial first-class model parameters; and the reasoning server obtains the part of updated first-class model parameters and all updated second-class model parameters from the parameter server so as to update the model parameters. The efficient processing of model parameters from training updating to reasoning effectiveness is realized, the parameter processing time delay can be reduced, and the timeliness of reasoning service is improved.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to an inference service system, a model parameter update method, a device, and a storage medium. Background Technology

[0002] With the development of artificial intelligence technology, machine learning models (such as recommender system models, natural language processing models, and computer vision models) have been widely applied in e-commerce, social media, and autonomous driving. These machine learning models typically contain massive amounts of parameters and require dynamic updates based on real-time data streams to adapt to changes in the environment and user needs. Therefore, real-time synchronization of model parameters from the training phase to the inference service phase has become a key technical requirement to ensure the accuracy of model predictions and the speed of business response.

[0003] Currently, the data processing workflow for machine learning models generally adopts a "training-storage-inference" separation architecture. Specifically, after completing model training, the training node needs to store the complete model file in a distributed file system or object storage. Subsequently, the inference node pulls the entire model file from the storage system via the network, loads and parses it locally, and updates the parameters to provide inference services based on the new model parameters. This process exposes significant timeliness bottlenecks when the model parameter size grows to tens of gigabytes or even larger: on the one hand, the process of storing the model file on disk and distributing it across nodes consumes a large amount of network bandwidth and input / output resources, resulting in data transmission latency of up to several minutes; on the other hand, when the inference node loads large model files, it also consumes a large amount of memory and computing resources to receive data distributed by the file system and perform data reading and parsing, further extending the parameter effectiveness period. Taking a recommendation system scenario as an example, the entire process of a 60-gigabyte model from training and update to inference takes about 15 minutes. Among them, the model storage, distribution and loading stages account for more than 80% of the total latency. This long-cycle update mechanism cannot meet the business needs driven by real-time data flow, making it difficult for the inference model to quickly respond to dynamic factors such as changes in user behavior, changes in activity strategies and hot events to update parameters, thereby affecting the accuracy, timeliness and business conversion effect of the inference service. Summary of the Invention

[0004] The main objective of this invention is to provide an inference service system, a model parameter update method, a device, and a storage medium, aiming to solve the technical problem of poor timeliness in the parameter update process from training to inference effectiveness.

[0005] The first aspect of this invention provides an inference service system, comprising a training server, a parameter server, and an inference server. The parameter server stores model parameters including first-type model parameters and second-type model parameters, wherein the data size of the first-type model parameters is larger than that of the second-type model parameters. The training server is used to acquire a portion of the first-type model parameters and all of the second-type model parameters from the parameter server for training, update the gradients of the acquired portion of the first-type model parameters to the parameter server, update the parameters of all the second-type model parameters using the gradients of the acquired second-type model parameters, and send the updated parameters to the parameter server. The parameter server is used to accumulate the gradients of the acquired portion of the first-type model parameters to obtain accumulated gradients, update the parameters of the acquired portion of the first-type model parameters based on the accumulated gradients to obtain updated acquired portion of the first-type model parameters, and receive the updated parameters of all the second-type model parameters. The inference server is used to acquire the updated acquired portion of the first-type model parameters and the updated parameters of all the second-type model parameters from the parameter server for model parameter updating.

[0006] Optionally, in a first implementation of the first aspect of the present invention, the training server is further configured to: store all the acquired second-type model parameters in a training-side cache; intercept gradient updates during training to obtain the gradients of some first-type model parameters and all second-type model parameters; update the gradients of some first-type model parameters to the parameter server; update all second-type model parameters cached in the training-side cache based on the gradients of all second-type model parameters to obtain updated all second-type model parameters and cache them; and send the updated all second-type model parameters in the training-side cache to the parameter server according to a preset training period.

[0007] Optionally, in a second implementation of the first aspect of the present invention, the parameter server includes a master service node and a slave service node; the master service node is configured to: provide the training server with the partial first-type model parameters and the complete second-type model parameters; receive the gradients of the partial first-type model parameters and the complete second-type model parameters sent by the training server; accumulate the gradients of the partial first-type model parameters to obtain the accumulated gradient, and update the partial first-type model parameters based on the accumulated gradient to obtain the updated partial first-type model parameters; receive the updated complete second-type model parameters; and synchronize the updated partial first-type model parameters and the updated complete second-type model parameters to the slave service node; the slave service node is configured to: provide the updated partial first-type model parameters and the updated complete second-type model parameters to the inference server.

[0008] Optionally, in a third implementation of the first aspect of the present invention, the first type of model parameters stored in the parameter server are stored in key-value format; the training server is further configured to: establish communication with a preset accumulation module in the parameter server; update the gradient of the portion of the first type of model parameters to the preset accumulation module based on the key-value accumulation operator; the parameter server is further configured to: accumulate the gradient of the portion of the first type of model parameters based on the preset accumulation module to obtain the accumulated gradient.

[0009] Optionally, in a fourth implementation of the first aspect of the present invention, the inference server is further configured to: store all the updated second-type model parameters in the inference-side cache; store the updated portion of the first-type model parameters that meet the preset high-frequency reading conditions in the inference-side cache; and remove the updated portion of the first-type model parameters that meet the preset low-frequency reading conditions from the inference-side cache.

[0010] Optionally, in a fifth implementation of the first aspect of the present invention, the first type of model parameters are embedding layer parameters, and the second type of model parameters are fully connected layer parameters.

[0011] A second aspect of the present invention also provides a model parameter update method, which is applied to the inference service system described above. The model parameter update method includes: a training server obtaining a portion of first-type model parameters and all second-type model parameters from a parameter server for training, updating the gradient of the trained portion of the first-type model parameters to the parameter server, updating the all second-type model parameters using the gradient of the trained all second-type model parameters, and sending the updated all second-type model parameters to the parameter server; the parameter server accumulating the gradient of the portion of the first-type model parameters to obtain an accumulated gradient, updating the portion of the first-type model parameters based on the accumulated gradient to obtain the updated portion of the first-type model parameters, and receiving the updated all second-type model parameters; and the inference server obtaining the updated portion of the first-type model parameters and the updated all second-type model parameters from the parameter server to perform model parameter update.

[0012] Optionally, in a first implementation of the second aspect of the present invention, the parameter server includes a master service node and a slave service node; the model parameter update method further includes: the master service node providing the training server with the partial first-type model parameters and the complete second-type model parameters, and receiving the gradient of the partial first-type model parameters and the updated complete second-type model parameters sent by the training server, accumulating the gradient of the partial first-type model parameters to obtain the accumulated gradient, updating the partial first-type model parameters based on the accumulated gradient to obtain the updated partial first-type model parameters, and synchronizing the updated partial first-type model parameters and the updated complete second-type model parameters to the slave service node; the slave service node providing the updated partial first-type model parameters and the updated complete second-type model parameters to the inference server.

[0013] A third aspect of the present invention also provides a computer device, the computer device comprising: a memory and at least one processor, the memory storing instructions; the at least one processor calling the instructions in the memory to cause the computer device to perform the model parameter update method as described above.

[0014] A fourth aspect of the present invention also provides a computer-readable storage medium storing instructions that, when executed by a processor, implement the model parameter update method as described above.

[0015] This invention provides an inference service system, model parameter update method, device, and storage medium. The system integrates a training server, a parameter server, and an inference server. First, the training server hierarchically acquires partial data of a first type of model parameters and all data of a second type of model parameters for training. The gradient increment of the first type of model parameters is then updated to the parameter server. Simultaneously, the second type of model parameters are updated and sent locally. Next, the parameter server accumulates the gradients of the received first type of model parameters and updates the parameters, enabling the inference server to retrieve updated, differentiated parameters from the parameter server on demand and update the model parameters to provide real-time inference services based on the latest parameters. This system constructs an end-to-end closed-loop parameter update architecture that integrates training, storage, and inference. Under this architecture, the training server and inference server can share the parameter server, achieving seamless integration from training parameter updates to inference effectiveness, and enabling rapid parameter updates from training to inference. Meanwhile, in this system, model parameter updates are not achieved through full model file storage and cross-node distribution, but rather through gradient incremental transmission and differentiated parameter processing. This effectively reduces the inference service's dependence on large model files, avoids the network bandwidth consumption, input / output resource consumption, and memory and computing resource consumption caused by full model file retrieval, and reduces the latency caused by corresponding processing. It can effectively improve the timeliness of model parameter updates from training to inference, solving the problems of high parameter update latency, large resource consumption, and insufficient inference service stability under the traditional full model update mechanism. This is conducive to the rapid adaptation of machine learning model-related businesses to the dynamic changes of application scenarios, ensuring low latency and high stability of the inference service under large-scale parameters, and improving the accuracy, timeliness, and business conversion effect of the inference service. Attached Figure Description

[0016] Figure 1 This is a functional module diagram of one embodiment of the inference service system of the present invention; Figure 2 This is a schematic flowchart of one embodiment of the model parameter update method in this invention; Figure 3 This is a schematic diagram of the functional modules of a computer device according to one embodiment of the present invention. Detailed Implementation

[0017] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” or “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0018] For ease of understanding, the specific process of the inference service system in this embodiment of the invention is described below. Please refer to... Figure 1 , Figure 1 This is a schematic diagram of the functional modules of the first embodiment of the inference service system in this invention.

[0019] The inference service system 100 in this embodiment specifically includes: a training server 110, a parameter server 120, and an inference server 130; the model parameters stored in the parameter server 120 include a first type of model parameters and a second type of model parameters, and the data size of the first type of model parameters is larger than the data size of the second type of model parameters. Training server 110 is used to obtain some first-class model parameters and all second-class model parameters from parameter server 120 for training, update the gradient of the trained first-class model parameters to parameter server 120, update all second-class model parameters using the gradient of the trained second-class model parameters, and send the updated second-class model parameters to parameter server 120. The parameter server 120 is used to accumulate the gradients of some of the first-class model parameters to obtain the accumulated gradients, and to update the parameters of some of the first-class model parameters based on the accumulated gradients to obtain the updated partial first-class model parameters, as well as to receive all the updated second-class model parameters. The inference server 130 is used to obtain updated partial first-class model parameters and updated all second-class model parameters from the parameter server 120 for model parameter updates.

[0020] In this embodiment, the inference service system 100 is a distributed system that updates model parameters and provides real-time inference services. Specifically, it can be an end-to-end intelligent service system for large-scale parameter model application scenarios such as recommendation systems, natural language processing, or computer vision. In this system, the training server 110 is a service carrier used to execute model training tasks and implement gradient differential updates. Specifically, it can be built using a deep learning framework, based on algorithm data-driven methods, or obtained through other separately developed or secondary deep learning framework interfaces. It is a server node that can communicate with the parameter server 120 and perform model parameter gradient calculation and update training. The parameter server 120 is a distributed middleware in the inference service system 100 used to store and collaboratively update model parameters. Specifically, it can be obtained based on a remote dictionary service cluster or other distributed key-value storage services. It can support high-concurrency read and write operations and has a storage architecture with built-in or customizable atomic gradient accumulation operation modules. The inference server 130 is a service carrier in this system used to load the latest parameters and respond to inference requests. Specifically, it can be built based on the service framework corresponding to the deep learning framework or other hybrid computing architectures. It is a server node that can pull parameters and provide inference service interfaces. It can provide real-time inference prediction results to the business system to achieve rapid response to business needs.

[0021] In this embodiment, the parameters stored by the parameter server 120 include a first type of model parameters and a second type of model parameters. These two types of model parameters are distinguished by their data size, and different training, update, and inference service methods are employed accordingly. This allows the training server 110 and the inference server 130 in the inference service system 100 to acquire parameters on demand, thereby reducing resource consumption. Specifically, during model training, model parameters with larger data sizes typically do not involve full updates and have a lower update frequency, while model parameters with smaller data sizes usually require full participation in training and have a higher update frequency. Therefore, the first type of model parameters can specifically be the parameters with larger data sizes and lower update frequencies, while the second type of model parameters can specifically be the parameters with smaller data sizes and higher update frequencies. This ensures that during parameter updates and inference services, only a portion of the first type of model data needs to be retrieved on demand, without requiring full transmission. The second type of model data, due to its smaller size, typically participates in full updates, and its full retrieval and update do not significantly impact network load. This also ensures timeliness in high-frequency update scenarios, achieving rapid parameter synchronization and improving the response speed of the inference service.

[0022] In this embodiment, during model training, the training server 110 generates gradient data corresponding to two types of model parameters. The gradient refers to the direction and magnitude of the model parameter to be optimized, which can be obtained through gradient update interception during model training. By acquiring the gradient, the model parameters can be adjusted based on the direction and magnitude of the gradient, thereby updating the corresponding model parameters. Specifically, for the gradients of some of the first type of model parameters obtained after training, since the data scale of the corresponding parameters is large, it is difficult to store and calculate them on the training server 110. After acquiring the corresponding gradients, they can be transmitted to the parameter server 120, where the parameter server 120 performs atomic accumulation calculation on the gradients of some of the first type of model parameters stored therein, realizing real-time synchronization of the first type of model parameters in its storage. As for the gradients of all the second type of model parameters obtained after training, since their scale is small and the update frequency is high, they can be directly updated locally in the training server 110 and synchronized to the parameter server 120 after a certain accumulation, thereby reducing the number of cross-node communications and reducing the read and write pressure on the parameter server 120.

[0023] In this embodiment, after receiving the gradients of a portion of the first-type model parameters, the parameter server 120, due to the incremental nature of these gradients, can integrate them through accumulation to ensure the atomicity and computational continuity of gradient updates, achieving efficient synchronization without the need for full parameter transmission. This accumulation process yields an accumulated gradient, which represents the overall parameter optimization direction confirmed by the model training update. Therefore, based on this accumulated gradient, the corresponding first-type model parameters can be updated, completing parameter iteration solely through incremental gradients. This fine-grained parameter update method, unlike the traditional full-model-file overwrite update process, can compress parameter synchronization time from minutes to seconds, achieving end-to-end rapid response from training to inference.

[0024] In this embodiment, after obtaining updated partial first-type model parameters by accumulating gradient data and receiving all updated second-type model parameters sent by the training server 110, the parameter server 120 stores the updated partial first-type model parameters and all second-type model parameters so that the inference server 130 can retrieve the latest parameters as needed. The inference server 130 obtains the updated model parameters from the parameter server 120, thereby enabling it to update the parameters of the built-in or external inference model based on the updated model parameters during periodic or event-triggered inference services. This provides inference services based on the latest parameters, meeting the timeliness requirements of relevant businesses.

[0025] The inference service system 100 in this embodiment integrates a training server 110, a parameter server 120, and an inference server 130. First, the training server 110 hierarchically acquires partial data of the first type of model parameters and all data of the second type of model parameters for training. It then updates the gradient increments of some of the first type of model parameters to the parameter server 120, while simultaneously updating and sending the second type of model parameters locally. Next, the parameter server 120 accumulates the received gradients of the first type of parameters and updates the parameters, enabling the inference server 130 to retrieve the updated differentiated parameters from the parameter server 120 and update the model parameters to provide real-time inference services based on the latest parameters. This system can construct an end-to-end closed-loop parameter update architecture that integrates training, storage, and inference. Under this architecture, the training server 110 and the inference server 130 can share the parameter server 120, achieving seamless integration from training parameter updates to inference effectiveness, and enabling rapid parameter updates from training to inference. Meanwhile, in this system, model parameter updates are not achieved through full model file storage and cross-node distribution, but rather through gradient incremental transmission and differentiated parameter processing. This effectively reduces the inference service's dependence on large model files, avoids the network bandwidth consumption, input / output resource consumption, and memory and computing resource consumption caused by full model file retrieval, and reduces the latency caused by corresponding processing. It can effectively improve the timeliness of model parameter updates from training to inference, solving the problems of high parameter update latency, large resource consumption, and insufficient inference service stability under the traditional full model update mechanism. This is conducive to the rapid adaptation of machine learning model-related businesses to the dynamic changes of application scenarios, ensuring low latency and high stability of the inference service under large-scale parameters, and improving the accuracy, timeliness, and business conversion effect of the inference service.

[0026] Optionally, in some embodiments, the training server 110 is further configured to: store all acquired second-type model parameters in the training-side cache; intercept gradient updates during training to obtain the gradients of some first-type model parameters and all second-type model parameters; update the gradients of some first-type model parameters to the parameter server 120; update all second-type model parameters cached in the training-side cache based on the gradients of all second-type model parameters to obtain updated second-type model parameters and cache them; and send the updated second-type model parameters in the training-side cache to the parameter server 120 according to a preset training period.

[0027] In this optional embodiment, the training-side cache is a storage module built into the training server 110 or provided by its training object for parameter caching. It can be built based on local storage technologies such as a cache, in-memory database, or distributed caching system. Since the total amount of acquired second-type model parameter data is relatively small and needs to be fully used in training, it can be stored in this training-side cache for quick access and parameter updates during subsequent training. The first-type model parameter data is larger in scale, so only a portion can be pulled from the parameter server 120 as needed during training, and subsequently updated to the parameter server 120 via gradient increments, achieving efficient decoupling of training and storage for large-scale sparse parameters.

[0028] In this optional embodiment, the gradients corresponding to the model parameters are obtained through gradient update interception. This gradient update interception can be performed in real-time by the front-end component of the preset training framework under the architecture of the training server 110. For example, when the training server 110 is built based on a deep learning framework, the optimizer of the deep learning framework can be overloaded to trigger interception logic after gradient calculation at each training step, separating the gradients into a first type and a second type according to parameter type to obtain differentiated gradient data. Alternatively, gradient update interception can be performed by registering a custom callback function of the training framework, developing a gradient interception plugin, or extending computation graph operators to achieve real-time gradient capture and classification routing. Through this gradient update interception, the gradients of some first-type model parameters and all second-type model parameters can be obtained. Therefore, based on the parameter type and update strategy, differentiated processing can be achieved through collaboration between the training server 110 and the parameter server 120 for the two types of gradients.

[0029] In this optional embodiment, the gradients corresponding to the two types of model parameters are used differently to update the model parameters. Specifically, for the gradients of some of the first type of model parameters, since the data scale of the corresponding first type of model parameters is large and the updates are incremental, they can be directly transmitted to the parameter server 120 for atomic accumulation calculation. However, for the gradients of all the second type of model parameters, since the data scale of the corresponding second type of parameters is small and the update frequency is high, they can first be used to update all the second type of model parameters in the training-side cache to obtain the updated second type of model parameters, realize local iteration, and then synchronize to the parameter server 120 according to a preset training cycle. The preset training cycle can be a pre-set number of training steps (such as every 10-100 steps) or a time period (such as every 10-60 seconds). Through this synchronization according to the preset cycle, the frequency of write operations to the parameter server 120 is reduced, the overall throughput of the system is improved, and the stability of the parameter server 120 in high-concurrency scenarios is further guaranteed.

[0030] Optionally, in some embodiments, the parameter server 120 includes a master service node and a slave service node; the master service node is used to: provide a portion of the first type of model parameters and all the second type of model parameters to the training server 110; receive the gradients of the portion of the first type of model parameters and all the second type of model parameters sent by the training server 110; accumulate the gradients of the portion of the first type of model parameters to obtain the accumulated gradient, and update the parameters of the portion of the first type of model parameters based on the accumulated gradient to obtain the updated portion of the first type of model parameters; receive the updated all the second type of model parameters; and synchronize the updated portion of the first type of model parameters and the updated all the second type of model parameters to the slave service node; the slave service node is used to: provide the updated portion of the first type of model parameters and the updated all the second type of model parameters to the inference server 130.

[0031] In this optional embodiment, the parameter server 120 is further configured as a master-slave structure. Through a read-write separation and real-time synchronization mechanism between the master and slave service nodes, the decoupling of training parameter writing and inference parameter reading is achieved. The master service node, acting as a write operation node, is responsible for receiving gradient data sent by the training server 110 and executing parameter update logic. On one hand, it performs atomic accumulation calculations on the gradients of some first-type model parameters to ensure consistency in incremental updates of large-scale sparse parameters. On the other hand, it directly receives the full update results of the second-type model parameters, enabling rapid iteration of small-volume parameters. Through the asynchronous data synchronization mechanism of the master-slave architecture, the master service node pushes the updated complete parameter set to the slave service nodes in real time, ensuring complete consistency between the slave node data and the master node data. The slave service nodes, acting as read operation nodes, focus on responding to parameter retrieval requests from the inference server 130. The inference server 130 obtains the latest parameters by accessing the slave service nodes, avoiding read-write conflicts caused by directly accessing the master node, significantly reducing the access latency of the inference service. Simultaneously, the slave service nodes can achieve load balancing through clustered deployment, further improving the parameter reading throughput in high-concurrency scenarios.

[0032] In this optional embodiment, the master-slave structure can be implemented through a distributed key-value storage system. These systems need to natively support master-slave replication architecture and asynchronous data synchronization mechanisms, ensuring data consistency between master and slave nodes and providing high availability services. This meets the requirements of the parameter server 120 for high-concurrency read / write, data reliability, and low-latency access. By decoupling the master and slave service nodes of the parameter server 120, the master service node can focus on gradient accumulation calculation and parameter updates, while the slave service nodes focus on efficient response to parameter read requests on the inference side. This ensures that high-frequency write operations on the training side do not affect the low-latency read performance on the inference side, forming an efficient closed loop of "training write - master-slave synchronization - inference read". This effectively reduces the end-to-end latency of parameter updates from training to inference, solves the problem of insufficient service stability caused by read / write resource contention in traditional centralized parameter storage, and further enhances the horizontal scalability of the distributed system, which is beneficial for supporting real-time updates and inference services for larger-scale model parameters.

[0033] Optionally, in some embodiments, the first type of model parameters stored in the parameter server 120 are stored in key-value form; the training server 110 is further configured to: establish communication with the preset accumulation module in the parameter server 120; update the gradient of some of the first type of model parameters to the preset accumulation module based on the key-value accumulation operator; the parameter server 120 is further configured to: accumulate the gradient of some of the first type of model parameters based on the preset accumulation module to obtain the accumulated gradient.

[0034] In this optional embodiment, the implementation mechanism of gradient accumulation is further clarified. It utilizes a key-value accumulation operation based on key-value storage characteristics and a custom accumulation module in a collaborative design. A hash mapping is used to store the unique identifier of the parameter as the key and the corresponding vector parameter as the value, facilitating fast access to specific parameters and efficient accumulation and updating of large-scale sparse parameters. This key-value format allows the first type of model parameters to be quickly located and retrieved using the key, avoiding the excessive memory consumption caused by full loading, thus enabling the corresponding accumulation operation and improving the specificity and real-time performance of gradient updates.

[0035] In this optional embodiment, the training server 110 needs to first establish a communication connection with the preset accumulation module in the parameter server 120. This communication can be established through remote procedure calls, message queues, or application programming interfaces (APIs) for the targeted transmission and interaction of gradient data. During the accumulation operation, the training server 110 generates gradient update instructions based on the key-value accumulation operator. Specifically, this operator can be a key-value operation operator customized by the training server 110 based on the underlying computation graph of the training framework. This operator can bind gradient data with parameter keys, thereby achieving precise matching between gradients and parameters. This updates the gradients of some of the first type of model parameters to the preset accumulation module of the parameter server 120, allowing the preset accumulation module to perform atomic accumulation calculations on the received gradients. Specifically, the preset accumulation module is a functional module that is either independently deployed in the parameter server 120 or integrated into the storage engine. Since the parameter server 120 typically does not natively support atomic gradient accumulation, it needs to be extended to implement custom accumulation logic. This can be achieved by overloading storage commands, developing custom storage engine plugins, or using distributed transaction mechanisms. The preset accumulation module can then be built and registered in the parameter server 120, enabling it to batch and real-time accumulate the received gradients based on this module, thus obtaining the accumulated gradients and completing the incremental update of the first type of model parameters. This supports parameter consistency in scenarios with concurrent updates across multiple training nodes. This method of obtaining accumulated gradients achieves atomicity and concurrent processing performance in gradient updates, reduces communication overhead between the training server 110 and the parameter server 120, and provides underlying technical support for efficient training of large-scale sparse parameters.

[0036] Optionally, in some embodiments, the inference server 130 is further configured to: store all updated second-type model parameters in the inference-side cache; store some updated first-type model parameters that meet preset high-frequency reading conditions in the inference-side cache; and remove some updated first-type model parameters that meet preset low-frequency reading conditions from the inference-side cache.

[0037] In this optional embodiment, the inference server 130 also adopts a tiered storage and dynamic eviction strategy with a caching mechanism. Its inference-side cache is a multi-level storage architecture used to accelerate parameter access, which can be built based on local storage technologies such as high-speed cache, in-memory database, or distributed caching system. The inference-side cache stores all updated second-type model parameters. Since the data size of these updated second-type model parameters is small and the access frequency is extremely high, they can be fully loaded and stored in the inference-side cache during the inference service process, avoiding the network overhead of frequently pulling from the parameter server 120 during inference. On the other hand, the inference-side cache stores frequently accessed updated first-type model parameters. The corresponding preset high-frequency reading conditions are used to filter the most frequently accessed parameter data in the inference service. Specifically, it can be dynamically determined based on factors such as real-time access frequency, business scenario priority, or parameter timeliness. For example, it can be set that if the number of accesses in the past 5 minutes is ≥100, it belongs to the core inference service scenario, and / or the parameter update time is within 24 hours, the parameter is determined to be a high-frequency access object. By caching the updated first-type model parameters that meet the preset high-frequency reading conditions, most parameter access requests can be intercepted in the local cache, reducing the access pressure on the 120 slave nodes of the parameter server.

[0038] In this optional embodiment, for the first type of cached model parameters, since the memory capacity of the inference-side cache has an upper limit, in order to prevent cache overflow and maintain efficient access, a preset low-frequency read condition is used to trigger cache eviction, thereby releasing storage space and ensuring cache freshness. This preset low-frequency read condition can be based on quantitative indicators such as the duration of recent inactivity, the cumulative number of misses, or the cache space occupancy rate. For example, it can be set that a parameter is considered a low-frequency access object when it has not been accessed for 30 consecutive minutes, has accumulated 6 inference request misses, and / or the cache space occupancy rate exceeds a preset threshold. By removing low-frequency first-type model parameters, it can be ensured that cache resources are concentrated on high-value parameters, avoiding invalid data occupying memory resources. Simultaneously, since the model parameters in parameter server 120 are updated in real-time by the gradients and parameters obtained from training server 110, a periodic refresh mechanism can be used to asynchronously update the inference-side cache. For example, the version number of the model parameters in parameter server 120 and the model parameters in the local cache can be detected, and asynchronous incremental refreshes can be performed on the inference-side cache through asynchronous updates or scheduled tasks to ensure consistency between cached data and parameter server 120. Furthermore, this caching strategy can be further configured with eviction priority weights based on dynamic factors such as business traffic fluctuations, model iteration cycles, and hardware resource utilization, to adaptively balance cache hit rate and resource consumption, ensuring the stability of the inference service response in high-concurrency scenarios. Through this caching and eviction mechanism that combines categorized storage with dynamic eviction, the reuse rate of high-frequency parameters can be maximized with limited memory resources, enabling the inference server 130 to effectively compress parameter access latency, thereby better meeting the stringent requirements of online businesses for inference performance and achieving a more efficient, low-latency real-time inference service.

[0039] Optionally, in some embodiments, the first type of model parameters are embedding layer parameters, and the second type of model parameters are fully connected layer parameters.

[0040] In this optional embodiment, the first type of model parameters are defined as embedding layer parameters, and the second type of model parameters are defined as fully connected layer (Dense) parameters. These two types of parameters differ significantly in data scale, update frequency, and storage strategy. Embedding layer parameters are used to process sparse features and are typically high-dimensional and large in volume, usually accounting for over 95% of the model parameters. Their updates depend on the dynamic feedback of sample features during training, and they are incrementally updated parameters. They are typically stored in key-value pair structures, thus supporting efficient dynamic addition, deletion, and querying, and can meet the processing requirements of the first type of model parameters in the aforementioned steps, ensuring flexible adaptation of sparse features. Fully connected layer parameters, on the other hand, are weight parameters responsible for feature integration and output prediction. They are typically low-dimensional and small in volume. Their updates depend on the overall model training iteration process and are fully involved in training and updating during model training to ensure the accuracy of the model's inference output, and can meet the requirements for full loading and real-time access to the second type of model parameters in the aforementioned steps. By specifying the first type of model parameters as embedding layer parameters and the second type of model parameters as fully connected layer parameters, the differences in their structural characteristics and update requirements can be utilized to implement a hierarchical caching and differentiated update strategy for model parameters, thereby achieving a fast parameter update link from training to inference, and thus improving the real-time response capability and resource utilization efficiency of the inference service system 100.

[0041] Optionally, in some embodiments, the training server 110 is built using a deep learning framework, the parameter server 120 is built using a master-slave cluster architecture, and the inference server 130 is built using a service framework corresponding to the deep learning framework.

[0042] In this optional embodiment, the technology stack selection of the training server 110, parameter server 120, and inference server 130 in the inference service system 100 is further clarified. Specifically, the training server 110 is built using a deep learning framework. This deep learning framework needs to support distributed training and custom computation graph operations, as well as core functional modules such as computation graphs, optimizers, and gradient interception plugins. This enables the implementation of a series of model training processes, such as gradient calculation, parameter update generation, and custom operator development, as required by the training server 110, achieving more stable large-scale parameter training engineering deployment. The parameter server 120 is built using a master-slave cluster architecture. This master-slave cluster architecture needs to provide low-latency key-value pair storage and atomic operation support, and natively support the aforementioned master and slave service nodes. The read / write separation deployment allows the master node to handle write requests and the slave nodes to handle read traffic. The master-slave cluster architecture also needs to support custom module development to build the aforementioned pre-defined accumulation module, enabling strong consistency accumulation for gradient updates and more efficiently supporting high-frequency access to large-scale parameters. The inference server 130 corresponds to the model service requirements of the deep learning framework. A suitable service framework is selected to achieve seamless integration with the training framework. This service framework needs to dynamically monitor model parameter version changes and trigger incremental loading to provide rapid inference response and low-latency service support for real-time inference scenarios. The inference service system 100 constructed through the above selection achieves a closed-loop technology stack for training, storage, and inference. Components collaborate efficiently through standardized interfaces, enabling end-to-end rapid updates from parameter training completion to inference availability. This facilitates support for real-time inference needs in complex business scenarios and ensures the stability and efficiency of the inference service.

[0043] Please see Figure 2 , Figure 2 This is a schematic flowchart of an embodiment of the model parameter update method according to the present invention. The model parameter update method includes: 201. The training server obtains some first-class model parameters and all second-class model parameters from the parameter server for training, updates the gradient of the trained first-class model parameters to the parameter server, updates all second-class model parameters using the gradient of the trained second-class model parameters, and sends the updated second-class model parameters to the parameter server. 202. The parameter server accumulates the gradients of some of the first-class model parameters to obtain the accumulated gradient, and updates the parameters of some of the first-class model parameters based on the accumulated gradient to obtain the updated partial first-class model parameters, and receives all the updated second-class model parameters. 203. The inference server obtains the updated partial first-class model parameters and the updated complete second-class model parameters from the parameter server to update the model parameters.

[0044] Optionally, in some embodiments, the model parameter update method further includes: the training server storing all acquired second-type model parameters in the training-side cache; the training server intercepting gradient updates during training to obtain the gradients of some first-type model parameters and all second-type model parameters; the training server updating the gradients of some first-type model parameters to the parameter server; the training server updating all second-type model parameters cached in the training-side cache based on the gradients of all second-type model parameters to obtain updated second-type model parameters and caching them; and the training server sending the updated second-type model parameters from the training-side cache to the parameter server according to a preset training period.

[0045] Optionally, in some embodiments, the parameter server includes a master service node and a slave service node; the model parameter update method further includes: the master service node providing a portion of the first type of model parameters and all the second type of model parameters to the training server, and receiving the gradient of the portion of the first type of model parameters and the updated all the second type of model parameters sent by the training server, accumulating the gradient of the portion of the first type of model parameters to obtain the accumulated gradient, and updating the portion of the first type of model parameters based on the accumulated gradient to obtain the updated portion of the first type of model parameters; and synchronizing the updated portion of the first type of model parameters and the updated all the second type of model parameters to the slave service node; the slave service node providing the updated portion of the first type of model parameters and the updated all the second type of model parameters to the inference server.

[0046] Optionally, in some embodiments, the first type of model parameters stored in the parameter server are stored in key-value form; the model parameter update method further includes: the training server establishing communication with the preset accumulation module in the parameter server; the training server updating the gradient of some of the first type of model parameters to the preset accumulation module based on the key-value accumulation operator; and the parameter server accumulating the gradient of some of the first type of model parameters based on the preset accumulation module to obtain the accumulated gradient.

[0047] Optionally, in some embodiments, the model parameter update method further includes: the inference server storing all updated second-type model parameters in the inference-side cache; the inference server storing some updated first-type model parameters that meet preset high-frequency reading conditions in the inference-side cache; and the inference server removing some updated first-type model parameters that meet preset low-frequency reading conditions from the inference-side cache.

[0048] Optionally, in some embodiments, the first type of model parameters are embedding layer parameters, and the second type of model parameters are fully connected layer parameters.

[0049] Optionally, in some embodiments, the training server is built using a deep learning framework, the parameter server is built using a master-slave cluster architecture, and the inference server is built using a service framework corresponding to the deep learning framework.

[0050] Since the embodiments of the method section correspond to the embodiments of the system described above, please refer to the embodiments of the inference service system 100 described above for an introduction to the model parameter update method provided by the embodiments of the present invention. The embodiments of the present invention will not be described again here, but have the same beneficial effects as the inference service system 100 described above.

[0051] The present invention also provides a computer device, the computer device including a memory and a processor, the memory storing computer-readable instructions, which, when executed by the processor, cause the processor to perform the model parameter update method as described above. Figure 3 This is a functional block diagram of a computer device 300 provided in an embodiment of the present invention. The computer device 300 can vary significantly due to different configurations or performance characteristics. It may include one or more central processing units (CPUs) 310 (e.g., one or more processors) and a memory 320, and one or more storage media 330 (e.g., one or more mass storage devices) for storing application programs 333 or data 332. The memory 320 and storage media 330 can be temporary or persistent storage. The program stored in the storage media 330 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the computer device 300. Furthermore, the processor 310 may be configured to communicate with the storage media 330 and execute the series of instruction operations in the storage media 330 on the computer device 300.

[0052] Computer device 300 may also include one or more power supplies 340, one or more wired or wireless network interfaces 350, one or more input / output interfaces 360, and / or one or more operating systems 331. Those skilled in the art will understand that... Figure 3 The computer device structure shown does not constitute a limitation on the computer device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0053] The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium or a volatile computer-readable storage medium, wherein the computer-readable storage medium stores instructions that, when executed on a computer, cause the computer to perform the model parameter update method as described above.

[0054] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the device described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. If the integrated module or unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0055] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An inference service system, characterized by, The inference service system comprises a training server, a parameter server and an inference server; the model parameters stored by the parameter server comprise first-type model parameters and second-type model parameters, and the data scale of the first-type model parameters is larger than that of the second-type model parameters; The training server is configured to obtain part of the first-type model parameters and all the second-type model parameters from the parameter server for training, update the gradient of the part of the first-type model parameters to the parameter server, and update the parameters of all the second-type model parameters based on the gradient of all the second-type model parameters obtained after training, and send the updated all the second-type model parameters to the parameter server; The parameter server is configured to accumulate the gradient of the part of the first-type model parameters to obtain an accumulated gradient, update the parameters of the part of the first-type model parameters based on the accumulated gradient to obtain updated part of the first-type model parameters, and receive the updated all the second-type model parameters; The inference server is configured to obtain the updated part of the first-type model parameters and the updated all the second-type model parameters from the parameter server for model parameter updating.

2. The inference service system of claim 1, wherein, The training server is further configured to: store the obtained all the second-type model parameters to a training-side cache; during training, intercept gradient updating to obtain the gradient of the part of the first-type model parameters and the gradient of all the second-type model parameters; update the gradient of the part of the first-type model parameters to the parameter server; update the all the second-type model parameters cached in the training-side cache based on the gradient of all the second-type model parameters to obtain updated all the second-type model parameters and cache them; send the updated all the second-type model parameters in the training-side cache to the parameter server according to a preset training period.

3. The inference service system of claim 1, wherein, The parameter server comprises a master service node and a slave service node; The master service node is configured to: provide the part of the first-type model parameters and all the second-type model parameters to the training server; receive the gradient of the part of the first-type model parameters and the updated all the second-type model parameters sent by the training server; accumulate the gradient of the part of the first-type model parameters to obtain an accumulated gradient, and update the parameters of the part of the first-type model parameters based on the accumulated gradient to obtain updated part of the first-type model parameters; synchronize the updated part of the first-type model parameters and the updated all the second-type model parameters to the slave service node; The slave service node is configured to: provide the updated part of the first-type model parameters and the updated all the second-type model parameters to the inference server.

4. The inference service system of claim 1, wherein, The first-type model parameters stored in the parameter server are stored in the form of key-value; The training server is further configured to: establish communication with a preset accumulation module in the parameter server; update the gradient of the part of the first-type model parameters to the preset accumulation module based on the operator of the key-value accumulation operation; The parameter server is further configured to: based on the preset accumulation module, accumulate the gradient of the part of the first type of model parameters to obtain the accumulated gradient.

5. The inference service system of claim 1, wherein, The inference server is further configured to: store the updated all second type of model parameters to an inference side cache; store the updated part of the first type of model parameters meeting the preset high-frequency reading condition to the inference side cache; remove the updated part of the first type of model parameters meeting the preset low-frequency reading condition in the inference side cache.

6. The inference service system of any one of claims 1-5, wherein, The first type of model parameters are embedding layer parameters, and the second type of model parameters are fully connected layer parameters.

7. A model parameter updating method characterized by, The method comprises: The training server obtains part of the first type of model parameters and all second type of model parameters from the parameter server for training, updates the gradient of the part of the first type of model parameters obtained by training to the parameter server, and updates the parameters of all second type of model parameters using the gradient of all second type of model parameters obtained by training, and sends the updated all second type of model parameters to the parameter server; The parameter server accumulates the gradient of the part of the first type of model parameters to obtain the accumulated gradient, and updates the parameters of the part of the first type of model parameters based on the accumulated gradient to obtain the updated part of the first type of model parameters, and receives the updated all second type of model parameters; The inference server obtains the updated part of the first type of model parameters and the updated all second type of model parameters from the parameter server for model parameter updating.

8. The model parameter updating method of claim 7, wherein, The parameter server comprises a master service node and a slave service node; The model parameter updating method further comprises: The master service node provides the part of the first type of model parameters and the all second type of model parameters to the training server, receives the gradient of the part of the first type of model parameters and the updated all second type of model parameters sent by the training server, accumulates the gradient of the part of the first type of model parameters to obtain the accumulated gradient, updates the parameters of the part of the first type of model parameters based on the accumulated gradient to obtain the updated part of the first type of model parameters, and synchronizes the updated part of the first type of model parameters and the updated all second type of model parameters to the slave service node; The slave service node provides the updated part of the first type of model parameters and the updated all second type of model parameters to the inference server.

9. A computer device, comprising: The computer device comprises a memory and at least one processor, and the memory stores instructions; The at least one processor invokes the instructions in the memory, so that the computer device executes the model parameter updating method in claim 7 or claim 8.

10. A computer-readable storage medium having stored thereon instructions, the instructions comprising, The instructions are executed by the processor to implement the model parameter updating method in claim 7 or claim 8.

Images