Offline training method and system for large business models, training nodes, and cache system

Abstract

The embodiment of the present invention provides a method and system for offline training of a large business model, a training node, and a cache system. The method includes: determining the target training file required for the large model to be trained under the target business; wherein the target business is one or a combination of the search business, the advertising business, and the recommendation business; when the target training file is not cached, triggering the caching of the target training file according to a preset time granularity; using the cached target training file to train the large model to be trained; after determining that the target training file has been trained, returning to the step of determining the target training file required for the large model to be trained under the target business, until all target training files have been trained. The embodiment of the present invention can reduce the cache cost during offline training, and can also advance the model training process according to time to achieve the purpose of rolling training, and can continuously improve the training quality of the model to meet the real-time requirements of the search business, the advertising business, and the recommendation business.

Description

Technical Field

[0001] The present invention relates to the field of model training technology, and in particular to a large-scale offline training method and system for business models, a training node, and a cache system. Background Art

[0002] Currently, Internet platforms all have search systems, advertising systems, and recommendation systems. These systems usually use machine learning algorithms, especially large models, to provide users with accurate and personalized business services.

[0003] Typically, model training for search, advertising, and recommendation services is divided into two stages: offline training and online training. Offline training uses large amounts of accumulated data for training and is generally used during service initialization. Online training uses real-time small batch data for online iteration based on the offline training model, enabling the model to adapt to changing user trends, thereby achieving the effect of providing personalized and real-time responsive services to users.

[0004] Offline model training requires a large amount of training data. Prior to offline training, existing techniques simply cache all required training data before model training. While implementing the present invention, the inventors discovered that this offline training approach requires a significant amount of storage space and is prohibitively expensive. Therefore, in business scenarios such as search, advertising, and recommendation, a training solution that reduces caching costs is a technical challenge that needs to be addressed. Summary of the Invention

[0005] One of the objectives of the present invention is to provide a method and system for offline training of large business models, training nodes, and a cache system. These methods can reduce cache costs during offline training of large business models, enable the model training process to progress on a timeline, and achieve rolling training. This method can continuously improve the training quality of the model and meet the real-time requirements of search, advertising, and recommendation services. The embodiments of the present invention can be implemented as follows:

[0006] In a first aspect, the present invention provides a method for offline training of a large business model, the method comprising: determining a target training file required for a large model to be trained under a target business; wherein the target business is one or a combination of a search business, an advertising business, and a recommendation business; when the target training file is not cached, triggering caching of the target training file according to a preset time granularity; using the cached target training file to train the large model to be trained; after determining that the target training file has been trained, returning to the step of determining the target training file required for the large model to be trained under the target business until all target training files have been trained.

[0007] In a second aspect, the present invention provides a method for offline training of a large business model, the method comprising: receiving a cache request; the cache request is sent by a training node when it perceives that a target training file required by a large model to be trained under a target business is not cached; the target business is one or a combination of a search business, an advertising business and a recommendation business; according to a preset time granularity, historical business data in the target business is pulled and cached to obtain the target training file; the training node is used to train the large model to be trained using the cached target training file until all target training files are trained.

[0008] In a third aspect, the present invention provides a training node, comprising: a cache file system and a training container; the cache file system is used to determine the target training files required for the large model to be trained under the target business; wherein the target business is one or a combination of a search business, an advertising business and a recommendation business; the cache file system is also used to trigger caching of the target training files according to a preset time granularity when the target training files are not cached; the training container is used to train the large model to be trained using the cached target training files; the cache file system is also used to determine that after the target training files are trained, return to the step of determining the target training files required for the large model to be trained under the target business, until all target training files are trained.

[0009] In a fourth aspect, the present invention provides a cache system, comprising: a control node and multiple storage nodes; the control node is used to receive a cache request; the cache request is sent by the training node when it perceives that the target training file required by the large model to be trained under the target business is not cached; the target business is one or a combination of search business, advertising business and recommendation business; the control node is also used to pull the historical business data in the target business according to a preset time granularity and cache it in multiple storage nodes to obtain the target training file; the training node is used to train the large model to be trained using the cached target training file until all target training files are trained.

[0010] In a fifth aspect, the present invention provides an offline training system for a large business model, comprising multiple training nodes, a cache system and a remote storage system; the cache system is communicatively connected to the training nodes and the remote storage system respectively; each of the training nodes is used to execute the offline training method for the large business model as described in any one of the aforementioned embodiments, and the cache system is used to execute the offline training method for the large business model as described in the aforementioned embodiments.

[0011] The offline training method and system, training node, and cache system for a large business model provided by an embodiment of the present invention have the following advantages: the embodiment of the present invention first determines the target training file for training the large model corresponding to the target business. If it is detected that the target training file has not been cached, it triggers a cache operation for the target training file, uses the cached target training file to perform model training, and determines the next target training file after training, and continues to execute the above process. This shows that the embodiment of the present invention does not need to cache all target training files before training, but performs real-time caching during the training process, which can achieve the purpose of reducing caching costs. At the same time, the embodiment of the present invention caches in real time according to a preset time granularity, which is equivalent to reducing the full-level cache in the prior art to the time granularity level cache, which can also achieve the purpose of reducing caching costs. In addition, since the target business is one or a combination of search business, advertising business and recommendation business, these businesses are real-time, and data and models need to be updated in a timely manner to meet user needs. The embodiment of the present invention caches the target training files according to a preset time granularity, so that the cached target training files have a time sequence. Then, when these target training files are used to train the model in sequence, the model training process can be advanced in time to achieve the purpose of rolling training, which can continuously improve the training quality of the model and meet the real-time requirements of search business, advertising business and recommendation business. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0013] Figure 1 A schematic diagram of an application scenario of the large model offline training method provided by an embodiment of the present invention;

[0014] Figure 2 yes Figure 1 Schematic diagram of the principle of the server processing search business, advertising business and recommendation business;

[0015] Figure 3 A schematic diagram of the structure of a large business model offline training system 30 provided in an embodiment of the present invention;

[0016] Figure 4 A schematic flow chart of an offline training method for a large business model provided by an embodiment of the present invention;

[0017] Figure 5 An example of a data storage directory with hourly granularity provided by an embodiment of the present invention;

[0018] Figure 6 Another schematic flow chart of the offline training method for a large business model provided by an embodiment of the present invention;

[0019] Figure 7 A schematic flowchart of another offline training method for a large business model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0021] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but rather merely represents selected embodiments of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort shall fall within the scope of protection of the present invention.

[0022] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not require further definition or explanation in subsequent drawings.

[0023] In the description of the present invention, it should be noted that if the terms "upper", "lower", "inside", "outside", etc. appear, the orientation or position relationship indicated is based on the orientation or position relationship shown in the accompanying drawings, or is the orientation or position relationship in which the product of the invention is usually placed when in use. It is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it should not be understood as a limitation on the present invention.

[0024] In addition, the terms "first", "second", etc., if used, are merely used to distinguish and describe, and should not be understood as indicating or implying relative importance.

[0025] It should be noted that, in the absence of conflict, the features in the embodiments of the present invention may be combined with each other.

[0026] Please refer to Figure 1 , Figure 1 Schematic diagram of an application scenario of the large-model offline training method provided in an embodiment of the present invention.

[0027] The terminals may include, but are not limited to, various desktop computers, laptops, smartphones, tablet computers, IoT devices, and portable wearable devices. IoT devices may include smart speakers, smart TVs, smart air conditioners, smart car devices, etc. Portable wearable devices may include smart watches and smart bracelets, etc.

[0028] The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, as well as big data and artificial intelligence platforms.

[0029] The terminal and the server may be connected directly or indirectly via wired or wireless communication, which is not limited in the embodiment of the present invention.

[0030] In one embodiment of the present invention, Figure 1 The application scenarios shown include advertising, search, and recommendation. In the search service, a user enters a query into the search box on the terminal. The terminal sends the search request to the server, which uses the search model to process the query and provide relevant results. In the advertising service, the server uses the ad matching model to match ads based on the page content viewed by the user on the terminal. In the recommendation service, the server uses the recommendation model to provide content of interest to the user based on the user's clicks, favorites, browsing, or purchases on the terminal.

[0031] See Figure 2 , Figure 2 yes Figure 1 Schematic diagram of the principle of the server processing search business, advertising business and recommendation business, Figure 2 The search, advertising and recommendation big model is the abbreviation of the business big models corresponding to the search business, advertising business and recommendation business in the embodiment of the present invention.

[0032] In order to provide users with personalized search, recommendation and advertising services, the server can use deep learning and other technologies to train large business models for search, advertising and recommendation services, thereby improving business processing efficiency and service quality.

[0033] like Figure 2 As shown, before model training, the server can first build a training dataset. The data in the dataset can be accumulated historical business data and real-time search, advertising, and recommendation data. Historical business data can be used for offline training, and real-time search, advertising, and recommendation business data can be used for online training based on offline training.

[0034] During the offline training phase, the server collects and labels a large amount of historical business data, which contains the user's personalized behavior records. The server then uses this accumulated historical business data to initialize the network model and obtain a stable basic model.

[0035] During the online training phase, the server collects real-time user interaction data, such as clicks, browsing, and searches, and feeds this data back into the existing model for parameter updates. This continuous, small-step iteration builds on the foundational model to adapt to evolving user trends, continuously improving the model's predictive capabilities and enhancing service quality.

[0036] Continue to see Figure 2 After the business models corresponding to the search business, advertising business and recommendation business obtained through offline training and online training pass the model testing, they can be applied to specific businesses. Specifically: taking the search business as an example, the server receives the search business request, and then inputs the business request data into the business model corresponding to the search business for processing. The model outputs the corresponding search results, and then the server sends the search results to the terminal for display.

[0037] Typically, real-time business data for online training is in small batches, occupying little cache space. However, offline training requires a large amount of historical business data. If this historical business data is stored in low-speed storage media (such as HDFS), directly reading it for training will seriously affect training speed. Therefore, transferring this data to high-speed storage media before training can improve training speed.

[0038] In the process of implementing the present invention, the inventors found that most of the current offline training methods are to first cache all the data used for training in a high-speed cache system before starting training. This training method undoubtedly requires a large amount of storage space and is too costly.

[0039] In order to solve the above problems, in one embodiment of the present invention, the caching characteristics of data in search business, advertising business and recommendation business are combined to provide an offline training system for large business models, which can not only significantly reduce the cost required for caching, but also improve the quality of business services.

[0040] See Figure 3 , Figure 3 The structural diagram of the offline training system for a large business model provided in an embodiment of the present invention includes a remote storage system 301, a cache system 302 and a training node 303, and the cache system 302 is communicated with the remote storage system 301 and the training node 303 respectively.

[0041] Remote storage system 301 is used to store all raw data used for machine learning model training. It is independent of the big data storage system designed for training node 303, where the machine learning model is trained, and does not participate in the training process. To reduce the cost of long-term storage of this data, remote storage system 301 can use low-performance storage media such as SATA, HDD, and linear tape.

[0042] The cache system 302 is used to cache training files. For search, advertising, and recommendation services, training files can be stored at different time granularities (e.g., by day, hour, or minute). This ensures a natural chronological order between training files, allowing for subsequent rolling training of the model based on this chronological order. For search, advertising, and recommendation services, which are highly real-time, rolling training ensures the timeliness and relevance of service content.

[0043] In an optional embodiment, the cache system 302 may be a high-speed storage medium, such as a memory. The cache system 302 may be deployed in a cluster to provide faster reading performance than remote storage.

[0044] In one embodiment of the present invention, the cache system 302 and the training node 303 are deployed in different devices or device clusters. The training node 303 senses whether the target training file is cached and executes the model training task, and the cache system 302 performs real-time caching operations on the training file. This allows the training node 303 to focus on training calculations, while the cache system 302 focuses on storage loading, thereby improving the overall performance of the system.

[0045] In one embodiment of the present invention, the cache system 302 may include a control node 3021 and a storage node 3022. The control node 3021 can pull business data for model training from the remote storage system 301 and cache it in the form of training files, and can also read training files from the storage node; the storage node 3022 is used to store training files.

[0046] In one embodiment of the present invention, the cache system 302 is used to receive cache requests; the cache request is sent by the training node when it perceives that the target training file required by the large model to be trained under the target business is not cached; the target business is one or a combination of search business, advertising business and recommendation business; it is also used to pull and cache historical business data in the target business according to a preset time granularity to obtain a target training file; the training node is used to train the large model to be trained using the cached target training file until all target training files are trained, thereby reducing the cache cost during offline training, and also allowing the model training process to advance on time to achieve the purpose of rolling training, and can continuously improve the training quality of the model to meet the real-time requirements of the search business, advertising business and recommendation business.

[0047] In an optional embodiment, the cache system 302 may adopt a distributed architecture, that is, Figure 3 There can be multiple control nodes 3021, and each control node 3021 can be responsible for multiple storage nodes 3022, so that large-scale concurrent fast caching can be achieved on multiple machines and multiple processes.

[0048] In addition, when the cache system 302 adopts a distributed architecture, during the process of caching training files, the control node 3021 can split a single training file into multiple small blocks and cache them in each storage node 3022. When reading a training file, the control node 3021 can read the offset and size from the corresponding small block and return the read result. If the read result is empty, it indicates that the training file is not cached. In this case, the data can be directly pulled from the remote storage system for caching.

[0049] In an optional embodiment, the cache system 302 can also eliminate the used cache when the capacity is insufficient, thereby achieving the purpose of releasing memory in time and providing storage space for subsequent cache files used for model training.

[0050] The training node 303 is used to read the training data in the training file from the cache system 302 for model training. Multiple training nodes 303 can be deployed to form a cluster.

[0051] In an optional embodiment, the training node 303 may be, but is not limited to, a group of physical or virtual servers, GPUs, etc.

[0052] In one embodiment of the present invention, training node 303 may include a cache file system 3031 and a training container 3032. Training container 3032 is a workspace for model training, specifically executing model training tasks based on the training data provided by cache file system 3031. Cache file system 3031 is used to interact with cache system 302 and use the training data cached in cache system 302 in the form of a normal file system in training container 3032.

[0053] In one embodiment of the present invention, the cache file system 3031 may include a cache awareness module 3031-1 and a file IO 3031-2. The cache awareness module 3031-1 is configured to detect in real time whether a training file for training a large business model is cached. If not, the cache system 302 may be triggered to cache the training file in real time according to different time granularities.

[0054] Time granularity refers to the storage time unit for historical business data and is also the time unit for the training files used during online training. During the training of the large business models corresponding to the search, advertising, and recommendation services, if online training uses training files cached at hourly granularity, offline training will also use the same training files cached at hourly granularity, starting from the beginning and ending points.

[0055] In an optional implementation, the time granularity may be by date, by hour, by minute, etc. Users may flexibly select according to actual needs, and the embodiment of the present invention does not limit this.

[0056] In one embodiment of the present invention, the training node 303 is used to determine the target training files required for the large model to be trained under the target business; wherein the target business is one or a combination of the search business, the advertising business and the recommendation business; when the target training file is not cached, the target training file is triggered to be cached according to the preset time granularity; the large model to be trained is trained using the cached target training file; after determining that the target training file has been trained, the step of determining the target training files required for the large model to be trained under the target business is returned until all the target training files are trained, thereby reducing the cache cost during the offline training process, and also enabling the model training process to advance according to time to achieve the purpose of rolling training. In this way, the training can continuously improve the training quality of the model and meet the real-time requirements of the search business, the advertising business and the recommendation business.

[0057] File IO 3031-2 is used to provide the cached training data to the training container 3032 in the form of a normal file system.

[0058] In an optional embodiment, the functions that file IO 3031-2 can implement include but are not limited to: obtaining file attributes, such as file size, creation time, modification time, etc.; directory attributes, such as the number of files contained in the directory, directory size, etc.; reading file content, allowing users or programs to read file data; changing file content, allowing users or programs to write or modify files.

[0059] Combining the functions of the remote storage system 301, the cache system 302 and the training node 303, in one embodiment of the present invention, Figure 3The business model training system shown can determine the target training files required for the large model to be trained under the target business through the training node 303; wherein the target business is one or a combination of the search business, the advertising business and the recommendation business; when the target training file is not cached, the cache system 302 is triggered to cache the target training file according to the preset time granularity; then the large model to be trained is trained using the cached target training file through the training node 303; after determining that the target training file has been trained, the step of determining the target training files required for the large model to be trained under the target business is returned until all target training files are trained. Therefore, the business model offline training system of the embodiment of the present invention can reduce the cache cost, and can also make the model training process advance according to time to achieve the purpose of rolling training, and can continuously improve the training quality of the model to meet the real-time requirements of the search business, the advertising business and the recommendation business.

[0060] based on Figure 3 The provided business model offline training system, the embodiment of the present invention provides a business model offline training method, see Figure 4 , Figure 4 A schematic flow chart of a method for offline training of a large business model provided by an embodiment of the present invention, which is applied to Figure 3 On the training node 303 in the process, the following steps are included:

[0061] S401: Determine the target training file required for the large model to be trained under the target business; wherein the target business is one or a combination of search business, advertising business and recommendation business;

[0062] S402: When the target training file is not cached, triggering caching of the target training file according to a preset time granularity;

[0063] S403: Using the cached target training file to train the large model to be trained;

[0064] S404: After determining that the target training file has been trained, return to step S401 until all target training files have been trained.

[0065] In the scheme of steps S401 to S404, the embodiment of the present invention first determines the target training file for training the large model corresponding to the target business. If it is detected that the target training file is not cached, it triggers the caching operation for the target training file, uses the cached target training file to perform model training, and determines the next target training file after training, and continues to execute the above process. This shows that the embodiment of the present invention does not need to cache all target training files before training, but performs real-time caching during the training process, which can achieve the purpose of reducing caching costs. At the same time, the embodiment of the present invention caches in real time according to the preset time granularity, which is equivalent to reducing the full-level cache in the prior art to the time granularity level cache, which can also achieve the purpose of reducing caching costs. In addition, since the target business is one or a combination of search business, advertising business and recommendation business, these businesses are real-time, and data and models need to be updated in a timely manner to meet user needs. The embodiment of the present invention caches the target training files according to a preset time granularity, so that the cached target training files have a time sequence. Then, when these target training files are used to train the model in sequence, the model training process can be advanced in time to achieve the purpose of rolling training, which can continuously improve the training quality of the model and meet the real-time requirements of search business, advertising business and recommendation business.

[0066] The following embodiments of the present invention will provide a detailed description of each of the above steps.

[0067] Regarding the above step S401, an embodiment of the present invention provides such an implementation method:

[0068] Step a1: respond to the file opening operation;

[0069] Step a2: Determine a target cache directory according to a preset time granularity, and determine the training file corresponding to the target cache directory as the target training file.

[0070] In one embodiment of the present invention, the business data of the search business, the advertising business and the recommendation business can be stored at different time granularities such as day, hour or minute. For example, see Figure 5 , Figure 5 This is an example of a data storage directory with hourly granularity provided by an embodiment of the present invention. Part-r-* represents actual data.

[0071] Data with low update frequency, such as overall user profile data, can be stored at larger time granularity, such as year / month / day. Data with high update frequency, such as real-time interaction data such as user searches, clicks, and browsing, can be stored at smaller time granularity, such as hours or minutes.

[0072] Therefore, based on this storage characteristic, the embodiment of the present invention can determine which time period of cached data should be used for model training according to a preset time granularity. This time granularity-based caching strategy can not only reduce the caching cost, but also achieve the purpose of rolling training of the model and improve the quality of model training.

[0073] Combine Figure 3 For cache node 303 in the training container, steps a1 to a2 can be understood as follows: When a training task is initiated, training container 3032 opens the required training files from the mounted cache file system 3031. Upon detecting the training file open operation, cache file system 3031 determines which time period to use for the cache directory based on a preset time granularity. This step prepares for accessing or updating the cache, ensuring that training container 3032 can quickly read the relevant training data.

[0074] For example, a preset time granularity is configured in the system, which is "hourly" here. Therefore, the cache file system 3031 responds to the training file open operation, obtains the cache directory, such as / 2024 / 03 / 31 / 00, and then detects whether the / 2024 / 03 / 31 / 00 directory exists in the cache. If it exists, it means that the training data under the / 2024 / 03 / 31 / 00 directory has been pre-cached on the fast storage medium, and the training data can be read directly from the cache. If the / 2024 / 03 / 31 / 00 directory is not in the cache, the file caching operation for / 2024 / 03 / 31 / 00 can be triggered.

[0075] It can be seen that the above steps a1 to a2 are more suitable for determining the first target training file at the beginning of the training task. If you want to achieve the purpose of rolling training using target training files at a preset time granularity, after each target training file is trained, you can also determine the next target training file according to the following method:

[0076] Step b1: Determine the next cache directory based on the cache directory of the trained target training file and the preset time granularity, and determine the training file corresponding to the next cache directory as the target training file.

[0077] In other words, after the current target training file is trained, the cache directory for the next target training file can be accurately calculated based on the preset time granularity and the cache directory of the currently trained target training file. For example, referring to step 5, if the preset time granularity is hours, and the training file corresponding to directory / 2024 / 03 / 31 / 00 has been trained, then the next target training file will be the training file corresponding to directory / 2024 / 03 / 31 / 01. This allows for rapid iteration and speeds up training.

[0078] In another embodiment, the embodiment of the present invention can also set a file opening order, so after a target training file is trained, it can be determined which training file is the next target training file based on the file opening order.

[0079] Regarding step S402, in order to detect whether the target training file is cached, an embodiment of the present invention provides the following implementation:

[0080] Step c1: If the cache directory of the target training file does not exist, or if the cache directory of the target training file does not have a cache identifier, it is determined that the target training file is not cached.

[0081] That is to say, in one embodiment, a cache identifier may be pre-configured for a cached training file, and the cache identifier may be used to quickly identify whether the target training file has been cached.

[0082] Based on the above implementation, in one embodiment of the present invention, a cache identifier can be pre-configured for a cached training file after it is cached. When the training file is deleted due to insufficient capacity, the cache identifier can be deleted at the same time to avoid subsequent misjudgments.

[0083] In addition, the cache identifier can be flexibly represented in the form of text, numbers, etc., which is not limited in the embodiment of the present invention.

[0084] In one embodiment of the present invention, considering that the cache system can eliminate the used cache when the capacity is insufficient, there are two types of perception results for the target training file:

[0085] The first result is that the target training file has been cached.

[0086] In this case, the target training file can be directly read to read the training data for model training.

[0087] The second result is that the target training file is not cached. In this case, the target training file has either never been cached or has been cached but deleted.

[0088] Under this result, as shown in step S402, a caching operation of the target training file according to a preset time granularity can be triggered.

[0089] In one embodiment of the present invention, a method for triggering the caching operation of the target training file according to the preset time granularity may be:

[0090] Step d1: Generate a cache request and send the cache request to the cache system, so that the cache system pulls and caches the historical business data of the target business according to a preset time granularity.

[0091] It is understood that a cache request can be used to instruct the cache system to pull and cache historical business data corresponding to the target business at a preset time granularity. For example, if the directory is / 2024 / 03 / 31 / 00, then this directory can be sent to the cache system. After receiving it, the cache system can pull the historical business data as of 0:00 on March 31, 2024, and cache it in the file in the directory / 2024 / 03 / 31 / 00, thus obtaining the target training file.

[0092] In addition, if the directory / 2024 / 03 / 31 / 00 does not exist in the cache system, the cache system can first create a directory / 2024 / 03 / 31 / 00, and then cache the historical business data at 0:00 on March 31, 2024 into it.

[0093] In one embodiment of the present invention, since the cache system and training nodes are deployed in different devices, the training nodes sense whether the target training files are cached, and the cache system performs the cache operation, which allows the training nodes to focus on training calculations and the cache system to focus on storage loading, thereby improving the overall performance of the system.

[0094] In one embodiment of the present invention, after determining that the cache system has completed caching of the target training file, the training data in the target training file may be used to perform model training, ie, executing step S403.

[0095] The method of determining whether the target training file has been cached may be: determining that the target cache file has a cache identifier, or receiving a message indicating successful caching from the cache system, or receiving training data in the target training file returned by the cache system.

[0096] Regarding step S403, in one embodiment of the present invention, a training method is provided:

[0097] Step e1: Read a preset number of training data from the target training file to perform model training;

[0098] In an optional implementation, the amount of training data read each time can be customized by the user, for example, 10 or 100 pieces of training data can be read each time, which is not limited here.

[0099] Step e2: After the training data is trained, determine whether the training data in the target training file has been read;

[0100] If not, return to step e1; otherwise, return to step S401.

[0101] The above implementation method can promptly determine whether the target training file has been trained, thereby accelerating the training speed.

[0102] To better understand the overall implementation process of the offline training method for large business models on the training node side, the following is an explanation from the perspective of code. Figure 6 , Figure 6 Another schematic flowchart of the offline training method for a large business model provided in an embodiment of the present invention.

[0103] S1: Open the target training file required for the large model to be trained under the target business;

[0104] S2: Get the directory of the target training file according to the preset time granularity;

[0105] S3: Determine whether the directory is cached;

[0106] If not, execute S4; if so, execute S5;

[0107] S4: Send a cache request to the cache system so that the cache system caches the target training file;

[0108] S5: Read a preset number of training data from the target training file to perform model training;

[0109] S6: Determine whether the training data in the target training file has been read;

[0110] If not, return to execute S5; if so, return to execute S1;

[0111] By implementing the above-mentioned large business model training process, the purpose of rolling training of large business models in chronological order can be achieved, avoiding caching a large amount of data at one time, thereby reducing caching costs. At the same time, the rolling training method can also improve the performance and adaptability of the model.

[0112] Based on Figure 4 For the same inventive concept, please see Figure 7 , Figure 7 A schematic flow chart of another offline training method for a large business model provided by an embodiment of the present invention, the execution subject of the method is Figure 3 The cache system 302 includes the following steps:

[0113] S701: Receive a cache request; the cache request is sent by the training node when it detects that the target training file required by the large model to be trained under the target service is not cached; the target service is one or a combination of the search service, the advertising service, and the recommendation service;

[0114] S702: Pull and cache the historical business data in the target business according to the preset time granularity to obtain the target training file; the training node is used to train the large model to be trained using the cached target training file until all target training files are trained.

[0115] Through the above-mentioned scheme of steps S701 to S702, the cache system in the embodiment of the present invention does not need to directly cache all training files during model training, but caches them after receiving the cache request sent by the training node, thereby reducing the cache cost.

[0116] In an optional implementation, when the cache capacity is insufficient, the cache system can also automatically delete earlier cached data to free up storage space and provide sufficient storage space for training files required for subsequent model training.

[0117] In one embodiment of the present invention, Figure 3 The training node 303 shown includes a cache file system 3031 and a training container 3032. Figure 4 The same inventive concept is that the cache file system 3031 is used to determine the target training files required for the large model to be trained under the target business; wherein the target business is one or a combination of search business, advertising business and recommendation business. The cache file system 3031 is also used to trigger the caching of the target training files according to the preset time granularity when the target training files are not cached. The training container 3032 is used to train the large model to be trained using the cached target training files; the cache file system 3031 is also used to determine that after the target training files are trained, return to the step of determining the target training files required for the large model to be trained under the target business until all target training files are trained.

[0118] It is understandable that the cache file system 3031 and the training container 3032 can be executed in coordination. Figure 4 Each step in the process is performed to achieve the corresponding technical effects.

[0119] In an optional implementation, the cache file system 3031 is further used to generate a cache request and send the cache request to the cache system, so that the cache system pulls and caches the historical business data of the target business according to a preset time granularity.

[0120] In an optional embodiment, the cache file system 3031 is further used to determine that the target training file is not cached if there is no cache directory for the target training file, or if there is no cache identifier in the cache directory.

[0121] In an optional implementation, the cache file system 3031 is also used to respond to a file open operation; determine a target cache directory according to a preset time granularity, and determine the training file corresponding to the target cache directory as the target training file.

[0122] In an optional embodiment, after determining that the target training file has been trained, the target training file required for the large model to be trained under the target business is determined, and the cache file system 3031 is also used to determine the next cache directory based on the cache directory of the trained target training file and the preset time granularity, and determine the training file corresponding to the next cache directory as the target training file.

[0123] In an optional embodiment, the training container 3032 is specifically used to read a preset number of training data from the target training file to perform model training; after the training data is trained, it is determined whether the training data in the target training file has been read; if not, the step of reading a preset number of training data from the target training file to perform model training is returned; otherwise, the step of determining the target training file required for the large model to be trained under the target business is returned.

[0124] In one embodiment of the present invention, Figure 3 The cache system 302 shown includes a control node 3021 and a storage node 3022. Figure 7 The same inventive concept is that the control node 3021 is used to receive a cache request; the cache request is sent by the training node when it perceives that the target training file required by the large model to be trained under the target business is not cached; the target business is one or a combination of search business, advertising business and recommendation business; the control node 3021 is also used to pull the historical business data in the target business according to the preset time granularity and cache it in multiple storage nodes 3022 to obtain the target training file; the training node is used to use the cached target training file to train the large model to be trained until all target training files are trained.

[0125] It is understandable that the control node 3021 and the storage node 3022 can execute collaboratively. Figure 7 Each step in the process is performed to achieve the corresponding technical effects.

[0126] It should be noted that the division of modules in the above embodiments of the present application is illustrative and is only a logical functional division. In actual implementation, there may be other division methods. In addition, the functional units in the various embodiments of the present application may be integrated into a processing unit, or may exist separately physically, or two or more units may be integrated into a single unit. The above-mentioned integrated units may be implemented in the form of hardware or in the form of software functional units.

[0127] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling an electronic device (which can be a personal computer, server, or electronic device, etc.) or a processor to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0128] In one embodiment of the present invention, a first computer device and a second computer device are further provided. The first computer device is configured to execute the following Figure 4 The offline training method of the business big model shown in FIG. Figure 7 In the illustrated method for offline training a large business model, the first and second computing devices may each include a memory, a processor, and a communication interface. The memory, processor, and communication interface are electrically connected to each other, directly or indirectly, to enable data transmission or interaction. For example, these components may be electrically connected to each other via one or more communication buses or signal lines.

[0129] In an embodiment of the present invention, the processor may be a general-purpose processor, a digital signal processor, an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component, and may implement or execute the various methods, steps, and logic block diagrams disclosed in the embodiment of the present invention. A general-purpose processor may be a microprocessor or any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiment of the present application may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor. The software module may be located in a memory, and the processor reads the program instructions in the memory and completes the steps of the above method in conjunction with its hardware.

[0130] In an embodiment of the present invention, the memory may be a non-volatile memory, such as a hard disk drive (HDD) or a solid-state drive (SSD), or a volatile memory (volatile memory), such as RAM. The memory may also be any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory in the embodiment of the present application may also be a circuit or any other device that can implement a storage function, for storing instructions and / or data.

[0131] The memory can be used to store software programs and modules, which can be stored in the memory in the form of software or firmware or embedded in the operating system (OS) of the first and second computer devices. The processor executes the software programs and modules stored in the memory 601 to perform various functional applications and data processing. The communication interface can be used to communicate signaling or data with other node devices.

[0132] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0133] Based on the above embodiments, the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a first computer device, the first computer device executes the offline training method for the large business model provided in the above embodiments.

[0134] Based on the above embodiments, the present application also provides a computer program, which, when executed on a first computer device, causes the first computer device to execute Figure 4 The provided business model offline training method. When the computer program is run on the second computer device, the second computer device executes Figure 7 Provides offline training methods for large business models.

[0135] Based on the above embodiments, the embodiments of the present application also provide a chip, which is used to read the computer program stored in the memory and to execute the offline training method for the large business model provided in the above embodiments.

[0136] A computer program product is also provided in an embodiment of the present application, including instructions, which, when running, enable a first computer device to execute the offline training method for the large business model provided in the above embodiment.

[0137] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by instructions. These instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0138] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0139] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0140] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.

Claims

1. A method for offline training of a large business model, characterized in that: The method comprises: Determining a target training file required for a large model to be trained under a target business, including: determining a target cache directory according to a preset time granularity, and determining a training file corresponding to the target cache directory as the target training file; wherein the target business is one or a combination of a search business, an advertising business, and a recommendation business; When the perception cache module of the cache file system in the training node senses that the target training file is not cached, it triggers the cache system to cache the target training file according to the preset time granularity; Using the cached target training file to train the large model to be trained; After determining that the target training file has been trained, return to the step of determining the target training file required for the large model to be trained under the target business, including: determining the next cache directory based on the cache directory of the trained target training file and the preset time granularity, and determining the training file corresponding to the next cache directory as the target training file, until all target training files are trained.

2. The offline training method for a large business model according to claim 1, characterized in that: Triggering caching of the target training file according to a preset time granularity includes: A cache request is generated and sent to a cache system, so that the cache system pulls and caches the historical service data of the target service according to the preset time granularity.

3. The offline training method for a large business model according to claim 1, characterized in that: The method for sensing that the target training file is not cached includes: If the cache directory of the target training file does not exist, or if the cache directory of the target training file does not have a cache identifier, it is determined that the target training file is not cached.

4. The offline training method for a large business model according to claim 1, characterized in that: Determine the target training files required for the large model to be trained under the target business, including: Responds to a file open operation.

5. The offline training method for a large business model according to claim 1, characterized in that: Training the large model to be trained using the cached target training file includes: Read a preset number of training data from the target training file to perform model training; After the training data is trained, determining whether the training data in the target training file has been read; If not, return to the step of reading a preset number of training data from the target training file to perform model training; otherwise, return to the step of determining the target training file required for the large model to be trained under the target business.

6. A method for offline training of a large business model, characterized in that: The method comprises: Receive a cache request; the cache request is sent by a perception cache module of a cache file system in a training node when it perceives that a target training file required by a large model to be trained under a target service is not cached; the target service is one or a combination of a search service, an advertising service, and a recommendation service; the target training file is determined by: determining a target cache directory according to a preset time granularity, and determining the training file corresponding to the target cache directory as the target training file; The cache system pulls and caches the historical business data in the target business according to the preset time granularity to obtain the target training file; the training node is used to train the large model to be trained using the cached target training file, and after determining that the target training file is trained, the following steps are performed: according to the cache directory of the trained target training file and the preset time granularity, the next cache directory is determined, and the training file corresponding to the next cache directory is determined as the target training file, until all target training files are trained.

7. A training node, characterized in that: include: Cache file system and training container; The cache file system is used to determine the target training file required for the large model to be trained under the target business, including: determining a target cache directory according to a preset time granularity, and determining the training file corresponding to the target cache directory as the target training file; wherein the target business is one or a combination of a search business, an advertising business, and a recommendation business; The cache file system is further configured to trigger the cache system to cache the target training file according to a preset time granularity when the perception cache module of the cache file system perceives that the target training file is not cached; The training container is used to train the large model to be trained using the cached target training file; The cache file system is also used to determine that after the target training file is trained, it returns to the step of determining the target training file required for the large model to be trained under the target business, including: determining the next cache directory based on the cache directory of the trained target training file and the preset time granularity, and determining the training file corresponding to the next cache directory as the target training file until all target training files are trained.

8. A cache system, characterized in that: include: Control node and multiple storage nodes; The control node is configured to receive a cache request; The cache request is sent when a perception cache module of a cache file system in a training node perceives that a target training file required by a large model to be trained under a target service is not cached; the target service is one or a combination of a search service, an advertising service, and a recommendation service; the target training file is determined by: determining a target cache directory according to a preset time granularity, and determining the training file corresponding to the target cache directory as the target training file; The control node is further configured to pull historical business data of the target business according to a preset time granularity and cache the data in the plurality of storage nodes to obtain the target training file; The training node is used to train the large model to be trained using the cached target training file. After determining that the target training file has been trained, the following steps are performed: based on the cache directory of the trained target training file and the preset time granularity, the next cache directory is determined, and the training file corresponding to the next cache directory is determined as the target training file, until all target training files are trained.

9. A large business model offline training system, characterized by: It includes multiple training nodes, a cache system and a remote storage system; the cache system is communicatively connected to the training nodes and the remote storage system respectively; each of the training nodes is used to execute the offline training method for the large business model as described in any one of claims 1 to 5, and the cache system is used to execute the offline training method for the large business model as described in claim 6.

Images