Service cold start method, device and equipment

By partitioning the processing of historical log data in the Kafka message queue, the problems of low cold start efficiency and high cost in the existing technology are solved, efficient and low-cost data processing is achieved, and large-scale data needs are met.

CN115016860BActive Publication Date: 2024-11-12ANT BLOCKCHAIN TECHNOLOGY (SHANGHAI) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202210636538.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-06-07
Publication Date
2024-11-12
Estimated Expiration
2042-06-07

AI Technical Summary

Technical Problem

In the cold start process of business, especially in big data offline computing platforms and business systems, the traceability computing is inefficient and costly, and cannot meet the average daily data processing needs of tens of millions.

Method used

By determining the partition of each trace node in the Kafka message queue and saving the target historical log data into each partition, the trace nodes are used to process the data in parallel, avoiding duplicate processing and omission problems and improving processing efficiency.

Benefits of technology

It has achieved low-cost and high-efficiency cold start of business, meeting the processing needs of tens of millions of daily data or even more, and improving cold start efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN115016860B_ABST
    Figure CN115016860B_ABST
Patent Text Reader

Abstract

The embodiments of this specification provide a method, apparatus and equipment for cold starting a business, wherein the method comprises: determining the metadata of the target cold start task according to an acquired processing request of the target cold start task; wherein the metadata comprises the partition corresponding to each backtracking node in a Kafka message queue among a plurality of backtracking nodes; obtaining the target historical log data of the first business corresponding to the target cold start task according to the processing request; saving the target historical log data evenly in each partition; wherein the target historical log data in each partition is used for the backtracking node corresponding to the partition to determine the indicator data of the cumulative indicator of the second business having a dependency relationship with the first business; and performing cold start processing on the second business according to the indicator data.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This document relates to the field of data processing technology, and in particular to a method, device and equipment for cold starting a business. Background Art

[0002] The cold start of a business is the process of accumulating the target data that a new business relies on from scratch and reaching the available state of the new business. At present, in order to improve the cold start efficiency of the business, historical event data is usually imported into a big data offline computing platform such as MapReduce, and the big data offline computing platform performs backtracking calculations based on the imported historical event data to obtain the target data. Alternatively, the historical event data is imported into its corresponding business system to perform backtracking calculations using the original business logic to obtain the target data. However, in the former, due to the limited data processing capabilities of the big data offline computing platform, it is often unable to handle complex backtracking calculation logic, so the scope of application is limited and the cost is high. In the latter, when the historical event data reaches an average of tens of millions or even more per day, the backtracking calculation takes a long time, so it cannot meet the cold start requirements of the business. Summary of the invention

[0003] One or more embodiments of the present specification provide a cold start method for a service. The method includes determining the meta information of the target cold start task according to the acquired processing request of the target cold start task. The meta information includes the partition corresponding to each backtracking node in the Kafka message queue among multiple backtracking nodes. The target historical log data of the first service corresponding to the target cold start task is obtained according to the processing request. The target historical log data is evenly saved in each of the partitions. The target historical log data in each of the partitions is used for the backtracking node corresponding to the partition to determine the indicator data of the cumulative indicator of the second service that has a dependency relationship with the first service. Cold start processing is performed on the second service according to the indicator data.

[0004] One or more embodiments of the present specification provide a cold start device for a service. The device includes a determination module, which determines the metadata of the target cold start task according to the acquired processing request of the target cold start task. The metadata includes the partition corresponding to each backtracking node in the Kafka message queue among multiple backtracking nodes. The device also includes an acquisition module, which acquires the target historical log data of the first service corresponding to the target cold start task according to the processing request. The device also includes a saving module, which saves the target historical log data evenly to each of the partitions. The target historical log data in each of the partitions is used for the backtracking node corresponding to the partition to determine the indicator data of the cumulative indicator of the second service that has a dependency relationship with the first service. The device also includes a processing module, which performs cold start processing on the second service according to the indicator data.

[0005] One or more embodiments of the present specification provide a cold start device for a business. The device includes a processor. The device also includes a memory arranged to store computer executable instructions. When the computer executable instructions are executed, the processor determines the meta information of the target cold start task according to the processing request of the target cold start task obtained. Wherein, the meta information includes the partition corresponding to each backtracking node in the Kafka message queue among multiple backtracking nodes. According to the processing request, the target historical log data of the first business corresponding to the target cold start task is obtained. The target historical log data is evenly saved in each of the partitions. Wherein, the target historical log data in each of the partitions is used for the backtracking node corresponding to the partition to determine the indicator data of the cumulative indicator of the second business that has a dependency relationship with the first business. Cold start processing is performed on the second business according to the indicator data.

[0006] One or more embodiments of the present specification provide a storage medium. The storage medium is used to store computer executable instructions. When the computer executable instructions are executed by the processor, the meta information of the target cold start task is determined according to the processing request of the target cold start task obtained. The meta information includes the partition corresponding to each backtracking node in the Kafka message queue among multiple backtracking nodes. The target historical log data of the first business corresponding to the target cold start task is obtained according to the processing request. The target historical log data is evenly saved in each of the partitions. The target historical log data in each of the partitions is used for the backtracking node corresponding to the partition to determine the indicator data of the cumulative indicator of the second business that has a dependency relationship with the first business. Cold start processing is performed on the second business according to the indicator data. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] In order to more clearly illustrate one or more embodiments of this specification or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0008] Figure 1 A schematic diagram of a first scenario of a cold start method for a service provided in one or more embodiments of this specification;

[0009] Figure 2 A schematic diagram of a second scenario of a cold start method for a service provided in one or more embodiments of this specification;

[0010] Figure 3 A third scenario diagram of a cold start method for a service provided in one or more embodiments of this specification

[0011] Figure 4 A first flow chart of a cold start method for a service provided in one or more embodiments of this specification;

[0012] Figure 5 A second flow chart of a cold start method for a service provided in one or more embodiments of this specification;

[0013] Figure 6 A third flow chart of a cold start method for a service provided in one or more embodiments of this specification;

[0014] Figure 7 A schematic diagram of the module composition of a cold start device for a service provided in one or more embodiments of this specification;

[0015] Figure 8 A schematic diagram of the structure of a cold start device for a service provided in one or more embodiments of this specification. DETAILED DESCRIPTION

[0016] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of this specification, the following will be combined with the drawings in one or more embodiments of this specification to clearly and completely describe the technical solutions in one or more embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this specification, not all of the embodiments. Based on one or more embodiments of this specification, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of this document.

[0017] Figure 1A schematic diagram of an application scenario of a cold start method for a service provided in one or more embodiments of this specification, such as Figure 1 As shown, the scenario includes: a cold start processing node, multiple backtracking nodes and a service system providing a second service; wherein the second service is a service to be cold started; each node can be a terminal device or a server; the terminal device can be a mobile phone, a tablet computer, a desktop computer, a portable notebook computer, etc.; the server can be an independent server ( Figure 1 Only a server is shown in the figure), or it may be a server cluster composed of multiple servers. The service system of the second service may provide the first service or may not provide the first service.

[0018] Specifically, when the cold start processing node obtains the processing request of the target cold start task, the meta information of the target cold start task is determined; the meta information includes the partition corresponding to each backtracking node in the Kafka message queue; the cold start processing node sends a backtracking request to each backtracking node according to the determined meta information, and obtains the target historical log data of the first business corresponding to the target cold start task according to the processing request of the target cold start task, and evenly saves the target historical log data to the partition corresponding to each backtracking node. Each backtracking node determines its own corresponding partition in the Kafka message queue according to the received backtracking request, and determines the indicator data of the cumulative indicator of the second business that has a dependency relationship with the first business according to the target historical log data in the corresponding partition, and provides the indicator data to the business system. The business system cold starts the second business according to the obtained indicator data.

[0019] Among them, the cold start processing node can be installed with a pre-processing application and a log extraction application, and obtain the processing request of the target cold start task through the pre-processing application to determine the metadata of the target cold start task; and obtain the target historical log data of the first business corresponding to the target cold start task through the log extraction application, and save the target historical log data evenly to the partitions corresponding to each backtracking node; or, the cold start processing node can start a log extraction thread, obtain the target historical log data of the first business corresponding to the target cold start task through the log extraction thread, and save the target historical log data evenly to the partitions corresponding to each backtracking node.

[0020] Optionally, providing the indicator data to the business system may include: each backtracking node sends the determined indicator data to the cold start processing node, and the cold start processing node uploads the received indicator data to the business system of the second business according to a preset upload interface; or, each backtracking node sends the determined indicator data to the cold start processing node, and the cold start processing node saves the received indicator data to a designated storage location, and the business system of the second business obtains the indicator data from the storage location; or, each backtracking node saves the determined indicator data to a designated storage location, and the business system of the second business obtains the indicator data from the storage location. The designated storage location may be a storage area of ​​the business system of the second business, or may be another storage area.

[0021] In one or more embodiments of this specification, the above-mentioned cold start node can also be replaced by a pre-processing node and a log extraction node. Specifically, Figure 2 A schematic diagram of an application scenario of another service cold start method provided in one or more embodiments of this specification, such as Figure 2 As shown in FIG. 1 , the scenario includes: a pre-processing node, a log extraction node, multiple backtracking nodes, and a business system that provides a second business. Each node can be a terminal device or a server; the terminal device can be a mobile phone, a tablet computer, a desktop computer, a portable notebook computer, etc.; the server can be an independent server ( Figure 2 Only a server is shown in the figure), or it may be a server cluster composed of multiple servers. The business system may provide the first business, or may not provide the first business.

[0022] When the front processing node obtains the processing request of the target cold start task, the meta information of the target cold start task is determined; the meta information includes the partition corresponding to each backtracking node in the Kafka message queue; the front processing node sends a backtracking request to each backtracking node according to the determined meta information, and sends an event extraction request to the log extraction node according to the processing request and meta information of the target cold start task. The log extraction node obtains the target historical log data of the first business corresponding to the target cold start task according to the received event extraction request, and evenly saves the target historical log data to the partition corresponding to each backtracking node. Each backtracking node determines its own corresponding partition in the Kafka message queue according to the received backtracking request, and determines the indicator data of the cumulative indicator of the second business that has a dependency relationship with the first business according to the target historical log data in the corresponding partition, and provides the indicator data to the business system. The business system cold starts the second business according to the obtained indicator data.

[0023] Among them, a pre-processing application can be installed in the pre-processing node, and the processing request of the target cold start task and the meta information of the target cold start task can be obtained through the pre-processing application. A log extraction application can be installed in the log extraction node, and the target historical log data of the first business corresponding to the target cold start task can be obtained through the log extraction application, and the target historical log data is evenly saved in the partitions corresponding to each backtracking node; or, the log extraction node starts a log extraction thread, obtains the target historical log data of the first business corresponding to the target cold start task through the log extraction thread, and the target historical log data is evenly saved in the partitions corresponding to each backtracking node.

[0024] Optionally, the above-mentioned providing of the indicator data to the business system may include: each backtracking node sends the determined indicator data to the front-processing node, and the front-processing node uploads the received indicator data to the business system of the second business according to a preset upload interface; or, each backtracking node sends the determined indicator data to the front-processing node, the front-processing node saves the received indicator data to a designated storage location, and the business system of the second business obtains the indicator data from the storage location; or, each backtracking node saves the determined indicator data to the designated storage location, and the business system of the second business obtains the indicator data from the storage location.

[0025] In one or more embodiments of this specification, the cold start processing node mentioned above may also be the same node as a certain backtracking node; or, the pre-processing node and log extraction node mentioned above may also be the same node as a certain backtracking node. Figure 3 A schematic diagram of an application scenario of a cold start method for another service provided in one or more embodiments of this specification, such as Figure 3 As shown, the scenario includes: multiple backtracking nodes and a service system that provides a second service. The backtracking node can be a terminal device or a server; the terminal device can be a mobile phone, a tablet computer, a desktop computer, a portable notebook computer, etc.; the server can be an independent server ( Figure 3 Only a server is shown in the figure), or it may be a server cluster composed of multiple servers. The business system may provide the first business, or may not provide the first business.

[0026] A certain backtracking node can be designated as the main backtracking node in multiple backtracking nodes in advance, and other backtracking nodes can be determined as sub-backtracking nodes. When the main backtracking node obtains the processing request of the target cold start task, the meta information of the target cold start task is determined; the meta information includes the partition corresponding to each backtracking node (including the main backtracking node itself) in the Kafka message queue; the main backtracking node sends a backtracking request to each sub-backtracking node according to the determined meta information, and obtains the target historical log data of the first business corresponding to the target cold start task according to the processing request of the target cold start task and the determined meta information, and saves the target historical log data in a balanced manner to the partition corresponding to each backtracking node. Each sub-backtracking node determines its own corresponding partition in the Kafka message queue according to the received backtracking request, and the main backtracking node and each sub-backtracking node determine the index data of the cumulative index of the second business having a dependency relationship with the first business according to the target historical log data in the corresponding partition, and provide the index data to the business system. The business system cold starts the second business according to the obtained index data.

[0027] Among them, the main backtracking node can be installed with a pre-processing application, a log extraction application and a backtracking application, and the processing request of the target cold start task and the meta-information of the target cold start task are obtained through the pre-processing application. The target historical log data of the first business corresponding to the target cold start task is obtained through the log extraction application, and the target historical log data is evenly saved in the partitions corresponding to each backtracking node. The backtracking application determines the indicator data of the cumulative indicator of the second business that has a dependency relationship with the first business based on the target historical log data in the corresponding partition, and provides the indicator data to the business system. Alternatively, the main backtracking node starts a log extraction thread, extracts the target historical log data of the first business corresponding to the target cold start task through the log extraction thread, and saves the target historical log data in the partitions corresponding to each backtracking node.

[0028] Optionally, the above-mentioned providing of the indicator data to the business system may include: each sub-backtracing node sends the determined indicator data to the main backtracing node, and the main backtracing node uploads the indicator data determined by itself and the indicator data sent by each sub-backtracing node to the business system of the second business according to a preset upload interface; or, each sub-backtracing node sends the determined indicator data to the main backtracing node, the main backtracing node saves the indicator data determined by itself and the indicator data sent by each sub-backtracing node to a designated storage location, and the business system of the second business obtains the indicator data from the storage location; or, the main backtracing node and each sub-backtracing node save the indicator data determined by themselves to a designated storage location, and the business system of the second business obtains the indicator data from the storage location.

[0029] Based on the above application scenarios, it can be seen that in the embodiments of the present specification, by using the Kafka message queue to store the target historical log data of the first business, and determining the partition corresponding to each backtracking node in the Kafka message queue, each backtracking node can parallelly process the target historical log data in its corresponding partition in an orderly manner. This not only avoids the problem of the same target historical log data being repeatedly processed by multiple backtracking nodes or the omission of processing of a certain target historical log data, but also greatly improves the processing efficiency of the target historical log data, and can meet the daily processing requirements of tens of millions or even more log data, and thus can meet the cold start requirements of the business, achieving low-cost and high-efficiency business cold start.

[0030] Based on the above application scenario architecture, one or more embodiments of this specification provide a cold start method for a service. Figure 4 A flowchart of a cold start method for a service provided in one or more embodiments of this specification is shown as follows: Figure 4 As shown, the method comprises the following steps:

[0031] Step S102, determining the meta information of the target cold start task according to the acquired processing request of the target cold start task; wherein the meta information includes the partition corresponding to each backtracking node in the Kafka message queue of the multiple backtracking nodes;

[0032] In an embodiment of the present specification, the target cold start task may be a task of cold starting a second business that has a dependency on the first business based on the historical log data of the first business. Among them, the first business and the second business can be set as needed in actual applications. As an example, the first business is an account login business, and the second business is a risk control business based on the number of logins. For example, the risk control strategy of the risk control business is that if the number of logins of an account in the last 30 days is greater than 300 times, then the account is determined to be a risk account, etc. As another example, the first business is a transfer business, and the second business is a risk control business based on the number of transfers and the total amount of transfers. For example, the risk control strategy of the risk control business is that if the number of transfers of a personal financial account in the last 10 days is greater than 100 times and the total amount of transfers is greater than 10 million, then the personal financial account is determined to be a risk account, etc.

[0033] When step S102 is executed by the aforementioned cold processing node, the user can directly operate the cold processing node to edit the screening parameters of the target historical log data and initiate the target cold start task. The user can also operate the business system of the second business to edit the screening parameters of the target historical log data to initiate the target cold start task, and the business system sends a processing request for the target cold start task to the cold start node based on the screening parameters edited by the user. Accordingly, step S102 may include: in response to the user's submission operation, obtaining a processing request for the target cold start task; determining the meta information of the target cold start task according to the obtained processing request; or, receiving a processing request for the target cold start task sent by the business system of the second business, and determining the meta information of the target cold start task according to the received processing request.

[0034] When step S102 is executed by the aforementioned pre-processing node, the user can directly operate the pre-processing node to edit the screening parameters of the target historical log data and initiate the target cold start task. The user can also operate the business system of the second business to edit the screening parameters of the target historical log data to initiate the target cold start task, and the business system sends a processing request for the target cold start task to the aforementioned processing node based on the screening parameters edited by the user. Accordingly, step S102 may include: in response to the user's submission operation, obtaining a processing request for the target cold start task; determining the meta information of the target cold start task based on the obtained processing request; or, receiving a processing request for the target cold start task sent by the business system of the second business, and determining the meta information of the target cold start task based on the received processing request.

[0035] When step S102 is executed by the aforementioned main backtracking node, the user can directly operate the main backtracking node to edit the screening parameters of the target historical log data and initiate the target cold start task. The user can also operate the business system of the second business to edit the screening parameters of the target historical log data to initiate the target cold start task, and the business system sends a processing request for the target cold start task to the main backtracking node based on the screening parameters edited by the user. Accordingly, step S102 may include: in response to the user's submission operation, obtaining a processing request for the target cold start task; determining the meta information of the target cold start task based on the obtained processing request; or, receiving a processing request for the target cold start task sent by the business system of the second business, and determining the meta information of the target cold start task based on the received processing request.

[0036] The screening parameters may vary with the first business. As an example, the first business is a login business, and the screening parameters may include screening time, account information of the login account to be risk-controlled, and the like.

[0037] Step S104, obtaining target historical log data of the first business corresponding to the target cold start task according to the processing request;

[0038] Specifically, when step S104 is executed by the aforementioned cold start node or main backtracking node, step S104 may include: obtaining a filter parameter from a processing request, and obtaining the target historical log data of the first business corresponding to the target cold start task from the designated log storage area according to the filter parameter. When step S104 is executed by the aforementioned log extraction node, the pre-processing node may obtain the filter parameter from the obtained processing request of the target backtracking task, and send a log extraction request to the log extraction node according to the filter parameter, the determined meta information, the determined location information of the designated log storage area, etc.; accordingly, step S104 may include: obtaining the filter parameter and location information from the log extraction request, and obtaining the target historical log data of the first business corresponding to the target cold start task from the designated log storage area corresponding to the location information according to the filter parameter. Among them, the location information of the specified log storage area can be included in the processing request of the target backtracing task; the location information of the specified log storage area can also be pre-stored in the cold start node or the main backtracing node or the front-processing node in correspondence with the system identifier of the business system of the second business. Accordingly, the processing request of the target backtracing task can include the system identifier of the business system of the second business, and the cold start node or the main backtracing node or the front-processing node obtains the corresponding saved location information according to the system identifier.

[0039] The designated log storage area may be a storage area of ​​a business system of a second business. When the business system of the second business also provides the first business, the business system saves the data related to the second business in the business data of the first business as log data of the first business to the designated log storage area. When the business system of the second business does not provide the first business, and the first business is provided by other business systems, the other business systems use the data related to the second business in the business data of the first business as log data of the first business, and send the log data to the business system of the second business, and the business system of the second business saves the received log data to the designated log storage area.

[0040] Step S106, the target historical log data is evenly saved in each partition; wherein the target historical log data in each partition is used for the backtracking node corresponding to the partition to determine the index data of the cumulative index of the second business having a dependency relationship with the first business;

[0041] Specifically, according to the determined meta-information and the preset balancing rules, the target historical log data is evenly saved in each partition. It can be understood that the target historical log data in each partition is part of the target historical log data required for the cold start process, and the combination of the target historical log data in each partition is all the target historical log data required for the cold start process. In one embodiment, the balancing rule can adopt a polling strategy, that is, each target historical log data is allocated to each partition in turn in sequence. For example, if there are three partitions, the first target historical log data is saved in the first partition, the second target historical log data is saved in the second partition, the third target historical log data is saved in the third partition, and the fourth target historical log data is saved in the first partition, and so on, until all target historical log data are saved in the partition. It should be pointed out that the balancing rule is not limited to the aforementioned rule adopting the polling strategy, which can be set by itself according to needs in actual applications, and this specification does not make specific restrictions on this.

[0042] Furthermore, each backtracking node processes the target log data in its corresponding partition according to a preset processing method to obtain indicator data of the corresponding cumulative indicator of the second business; wherein, the preset processing method can be set by itself as needed in actual application, and this specification does not make specific limitations on this.

[0043] In the embodiments of the present specification, each backtracking node can only process the target historical log data in the partition corresponding to itself, thereby ensuring the orderliness and effectiveness of the target historical log data during the processing, and avoiding the problem that the same target historical log data is repeatedly processed by multiple backtracking nodes; at the same time, each backtracking node can process the target historical log data in the partition corresponding to itself in parallel, which greatly improves the processing efficiency and thereby improves the cold start efficiency of the second business.

[0044] Step S108: cold start the second service according to the indicator data.

[0045] Optionally, the indicator data is saved to a designated storage location so that the business system of the second business obtains the indicator data from the designated storage location and performs a cold start process on the second business according to the obtained indicator data; or, the indicator data is uploaded to the business system of the second business so that the business system performs a cold start process on the second business according to the uploaded indicator data. For different application scenarios, the specific implementation of step S108 can be found in the relevant description above, and the repeated parts will not be repeated here.

[0046] In one or more embodiments of the present specification, according to the processing request of the target cold start task obtained, the meta information of the target cold start task is determined; the meta information includes the partition corresponding to each backtracking node in the Kafka message queue among the multiple backtracking nodes; the target historical log data of the first business corresponding to the target cold start task is obtained according to the processing request and saved in each partition; wherein the target historical log data in each partition is used for the backtracking node corresponding to the partition, and the index data of the cumulative index of the second business having a dependency relationship with the first business is determined; the second business is cold started according to the index data. Thus, by using the kafka message queue to store the target historical log data of the first business, and determining the partition corresponding to each backtracking node in the Kafka message queue, each backtracking node can process the target historical log data in the corresponding partition in parallel in an orderly manner, which can not only avoid the problem that the same target historical log data is repeatedly processed by multiple backtracking nodes or the omission of a certain target historical log data, but also greatly improve the processing efficiency of the target historical log data, which can meet the processing requirements of the daily average of tens of millions or even more log data, thereby meeting the cold start requirements of the business, and realizing low-cost and high-efficiency business cold start.

[0047] In order to ensure that the target historical log data can be processed in an orderly manner and each backtracking node can be effectively managed, in one or more embodiments of this specification, Figure 5 As shown, step S102 may include the following steps S102-2 to S102-10:

[0048] Step S102-2, obtaining a processing request for a target cold start task;

[0049] Step S102-4, creating a first number of topics for the target cold start task according to the total number of backtracking nodes and the preset concurrency configuration parameters, and dividing each backtracking node into a first number of backtracking node groups; wherein the backtracking node groups correspond to the topics one by one;

[0050] Among them, the concurrency configuration parameter is the maximum number of backtracking nodes that can run simultaneously. Specifically, the total number of backtracking nodes is divided by the preset concurrency configuration parameter to determine whether the calculation result is an integer. If so, the calculation result is determined as the first number; if not, the calculation result is rounded down and the rounded result is determined as the first number; a first number of topics are created for the target cold start task, and each backtracking node is divided into a first number of backtracking node groups. Among them, dividing each backtracking node into a first number of backtracking node groups can be to evenly divide each backtracking node into a first number of backtracking node groups; in particular, when the total number of backtracking nodes is not divisible by the first number, the backtracking nodes corresponding to the indivisible part can be randomly assigned to the backtracking node groups corresponding to the divisible part. It should be pointed out that for a certain backtracking node, it only belongs to one backtracking node group, and cannot belong to multiple backtracking node groups at the same time.

[0051] As an example, the total number of backtracking nodes is 20, and the preset concurrency configuration parameter is 5, then the first number is 20 / 5=4; 4 topics are created for the target cold start task, and the 20 backtracking nodes are evenly divided into 4 backtracking node groups, each backtracking node group includes 5 backtracking nodes.

[0052] As another example, the total number of backtracking nodes is 32, which are recorded as backtracking node 1, backtracking node 2, backtracking node 3...backtracking node 32 in sequence. The preset concurrency configuration parameter is 5, and the backtracking node groups are divided according to the numbering order of the backtracking nodes; then 32 / 5=6.4, and the first number obtained by rounding down 6.4 is 6; 6 topics are created for the target cold start task; and the 32 backtracking nodes are evenly divided into 6 backtracking node groups, each backtracking node group includes 5 backtracking nodes; the 6 backtracking node groups are recorded as backtracking node group 1 (including backtracking nodes 1 to backtracking node 5), backtracking node group 2 (including backtracking nodes 6 to backtracking node 10), backtracking node group 3 ( Including backtracking node 11 to backtracking node 15)…backtracking node group 6 (including backtracking node 26 to backtracking node 30); at this time there are still two backtracking nodes left, namely backtracking node 31 and backtracking node 32, then backtracking node 31 can be randomly assigned to any one of the six backtracking node groups, such as divided into backtracking node group 1, and backtracking node 32 can be randomly assigned to any one of the backtracking node groups from backtracking node group 2 to backtracking node group 6, such as divided into backtracking node group 3; that is, the number of backtracking nodes in backtracking node group 1 and backtracking node group 3 is 6, and the number of backtracking nodes in backtracking node group 2, backtracking node group 4 to backtracking node group 6 is 5.

[0053] Step S102-6, allocating a second number of partitions to each topic in the Kafka message queue; wherein the second number is not less than a third number of backtracking nodes in the backtracking node group;

[0054] In order to improve the processing efficiency of the target historical log data, and to ensure that the target historical log data can be processed in order, and will not be processed by multiple backtracking nodes at the same time, in one or more embodiments of this specification, a second number of partitions is allocated to each topic in the Kafka message queue, and the second number is not less than the third number of backtracking nodes in the backtracking node group. In particular, when the number of backtracking nodes in the backtracking node group is not unique, the maximum number of backtracking nodes can be determined as the third number. For example, in another aforementioned example, 6 is determined as the third number.

[0055] Step S102-8, performing allocation processing on the partitions of each topic to obtain at least one partition corresponding to each backtracking node in the backtracking node group corresponding to each topic;

[0056] Preferably, the partitions of each topic are evenly distributed to each backtracing node in the backtracing node group corresponding to the topic; when the number of partitions is not divisible by the number of backtracing nodes in the backtracing node group, the quotient of the division is first determined as the number of partitions evenly distributed to each backtracing node, and then the remainder of the division is determined as the fourth number of remaining partitions, and the fourth number of third target backtracing nodes is randomly selected from the backtracing nodes included in the corresponding backtracing node group, and the remaining partitions are evenly distributed to the selected third target backtracing nodes.

[0057] Following the above example, the number of partitions for each topic is 6. For backtracking node group 1 and backtracking node group 3, after the partitions of the corresponding topics are allocated, each backtracking node corresponds to a partition. For backtracking node group 2, backtracking node group 4 to backtracking node group 6, backtracking node group 2 is used as an example for explanation. Since the number of partitions of the corresponding topic 6 and the number of backtracking nodes 5 cannot be divided evenly, the quotient is 1 and the remainder is also 1. First, a partition is allocated to backtracking nodes 6 to 10 in backtracking node group 2 respectively; then the remainder 1 is determined as the fourth number of remaining partitions, and a third target backtracking node is randomly selected from backtracking nodes 6 to 10. For example, if the target backtracking node is backtracking node 8, 1 remaining partition is allocated to the third target backtracking node 8; that is, in backtracking node group 2, backtracking node 8 corresponds to two partitions, backtracking node 6 corresponds to 1 partition, backtracking node 7 corresponds to 1 partition, backtracking node 9 corresponds to 1 partition, and backtracking node 10 corresponds to 1 partition.

[0058] Step S102-10, generating meta information of the target cold start task according to the task identifier of the target cold start task, the first number of topics, the first number of backtracking node groups, and the partitions corresponding to each backtracking node.

[0059] Optionally, determine the task identifier of the target cold start task, the topic identifier of each topic in the first number of topics corresponding to the target cold start task, the group identifier of each backtracking node group in the first number of backtracking node groups, the node identifier of each backtracking node, and the partition identifier of the partition corresponding to each backtracking node, record the task identifier and the topic identifier, record each topic identifier and the group identifier of the corresponding backtracking node group, record the group identifier of each backtracking node group and the node identifier of each backtracking node corresponding to it, and record the node identifier of each backtracking node and the partition identifier of its corresponding distinction; determine the recorded information as static information of the target cold start task, and generate meta information based on the static information. In other words, the meta information includes the static information of the target cold start task. Among them, the task identifier can be included in the processing request of the target backtracking task, or it can be determined by the execution subject of step S102 according to preset rules; the specific determination method of the topic identifier, group identifier, node identifier, and partition identifier is not specifically limited in this application.

[0060] Therefore, by determining the meta information of the target backtracking task, the target historical log data can be stored in order according to the meta information in the subsequent process, and each backtracking node can be guaranteed to process the target historical log data in order.

[0061] Considering that the target historical log data may have a large amount of data, acquiring all of it at once may cause data loss due to network and other factors. Based on this, in one or more embodiments of the present application, the target historical log data can be acquired in batches. Specifically, Figure 6 As shown, step S104 may include the following steps S104-2 and S104-4:

[0062] Step S104-2, obtaining screening parameters from the processing request;

[0063] Step S104-4, if it is determined that the target historical log data matching the screening parameter in the historical log data of the first business corresponding to the target backtracking task meets the preset batch acquisition condition, then partially acquiring the target historical log data from the target historical log data in sequence;

[0064] Specifically, determine whether the total amount of target historical log data that matches the screening parameters in the historical log data of the first business corresponding to the target backtracking task is greater than a preset data volume threshold. If so, determine that it meets the preset batch acquisition conditions, and obtain part of the target historical log data from the target historical log data in sequence.

[0065] Further, if it is determined that the preset condition for obtaining in batches is not met, all target historical log data are obtained from the target historical log data at one time.

[0066] Corresponding to the above steps S104-2 and S104-4, Figure 6 As shown, step S106 may include the following steps S106-2 and S106-4:

[0067] Step S106-2, determining, according to the meta information and the preset balancing rule, a first target partition in each partition for storing the partial target historical log data currently acquired;

[0068] Specifically, according to the task identifier of the target cold start task, query the meta-information of the target cold start task; obtain a first number of topics from the queried meta-information; determine a first target topic of the log data to be allocated in the first number of topics according to a preset balancing rule; obtain a second number of candidate partitions corresponding to the first target topic from the meta-information; determine a first target partition of the log data to be allocated in the candidate partitions according to the balancing rule; determine the first target partition as the first target partition for storing part of the target historical log data currently acquired.

[0069] Step S106-4, saving part of the target historical log data currently acquired to the first target partition; wherein the part of the target historical log data in the first target partition is used for the corresponding backtracking node to determine the indicator data of the cumulative indicator of the second business having a dependency relationship with the first business.

[0070] In one or more embodiments of the present specification, when the balancing rule adopts a polling strategy, step S106-4 may also include associating and recording the preset save identifier with the partition identifier of the current first target partition. Correspondingly, in step S106-2, according to the preset balancing rule, the topic identifier associated with the partition identifier associated with the preset save identifier is determined as the target topic identifier, and the topic corresponding to the target topic identifier is determined as the first target topic of the log data to be allocated; and according to the balancing rule, the next partition identifier of the partition identifier associated with the preset save identifier in the second number of candidate partitions is determined as the target partition identifier, and the candidate partition corresponding to the target partition identifier is determined as the first target partition of the log data to be allocated.

[0071] Therefore, when the target historical log data meets the preset conditions for obtaining the data in batches, the target historical log data is obtained in batches, thereby avoiding problems such as data loss due to excessive data volume and network factors.

[0072] In order to effectively manage the data processing progress of each backtracking node, in one or more embodiments of this specification, the method may further include:

[0073] Receive the current displacement parameters sent by each backtracking node; update the meta-information of the target backtracking task according to the current displacement parameters.

[0074] Among them, the current displacement parameter represents the position of the last target historical log data currently processed by the backtracking node in the partition corresponding to the backtracking node. According to the current displacement parameter, updating the meta-information of the target backtracking task may include: associating the current displacement parameter with the node identifier of the backtracking node, and determining the recorded information as the dynamic information of the target backtracking task, and saving the dynamic information to the meta-information of the target backtracking task. It is understandable that the dynamic information of the target backtracking task included in the meta-information will change over time.

[0075] Furthermore, considering that in actual applications, the backtracking node may be down due to various factors, in order to avoid the target historical log data in the partition corresponding to the backtracking node in the downtime state from being unable to be processed, one or more embodiments of this specification can also be based on the rebalancing mechanism, according to the dynamic information in the meta-information, to perform partition redistribution processing on the backtracking node group where the backtracking node in the downtime state is located. Specifically, the method can also include the following steps A2 to A8:

[0076] Step A2: if it is determined that there is a first target backtracking node that is in a downtime state within a preset time period, the topic corresponding to the target backtracking node group where the first target backtracking node is located is determined as the second target topic;

[0077] In order to facilitate the management of the status of each backtracking node, in one or more embodiments of the present application, Figure 1 Taking the application scenario shown as an example, each backtracking node sends heartbeat data to the cold start processing node at a preset time interval; the cold start processing node determines whether there is a first target backtracking node that is in a downtime state within a preset time period based on the received heartbeat data. Specifically, the method also includes:

[0078] Receive the heartbeat data sent by each backtracking node, and record the sending time of the last heartbeat data of each backtracking node; and determine the recorded sending time as the starting time point of the preset duration, and determine whether there is a first target backtracking node that has not sent heartbeat data within the preset duration; if so, determine the first target backtracking node as the first target backtracking node in a downtime state. Wherein, the preset duration is greater than the preset time interval, for example, the preset time interval is 1 minute, the preset duration is 3 minutes, etc., which can be set as needed in actual applications.

[0079] Step A4, re-allocating the partitions of the second target topic to obtain at least one new partition corresponding to each remaining backtracking node in the target backtracking node group;

[0080] Among them, the process of allocating and processing the partitions of the second target topic is the same as the aforementioned process of allocating and processing the partitions. Please refer to the relevant description in the previous text, and the repeated parts will not be repeated here.

[0081] Step A6, acquiring the associated current displacement parameter from the meta information according to the partition identifier of each new partition;

[0082] Step A8, according to the partition identifier of the new partition and the current displacement parameter associated with the new partition, sends a partition update message to the remaining backtracking nodes corresponding to the new partition; wherein the partition update message is used for the corresponding remaining backtracking nodes to determine the target log data after the target log data corresponding to the current displacement parameter in the new partition corresponding to the partition identifier as the data to be processed, and determine the indicator data based on the data to be processed.

[0083] When the remaining backtracking nodes receive a partition update message, they obtain the partition identifier and current displacement parameter from the partition update message, and determine the partition corresponding to the partition identifier as a new partition. In the new partition, the target log data after the target log data corresponding to the current displacement parameter is determined as the data to be processed, and the indicator data is determined based on the data to be processed.

[0084] Therefore, when there is a first target backtracking node that is in a downtime state, the partitions of the corresponding second target topic are reallocated based on the rebalancing mechanism, which not only avoids the target historical log data in the partition corresponding to the first target backtracking node before the downtime from being unable to be processed, and realizes the automatic transfer of the target historical log data, but also has no impact on the backtracking nodes in other backtracking node groups, thereby ensuring the effective processing of each target historical log data.

[0085] Furthermore, considering that in actual applications, a backtracking node in a downtime state may automatically restart, in order to enable the restarted backtracking node to continue to participate in the processing of the target historical log data, thereby improving data processing efficiency, in one or more embodiments of the present application, the second partition corresponding to the restarted backtracking node is determined based on the breakpoint execution mechanism after downtime recovery. Specifically, the method also includes the following steps B2 to B6:

[0086] Step B2, if it is determined that there is a second target backtracking node that is restarted after downtime, determine the second target partition corresponding to the second target backtracking node;

[0087] Step B4, acquiring the associated current displacement parameter from the meta information according to the partition identifier of the second target partition;

[0088] Step B6, sending a data processing message to the second target backtracing node according to the partition identifier of the second target partition and the current displacement parameter associated with the second target partition; the data processing message is used for the second target backtracing node to determine the target log data after the target log data corresponding to the current displacement parameter in the second target partition corresponding to the partition identifier as the data to be processed, and determine the indicator data based on the data to be processed.

[0089] Specifically, when it is determined before the aforementioned step A8 that there is a second target backtracing node that is restarted after a downtime, and the second target backtracing node is the same backtracing node as the first target backtracing node that was determined for the last time, then the second target partition corresponding to the second target backtracing node is determined, which can be: according to the node identifier of the second target backtracing node, the associated partition identifier is obtained from the metadata of the target backtracing task, and the partition corresponding to the partition identifier is determined as the second target partition corresponding to the second target backtracing node; that is, the second target partition is the partition corresponding to the second target backtracing node before the downtime. If it is determined after the aforementioned step A8 that there is a second target backtracing node that is restarted after a downtime, then the second target partition corresponding to the second target backtracing node is determined, which can be: the partitions of the second target topic are reassigned and processed to obtain new partitions corresponding to the current backtracing nodes in the backtracing node group corresponding to the second target topic, and the new partitions include the second target partition corresponding to the second target backtracing node. And after obtaining the new partitions corresponding to the current backtracking nodes in the backtracking node group corresponding to the second target theme, the associated current displacement parameters are obtained from the metadata according to the partition identifier of each new partition; according to the partition identifier of the new partition and the current displacement parameter associated with the new partition, a data processing message is sent to the backtracking node corresponding to the new partition; after receiving the data processing message, the current backtracking nodes in the backtracking node group corresponding to the second target theme obtain the partition identifier and the current displacement parameter from the data processing message, and determine the partition corresponding to the partition identifier as the new partition, in which the target log data after the target log data corresponding to the current displacement parameter is determined as the data to be processed, and the indicator data is determined based on the data to be processed.

[0090] When the second target backtracing node is not the same backtracing node as the first target backtracing node determined for the last time, the second target partition corresponding to the second target backtracing node can be determined as follows: the topic corresponding to the backtracing node group where the second target backtracing node is located is determined as the third target topic, and the partitions of the third target topic are reassigned to obtain new partitions corresponding to the current backtracing nodes in the backtracing node group corresponding to the third target topic, and the new partitions include the second target partition corresponding to the second target backtracing node. And after obtaining the new partitions corresponding to the current backtracking nodes in the backtracking node group corresponding to the second target theme, the associated current displacement parameters are obtained from the metadata according to the partition identifier of each new partition; according to the partition identifier of the new partition and the current displacement parameter associated with the new partition, a data processing message is sent to the backtracking node corresponding to the new partition; after receiving the data processing message, the current backtracking nodes in the backtracking node group corresponding to the second target theme obtain the partition identifier and the current displacement parameter from the data processing message, and determine the partition corresponding to the partition identifier as the new partition, in which the target log data after the target log data corresponding to the current displacement parameter is determined as the data to be processed, and the indicator data is determined based on the data to be processed.

[0091] Therefore, when there is a second target backtracking node that is restarted after a crash, by determining the second target partition corresponding to the second target backtracking node, the second target backtracking node can re-process the target historical log data, thereby improving the processing efficiency of the target historical log data without affecting the backtracking node groups other than the backtracking node group where the second target backtracking node is located.

[0092] Furthermore, considering that in actual applications, multiple target backtracking tasks may be initiated at the same time, in order to avoid the accumulation of target backtracking tasks, in one or more embodiments of this specification, each backtracking node can start a corresponding backtracking thread for each target backtracking task, and process the target historical log data of the corresponding target backtracking task through the backtracking thread. Accordingly, after obtaining the processing request of the target cold start task, the method also includes the following steps C2 and C4:

[0093] Step C2, sending a thread start message to each backtracking node according to the determined task identifier of the target cold start task; wherein the thread start message is used for the backtracking node to start the backtracking thread corresponding to the target cold start task;

[0094] Step C4, receiving the task identifier sent by the backtracking node and the thread identifier of the started backtracking thread.

[0095] Corresponding to step C2 and step C4, the aforementioned division of each backtracing node into a first number of backtracing node groups may include: dividing each backtracing thread corresponding to the target backtracing task into a first number of backtracing thread groups; wherein the backtracing thread groups correspond one-to-one to the topics; the implementation method of this step is similar to the aforementioned implementation method of dividing each backtracing node into a first number of backtracing node groups, and reference may be made to the aforementioned related description.

[0096] Corresponding to step C2 and step C4, the aforementioned allocation processing of the partitions of each topic to obtain at least one partition corresponding to each backtracing node in the backtracing node group corresponding to each topic may include: allocating processing of the partitions of each topic to obtain at least one partition corresponding to each backtracing thread in the backtracing thread group corresponding to each topic; the implementation method of this step is similar to the aforementioned allocation processing of the partitions of each topic to obtain at least one partition corresponding to each backtracing node in the backtracing node group corresponding to each topic, and please refer to the aforementioned related description.

[0097] Corresponding to step C2 and step C4, the aforementioned generation of the meta-information of the target cold start task according to the task identifier of the target cold start task, the first number of topics, the first number of backtracing node groups, and the partitions corresponding to each backtracing node may include: generating the meta-information of the target cold start task according to the task identifier of the target cold start task, the first number of topics, the first number of backtracing thread groups, and the partitions corresponding to each backtracing thread. The implementation method of this step is similar to the aforementioned generation of the meta-information of the target cold start task according to the task identifier of the target cold start task, the first number of topics, the first number of backtracing node groups, and the partitions corresponding to each backtracing node, and reference may be made to the aforementioned related description.

[0098] Furthermore, corresponding to step C2 and step C4, the method also includes: if it is determined that the target cold start task is in a completed state, sending a thread closing message to the backtracking node; the thread closing message is used for the backtracking node to close the corresponding backtracking thread.

[0099] Specifically, if it is determined that all target historical log data has been acquired, and each backtracking thread has processed all target historical log data in the corresponding partition, it is determined that the target cold start task is in a completed state, and a thread closing message is sent to the backtracking node.

[0100] Therefore, for each target backtracking task, the corresponding backtracking thread is started to determine the indicator data through the backtracking thread, so that multiple target backtracking tasks can be executed at the same time, avoiding the accumulation of target backtracking tasks. At the same time, when it is determined that a target backtracking task is in a completed state, the corresponding backtracking thread is closed to achieve the timely release of computing resources in the backtracking node.

[0101] In one or more embodiments of the present specification, according to the processing request of the target cold start task obtained, the meta information of the target cold start task is determined; the meta information includes the partition corresponding to each backtracking node in the Kafka message queue among the multiple backtracking nodes; the target historical log data of the first business corresponding to the target cold start task is obtained according to the processing request and saved in each partition; wherein the target historical log data in each partition is used for the backtracking node corresponding to the partition, and the index data of the cumulative index of the second business having a dependency relationship with the first business is determined; the second business is cold started according to the index data. Thus, by using the kafka message queue to store the target historical log data of the first business, and determining the partition corresponding to each backtracking node in the Kafka message queue, each backtracking node can process the target historical log data in the corresponding partition in parallel in an orderly manner, which can not only avoid the problem that the same target historical log data is repeatedly processed by multiple backtracking nodes or the omission of a certain target historical log data, but also greatly improve the processing efficiency of the target historical log data, which can meet the processing requirements of the daily average of tens of millions or even more log data, thereby meeting the cold start requirements of the business, and realizing low-cost and high-efficiency business cold start.

[0102] Corresponding to the cold start method of the service described above, based on the same technical concept, one or more embodiments of this specification also provide a cold start device for the service. Figure 7 A schematic diagram of the module composition of a cold start device for a service provided in one or more embodiments of this specification, such as Figure 7 As shown, the device comprises:

[0103] Determining module 201, determining meta information of the target cold start task according to the acquired processing request of the target cold start task; wherein the meta information includes a partition corresponding to each backtracking node in a Kafka message queue among a plurality of backtracking nodes;

[0104] An acquisition module 202 acquires target historical log data of the first business corresponding to the target cold start task according to the processing request;

[0105] The saving module 203 saves the target historical log data evenly to each of the partitions; wherein the target historical log data in each partition is used for the backtracking node corresponding to the partition to determine the index data of the cumulative index of the second business having a dependency relationship with the first business;

[0106] The processing module 204 performs cold start processing on the second service according to the indicator data.

[0107] Optionally, the determination module 201 creates a first number of topics for the target cold start task according to the total number of the backtracking nodes and a preset concurrency configuration parameter, and divides the backtracking nodes into the first number of backtracking node groups; wherein the backtracking node groups correspond one-to-one to the topics; and,

[0108] Allocating a second number of partitions to each of the topics in the Kafka message queue; wherein the second number is not less than a third number of backtracking nodes in the backtracking node group;

[0109] Performing allocation processing on the partitions of each of the topics to obtain at least one partition corresponding to each backtracking node in the backtracking node group corresponding to the topic;

[0110] Meta information of the target cold start task is generated according to the task identifier of the target cold start task, the first number of topics, the first number of backtracking node groups, and the partition corresponding to each of the backtracking nodes.

[0111] Optionally, the device further includes: a receiving module;

[0112] The receiving module receives the current displacement parameter sent by each backtracking node; wherein the current displacement parameter represents the position of the last target historical log data currently processed by the backtracking node in the partition corresponding to the backtracking node; and updates the meta information according to the current displacement parameter.

[0113] Optionally, the meta information includes a node identifier of the backtracking node, a partition identifier of a partition corresponding to the backtracking node, and an association relationship between the current displacement parameter, and the device further includes: a rebalancing module;

[0114] The rebalancing module, if it is determined that there is a first target backtracking node that is in a downtime state within a preset time period, determines the topic corresponding to the target backtracking node group where the first target backtracking node is located as the second target topic; and

[0115] Re-allocating the partitions of the second target topic to obtain at least one new partition corresponding to each remaining backtracking node in the target backtracking node group;

[0116] Acquire the associated current displacement parameter from the meta information according to the partition identifier of the new partition;

[0117] According to the partition identifier of the new partition and the current displacement parameter associated with the new partition, a partition update message is sent to the remaining backtracking nodes corresponding to the new partition; wherein the partition update message is used by the remaining backtracking nodes to determine the target log data after the target log data corresponding to the current displacement parameter in the new partition corresponding to the partition identifier as the data to be processed, and determine the indicator data based on the data to be processed.

[0118] Optionally, the device further comprises: a recovery module;

[0119] The recovery module determines a second target partition corresponding to the second target backtracking node if it is determined that there is a second target backtracking node that is restarted after a shutdown; and

[0120] Acquire the associated current displacement parameter from the meta information according to the partition identifier of the second target partition;

[0121] According to the partition identifier of the second target partition and the current displacement parameter associated with the second target partition, a data processing message is sent to the second target backtracing node; the data processing message is used by the second target backtracing node to determine the target log data after the target log data corresponding to the current displacement parameter in the second target partition corresponding to the partition identifier as data to be processed, and determine the indicator data based on the data to be processed.

[0122] The cold start device of the service provided by one or more embodiments of the present specification determines the meta information of the target cold start task according to the processing request of the target cold start task obtained; the meta information includes the partition corresponding to each backtracking node in the Kafka message queue among multiple backtracking nodes; the target historical log data of the first service corresponding to the target cold start task is obtained according to the processing request and saved in each partition; wherein the target historical log data in each partition is used for the backtracking node corresponding to the partition to determine the index data of the cumulative index of the second service having a dependency relationship with the first service; the second service is cold started according to the index data. Thus, by adopting the kafka message queue to store the target historical log data of the first service and determining the partition corresponding to each backtracking node in the Kafka message queue, each backtracking node can process the target historical log data in the corresponding partition in parallel in an orderly manner, which can not only avoid the problem that the same target historical log data is repeatedly processed by multiple backtracking nodes or the omission of a certain target historical log data, but also greatly improve the processing efficiency of the target historical log data, which can meet the processing requirements of the daily average of tens of millions or even more log data, thereby meeting the cold start requirements of the service and realizing low-cost and high-efficiency service cold start.

[0123] It should be noted that the embodiments of the cold start device for a service in this specification and the embodiments of the cold start method for a service in this specification are based on the same inventive concept, so the specific implementation of this embodiment can refer to the implementation of the cold start method for the corresponding service mentioned above, and the repeated parts will not be repeated.

[0124] Further, corresponding to the cold start method of the service described above, based on the same technical concept, one or more embodiments of this specification also provide a cold start device for a service, the device is used to execute the cold start method of the service described above, Figure 8 A schematic diagram of the structure of a cold start device for a service provided in one or more embodiments of this specification.

[0125] like Figure 8 As shown, the cold start device of the business may have relatively large differences due to different configurations or performances, and may include one or more processors 301 and memory 302, and one or more storage applications or data may be stored in the memory 302. Among them, the memory 302 can be a short-term storage or a persistent storage. The application stored in the memory 302 may include one or more modules (not shown in the figure), and each module may include a series of computer executable instructions in the cold start device of the business. Furthermore, the processor 301 can be configured to communicate with the memory 302 to execute a series of computer executable instructions in the memory 302 on the cold start device of the business. The cold start device of the business may also include one or more power supplies 303, one or more wired or wireless network interfaces 304, one or more input and output interfaces 305, one or more keyboards 306, etc.

[0126] In a specific embodiment, the cold start device of the service includes a memory and one or more programs, wherein the one or more programs are stored in the memory, and the one or more programs may include one or more modules, and each module may include a series of computer executable instructions in the cold start device of the service, and the one or more programs are configured to be executed by one or more processors, including computer executable instructions for performing the following:

[0127] Determine the meta information of the target cold start task according to the acquired processing request of the target cold start task; wherein the meta information includes the partition corresponding to each backtracking node in the Kafka message queue of the multiple backtracking nodes;

[0128] Acquire target historical log data of the first business corresponding to the target cold start task according to the processing request;

[0129] The target historical log data is evenly saved in each of the partitions; wherein the target historical log data in each partition is used for the backtracking node corresponding to the partition to determine the index data of the cumulative index of the second business having a dependency relationship with the first business;

[0130] The second service is cold started according to the indicator data.

[0131] The cold start device of the service provided by one or more embodiments of the present specification determines the meta information of the target cold start task according to the processing request of the target cold start task obtained; the meta information includes the partition corresponding to each backtracking node in the Kafka message queue in multiple backtracking nodes; the target historical log data of the first service corresponding to the target cold start task is obtained according to the processing request and saved in each partition; wherein the target historical log data in each partition is used for the backtracking node corresponding to the partition to determine the index data of the cumulative index of the second service having a dependency relationship with the first service; the second service is cold started according to the index data. Thus, by using the kafka message queue to store the target historical log data of the first service and determining the partition corresponding to each backtracking node in the Kafka message queue, each backtracking node can process the target historical log data in the corresponding partition in parallel in an orderly manner, which can not only avoid the problem that the same target historical log data is repeatedly processed by multiple backtracking nodes or the omission of a certain target historical log data, but also greatly improve the processing efficiency of the target historical log data, which can meet the processing requirements of the daily average of tens of millions or even more log data, thereby meeting the cold start requirements of the service and realizing low-cost and high-efficiency service cold start.

[0132] It should be noted that the embodiments of the cold start device for the service in this specification and the embodiments of the cold start method for the service in this specification are based on the same inventive concept, so the specific implementation of this embodiment can refer to the implementation of the cold start method for the corresponding service mentioned above, and the repeated parts will not be repeated.

[0133] Further, corresponding to the cold start method of the service described above, based on the same technical concept, one or more embodiments of this specification further provide a storage medium for storing computer executable instructions. In a specific embodiment, the storage medium may be a USB flash drive, an optical disk, a hard disk, etc. When the computer executable instructions stored in the storage medium are executed by the processor, the following process can be implemented:

[0134] Determine the meta information of the target cold start task according to the acquired processing request of the target cold start task; wherein the meta information includes the partition corresponding to each backtracking node in the Kafka message queue of the multiple backtracking nodes;

[0135] Acquire target historical log data of the first business corresponding to the target cold start task according to the processing request;

[0136] The target historical log data is evenly saved in each of the partitions; wherein the target historical log data in each partition is used for the backtracking node corresponding to the partition to determine the index data of the cumulative index of the second business having a dependency relationship with the first business;

[0137] The second service is cold started according to the indicator data.

[0138] The computer executable instructions stored in the storage medium provided by one or more embodiments of the present specification, when executed by the processor, determine the metadata of the target cold start task according to the processing request of the target cold start task obtained; the metadata includes the partition corresponding to each backtracking node in the Kafka message queue of multiple backtracking nodes; obtain the target historical log data of the first business corresponding to the target cold start task according to the processing request and save it to each partition; wherein the target historical log data in each partition is used for the backtracking node corresponding to the partition to determine the indicator data of the cumulative indicator of the second business that has a dependency relationship with the first business; and cold start processing is performed on the second business according to the indicator data. Therefore, by using the Kafka message queue to store the target historical log data of the first business and determining the partition corresponding to each backtracking node in the Kafka message queue, each backtracking node can process the target historical log data in its corresponding partition in parallel and in an orderly manner. This not only avoids the problem of the same target historical log data being repeatedly processed by multiple backtracking nodes or the omission of processing of a certain target historical log data, but also greatly improves the processing efficiency of the target historical log data, and can meet the processing needs of tens of millions or even more log data per day, thereby meeting the cold start needs of the business and achieving low-cost and high-efficiency business cold start.

[0139] It should be noted that the embodiments of the storage medium in this specification and the embodiments of the cold start method for the service in this specification are based on the same inventive concept, so the specific implementation of this embodiment can refer to the implementation of the cold start method for the corresponding service mentioned above, and the repeated parts will not be repeated.

[0140] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0141] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0142] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0143] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0144] For the convenience of description, the above devices are described in terms of functions and are divided into various units. Of course, when implementing the embodiments of this specification, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0145] It should be understood by those skilled in the art that one or more embodiments of this specification may be provided as a method, system or computer program product. Therefore, one or more embodiments of this specification may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0146] This specification is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of this specification. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0147] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0148] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0149] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0150] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0151] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0152] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0153] One or more embodiments of the present specification may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of the present specification may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0154] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0155] The above description is only an embodiment of this document and is not intended to limit this document. For those skilled in the art, this document may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this document should be included in the scope of the claims of this document.

Claims

1. A cold start method for a service, comprising: Determine the meta information of the target cold start task according to the acquired processing request of the target cold start task; wherein the meta information includes the partition corresponding to each backtracking node in the Kafka message queue of the multiple backtracking nodes; A backtracking request is sent to each backtracking node according to the meta-information, and the target historical log data of the first business corresponding to the target cold start task is obtained according to the processing request, and the target historical log data is evenly saved in each of the partitions, so that each backtracking node determines its own corresponding partition in the Kafka message queue according to the backtracking request, determines the indicator data of the cumulative indicator of the second business that has a dependency relationship with the first business according to the target historical log data in the partition corresponding to itself, and provides the indicator data to the business system of the second business, so that the business system performs cold start processing on the second business according to the indicator data provided by each backtracking node.

2. The method according to claim 1, characterized in that The determining of the meta information of the target cold start task includes: According to the total number of the backtracking nodes and the preset concurrency configuration parameters, a first number of topics are created for the target cold start task, and the backtracking nodes are divided into the first number of backtracking node groups; wherein the backtracking node groups correspond to the topics one by one; Allocating a second number of partitions to each of the topics in the Kafka message queue; wherein the second number is not less than a third number of backtracking nodes in the backtracking node group; Performing allocation processing on the partitions of each of the topics to obtain at least one partition corresponding to each backtracking node in the backtracking node group corresponding to the topic; Meta information of the target cold start task is generated according to the task identifier of the target cold start task, the first number of topics, the first number of backtracking node groups, and the partition corresponding to each of the backtracking nodes.

3. The method according to claim 2, characterized in that The acquiring, according to the processing request, target historical log data of the first business corresponding to the target cold start task includes: Obtaining screening parameters from the processing request; If it is determined that the target historical log data matching the screening parameter in the historical log data of the first service meets the preset condition for obtaining in batches, then obtaining part of the target historical log data from the target historical log data in sequence; The step of evenly saving the target historical log data to each partition includes: Determine, according to the meta information and a preset balancing rule, a first target partition in each of the partitions for storing the portion of the target historical log data currently acquired; The portion of target historical log data currently acquired is saved in the first target partition.

4. The method according to claim 3, wherein determining, according to the meta information and a preset balancing rule, a first target partition in each of the partitions for storing the portion of the target historical log data currently acquired comprises: Querying the meta information of the target cold start task according to the task identifier of the target cold start task; Obtaining the first number of topics from the meta information; Determine, according to a preset balancing rule, a first target topic to which the log data is to be allocated among the first number of topics; Acquire the second number of candidate partitions corresponding to the first target topic from the meta information; Determine, according to the balancing rule, a first target partition for the log data to be distributed among the candidate partitions; The first target partition is determined as the first target partition for storing the portion of target historical log data currently acquired.

5. The method according to claim 1, wherein providing the indicator data to the business system of the second business comprises: The indicator data is saved to a designated storage location, so that the service system of the second service obtains the indicator data from the storage location, and performs a cold start process on the second service according to the obtained indicator data; or, The indicator data is uploaded to the business system of the second business, so that the business system performs a cold start process on the second business according to the uploaded indicator data.

6. The method according to claim 2, further comprising: Receive the current displacement parameter sent by each of the backtracking nodes; wherein the current displacement parameter represents the position of the last target historical log data currently processed by the backtracking node in the partition corresponding to the backtracking node; The meta information is updated according to the current displacement parameter.

7. The method according to claim 6, wherein the meta information includes a node identifier of the backtracking node, a partition identifier of a partition corresponding to the backtracking node, and an association relationship between the current displacement parameter, and the method further comprises: If it is determined that there is a first target backtracking node that is in a downtime state within a preset time period, the topic corresponding to the target backtracking node group where the first target backtracking node is located is determined as the second target topic; Re-allocating the partitions of the second target topic to obtain at least one new partition corresponding to each remaining backtracking node in the target backtracking node group; Acquire the associated current displacement parameter from the meta information according to the partition identifier of the new partition; According to the partition identifier of the new partition and the current displacement parameter associated with the new partition, a partition update message is sent to the remaining backtracking nodes corresponding to the new partition; wherein the partition update message is used by the remaining backtracking nodes to determine the target log data after the target log data corresponding to the current displacement parameter in the new partition corresponding to the partition identifier as the data to be processed, and determine the indicator data based on the data to be processed.

8. The method according to claim 7, further comprising: Receive the heartbeat data sent by each backtracking node, and record the sending time of the last heartbeat data of each backtracking node; as well as, Determine the recorded sending time as the starting time point of the preset duration, and determine whether there is a first target backtracking node that has not sent the heartbeat data within the preset duration; If so, the first target backtracking node is determined as the first target backtracking node that is in a downtime state.

9. The method according to claim 6, wherein the meta information includes a node identifier of the backtracking node, a partition identifier of a partition corresponding to the backtracking node, and an association relationship between the current displacement parameter, and the method further comprises: If it is determined that there is a second target backtracking node that is restarted after downtime, determine the second target partition corresponding to the second target backtracking node; Acquire the associated current displacement parameter from the meta information according to the partition identifier of the second target partition; Sending a data processing message to the second target backtracking node according to the partition identifier of the second target partition and the current displacement parameter associated with the second target partition; The data processing message is used by the second target backtracking node to determine the target log data after the target log data corresponding to the current displacement parameter as data to be processed in the second target partition corresponding to the partition identifier, and to determine the indicator data based on the data to be processed.

10. The method according to claim 2, after obtaining the processing request of the target cold start task, the method further comprises: Sending a thread start message to each of the backtracking nodes according to the determined task identifier of the target cold start task; The thread start message is used by the backtracking node to start the backtracking thread corresponding to the target cold start task; Receiving the task identifier and the thread identifier of the backtracing thread sent by the backtracing node; The dividing the backtracking nodes into the first number of backtracking node groups includes: Dividing the backtracking thread into the first number of backtracking thread groups; wherein the backtracking thread groups correspond one-to-one to the topics; The allocating process for the partition of each of the topics to obtain at least one partition corresponding to each backtracking node in the backtracking node group corresponding to the topic includes: Performing allocation processing on the partition of each of the topics to obtain at least one partition corresponding to each backtracking thread in the backtracking thread group corresponding to the topic; The generating the meta information of the target cold start task according to the task identifier of the target cold start task, the first number of topics, the first number of backtracking node groups, and the partition corresponding to each of the backtracking nodes includes: Meta information of the target cold start task is generated according to the task identifier of the target cold start task, the first number of topics, the first number of backtracking thread groups, and the partition corresponding to each backtracking thread.

11. The method according to claim 10, further comprising: If it is determined that the target cold start task is in a completed state, sending a thread closing message to the backtracking node; The thread closing message is used by the backtracking node to close the corresponding backtracking thread.

12. The method according to claim 1, obtaining target historical log data of the first service, comprising: Starting a log extraction thread, and obtaining target historical log data of the first business through the log extraction thread; or, Starting a log extraction application, and obtaining target historical log data of the first service through the log extraction application; or, A log extraction message is sent to the log extraction node, and target historical log data of the first service is obtained through the log extraction node.

13. A cold start device for a service, comprising: processor; as well as, a memory arranged to store computer executable instructions which, when executed, cause the processor to: Determine the meta information of the target cold start task according to the acquired processing request of the target cold start task; wherein the meta information includes the partition corresponding to each backtracking node in the Kafka message queue of the multiple backtracking nodes; A backtracking request is sent to each backtracking node according to the meta-information, and the target historical log data of the first business corresponding to the target cold start task is obtained according to the processing request, and the target historical log data is evenly saved in each of the partitions, so that each backtracking node determines its own corresponding partition in the Kafka message queue according to the backtracking request, determines the indicator data of the cumulative indicator of the second business that has a dependency relationship with the first business according to the target historical log data in the partition corresponding to itself, and provides the indicator data to the business system of the second business, so that the business system performs cold start processing on the second business according to the indicator data provided by each backtracking node.

14. A storage medium for storing computer executable instructions, wherein the computer executable instructions, when executed by a processor, implement the following process: According to the acquired processing request of the target cold start task, the meta information of the target cold start task is determined; wherein, The meta information includes the partition corresponding to each backtracking node in the Kafka message queue among the multiple backtracking nodes; A backtracking request is sent to each backtracking node according to the meta-information, and the target historical log data of the first business corresponding to the target cold start task is obtained according to the processing request, and the target historical log data is evenly saved in each of the partitions, so that each backtracking node determines its own corresponding partition in the Kafka message queue according to the backtracking request, determines the indicator data of the cumulative indicator of the second business that has a dependency relationship with the first business according to the target historical log data in the partition corresponding to itself, and provides the indicator data to the business system of the second business, so that the business system performs cold start processing on the second business according to the indicator data provided by each backtracking node.