Data rollback methods, media, apparatuses, and computing devices

Abstract

This disclosure provides a data backtracking method, medium, apparatus, and computing device, relating to the field of data processing technology. The data backtracking method includes: acquiring configuration information corresponding to an offline model, the configuration information including a data backtracking start time and a backtracking flag; if the backtracking flag is set to backtracking, the data retrieval operator stops existing consumption threads and creates new consumption threads, the new consumption threads acquiring target data based on the data backtracking start time; and generating sample data for updating the offline model based on the target data. This disclosure can significantly improve the efficiency of data backtracking, thereby enhancing the real-time performance and maintainability of offline model online learning.

Description

Technical Field

[0001] The embodiments of this disclosure relate to the field of data processing technology, and more specifically, the embodiments of this disclosure relate to a data backtracking method, medium, apparatus, and computing device. Background Technology

[0002] This section is intended to provide background or context for the embodiments of this disclosure as set forth in the claims. The description herein is not intended to be a prior art simply because it is included in this section.

[0003] Offline models are models trained using all historical data. Data backtracking, on the other hand, involves replacing the real-time dictionary with the dictionary trained on the offline model (used to record the admission features of the offline model) after the offline model training is complete. At the same time, data is consumed again from the end point of the offline model data to generate real-time samples. The offline model is then updated based on the real-time samples, thus achieving the effect of updating the model based on the offline model.

[0004] Currently, data backtracking is typically performed using Flink tasks. Specifically, when backtracking real-time samples, the Flink task is manually restarted. When restarting, the startup mode is specified as timestamp mode, and the desired backtracking timestamp is specified. After restarting, the Flink task can resume consuming data from the specified backtracking timestamp, generating real-time samples and updating the offline model based on these samples. However, this method of manually restarting Flink tasks is inefficient. Summary of the Invention

[0005] This disclosure provides a data backtracking method, medium, apparatus, and computing device to address the problem of low efficiency in data backtracking by manually restarting Flink tasks.

[0006] In a first aspect of this disclosure, a data backtracking method is provided, applied to a distributed data processing system. The distributed data processing system runs multiple Flink operators, including a data fetch operator. The multiple Flink operators perform streaming processing to implement Flink tasks. The data backtracking method includes:

[0007] Retrieve the configuration information corresponding to the offline model. The configuration information includes the data backtracking start time and whether backtracking is enabled.

[0008] If the backtracking flag is set to backtracking, the data fetch operator stops the existing consumer thread and creates a new consumer thread. The new consumer thread retrieves the target data based on the data backtracking start time.

[0009] Based on the target data, generate sample data for updating the offline model.

[0010] In one possible implementation, the multiple Flink operators further include a feature processing operator. The configuration information also includes the address of the offline vocabulary of the offline model. Based on the target data, sample data for updating the offline model is generated, including: the feature processing operator extracts features from the target data and determines whether the features are in the feature processing operator's local vocabulary, which is the offline vocabulary obtained based on the address of the offline vocabulary; if they are, the feature processing operator determines the features as sample data and sends the sample data and configuration information to the downstream Flink operator of the feature processing operator, through which the downstream Flink operator outputs the sample data; if they are not, the feature processing operator increments the frequency corresponding to the feature by 1 and determines whether the frequency is greater than a threshold; if the frequency is greater than the threshold, the feature is determined as sample data, and the sample data and configuration information are sent to the downstream Flink operator of the feature processing operator, through which the downstream Flink operator outputs the sample data.

[0011] In one possible implementation, the multiple Flink operators further include a topic filtering operator and an output operator. The topic filtering operator includes a first channel and a second channel, and the configuration information also includes a specified channel. The downstream Flink operator outputs sample data, including: the topic filtering operator determines a target channel based on the specified channel, where the target channel is either the first channel or the second channel, and the specified channel is the channel switched when offline model training is completed; the topic filtering operator sends the sample data to the output operator corresponding to the target channel through the target channel; and the output operator outputs the sample data.

[0012] In one possible implementation, the multiple Flink operators also include an input operator to obtain configuration information corresponding to the offline model, including: in response to the start of a Flink task or in response to the completion of offline model training, the input operator obtains the configuration information corresponding to the offline model, or the input operator periodically obtains the configuration information corresponding to the offline model.

[0013] In one possible implementation, if the backtracking flag is set to backtracking, before the data retrieval operator stops the existing consumption thread, the data backtracking method further includes: the input operator determining whether the backtracking flag is set to backtracking; if so, the input operator broadcasts configuration information, and the data retrieval operator receives the configuration information.

[0014] In one possible implementation, the configuration information further includes a configuration version. Before the data pull operator stops the existing consumer thread and creates a new consumer thread, the data backtracking method further includes: the data pull operator determines whether to update the local configuration of the data pull operator based on the configuration version and the local configuration version of the data pull operator; if yes, then the local configuration of the data pull operator is updated; if no, then the local configuration of the data pull operator is not updated.

[0015] In one possible implementation, the configuration information further includes a configuration version. Before the feature processing operator extracts features from the target data, the data backtracking method further includes: the feature processing operator determines whether to update its local configuration based on the configuration version and its local configuration version; if yes, the local configuration of the feature processing operator is updated, and the local vocabulary of the feature processing operator is replaced with an offline vocabulary; if no, the local configuration of the feature processing operator is not updated.

[0016] In one possible implementation, the configuration information further includes a configuration version. Before the topic filtering operator determines the target channel based on the specified channel, the data backtracking method further includes: the topic filtering operator determines whether to update its local configuration based on the configuration version and its local configuration version; if yes, the local configuration of the topic filtering operator is updated, and the sequence number corresponding to the instance in the topic filtering operator whose local configuration has been updated is recorded based on the configuration version; if no, the local configuration of the topic filtering operator is not updated.

[0017] In one possible implementation, after recording the sequence number corresponding to the instance whose local configuration has been updated in the topic filtering operator, the data backtracking method further includes: when the input operator determines that the number of sequence numbers is the same as the parallelism of the topic filtering operator, it updates the backtracking flag to non-backtracking.

[0018] Secondly, embodiments of this disclosure provide a data backtracking device applied to a distributed data processing system. The distributed data processing system runs multiple Flink operators, including a data fetch operator. These multiple Flink operators perform streaming processing to implement Flink tasks. The data backtracking device includes:

[0019] The acquisition module is used to obtain the configuration information corresponding to the offline model. The configuration information includes the data backtracking start time and whether backtracking is enabled.

[0020] The processing module is used to stop the existing consumer thread and create a new consumer thread if the backtracking mark is set to backtracking. The new consumer thread retrieves the target data based on the data backtracking start time.

[0021] The generation module is used to generate sample data for updating the offline model based on the target data.

[0022] In one possible implementation, the multiple Flink operators further include a feature processing operator, and the configuration information also includes the address of the offline vocabulary of the offline model. The generation module is specifically used for: the feature processing operator extracting features from the target data and determining whether the features are in the local vocabulary of the feature processing operator, where the local vocabulary is the offline vocabulary obtained based on the address of the offline vocabulary; if they are, the feature processing operator determines the features as sample data, sends the sample data and configuration information to the downstream Flink operator of the feature processing operator, and outputs the sample data through the downstream Flink operator; if they are not, the feature processing operator increments the frequency corresponding to the feature by 1 to determine whether the frequency is greater than a threshold; if the frequency is greater than the threshold, the feature is determined as sample data, and the sample data and configuration information are sent to the downstream Flink operator of the feature processing operator, and outputs the sample data through the downstream Flink operator.

[0023] In one possible implementation, the multiple Flink operators further include a topic filtering operator and an output operator. The topic filtering operator includes a first channel and a second channel, and the configuration information also includes a specified channel. When the generation module outputs sample data through the downstream Flink operators, it specifically performs the following: the topic filtering operator determines a target channel based on the specified channel, where the target channel is either the first channel or the second channel, and the specified channel is the channel switched when offline model training is completed; the topic filtering operator sends the sample data to the output operator corresponding to the target channel through the target channel; and the output operator outputs the sample data.

[0024] In one possible implementation, the multiple Flink operators also include an input operator, and the acquisition module is specifically used to: in response to the start of a Flink task or in response to the completion of offline model training, the input operator acquires the configuration information corresponding to the offline model, or the input operator acquires the configuration information corresponding to the offline model periodically.

[0025] In one possible implementation, the processing module is further configured to: determine whether the backtracking flag is set to backtracking before the data retrieval operator stops the existing consumption thread if the backtracking flag is set to backtracking; if so, broadcast configuration information to the input operator and receive the configuration information to the data retrieval operator.

[0026] In one possible implementation, the configuration information further includes a configuration version, and the processing module is further configured to: before the data pull operator stops the existing consumer thread and creates a new consumer thread, the data pull operator determines whether to update the local configuration of the data pull operator based on the configuration version and the local configuration version of the data pull operator; if yes, then update the local configuration of the data pull operator; if no, then do not update the local configuration of the data pull operator.

[0027] In one possible implementation, the configuration information further includes a configuration version, and the generation module is further configured to: before the feature processing operator extracts features from the target data, the feature processing operator determines whether to update its local configuration based on the configuration version and its local configuration version; if yes, the local configuration of the feature processing operator is updated, and the local vocabulary of the feature processing operator is replaced with an offline vocabulary; if no, the local configuration of the feature processing operator is not updated.

[0028] In one possible implementation, the configuration information further includes a configuration version, and the generation module is further configured to: before the topic filtering operator determines the target channel according to the specified channel, the topic filtering operator determines whether to update the local configuration of the topic filtering operator based on the configuration version and the local configuration version of the topic filtering operator; if yes, then update the local configuration of the topic filtering operator, and record the sequence number corresponding to the instance in the topic filtering operator whose local configuration has been updated based on the configuration version; if no, then do not update the local configuration of the topic filtering operator.

[0029] In one possible implementation, the generation module is further configured to: after updating the sequence number corresponding to the locally configured instance in the topic filtering operator, update the backtracking flag to non-backtracking when the input operator determines that the number of sequence numbers is the same as the parallelism of the topic filtering operator.

[0030] Thirdly, embodiments of this disclosure provide a computing device, including: a processor, and a memory communicatively connected to the processor;

[0031] The memory stores the instructions that the computer executes;

[0032] The processor executes computer execution instructions stored in memory to implement the data backtracking method as described in the first aspect of this disclosure.

[0033] Fourthly, embodiments of this disclosure provide a storage medium storing computer program instructions, which, when executed, implement the data backtracking method as described in the first aspect of this disclosure.

[0034] Fifthly, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the data backtracking method as described in the first aspect of this disclosure.

[0035] The data backtracking method, medium, apparatus, and computing device provided in this disclosure acquire configuration information corresponding to the offline model. This configuration information includes the data backtracking start time and a backtracking flag. If the backtracking flag indicates backtracking, the data fetch operator stops existing consumer threads and creates new consumer threads. The new consumer threads acquire target data based on the data backtracking start time. Based on the target data, sample data for updating the offline model is generated. Because this disclosure performs data backtracking by stopping existing consumer threads and creating new ones when the backtracking flag in the configuration information is determined to be backtracking, without requiring manual restarting of Flink tasks, it significantly improves the efficiency of data backtracking, thereby enhancing the real-time performance and maintainability of the offline model's online learning. Attached Figure Description

[0036] The above and other objects, features, and advantages of this disclosure will become readily apparent from the following detailed description of exemplary embodiments, taken in conjunction with the accompanying drawings. Several embodiments of this disclosure are illustrated in the drawings by way of example and not limitation, in which:

[0037] Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of the present disclosure;

[0038] Figure 2 A flowchart of a data backtracking method provided in an embodiment of this disclosure;

[0039] Figure 3 A flowchart of a data backtracking method provided in another embodiment of this disclosure;

[0040] Figure 4 A flowchart of a data backtracking method provided in yet another embodiment of this disclosure;

[0041] Figure 5 This is a schematic diagram of the structure of a data backtracking device provided in an embodiment of the present disclosure;

[0042] Figure 6 A schematic diagram of a storage medium provided in an embodiment of this disclosure;

[0043] Figure 7 This is a schematic diagram of the structure of a computing device provided in an embodiment of the present disclosure.

[0044] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts. Detailed Implementation

[0045] The principles and spirit of this disclosure will now be described with reference to several exemplary embodiments. It should be understood that these embodiments are given merely to enable those skilled in the art to better understand and implement this disclosure, and are not intended to limit the scope of this disclosure in any way. Rather, these embodiments are provided to make this disclosure more thorough and complete, and to fully convey the scope of this disclosure to those skilled in the art.

[0046] Those skilled in the art will recognize that embodiments of this disclosure can be implemented as a system, apparatus, device, method, or computer program product. Therefore, this disclosure can be specifically implemented in the following forms: entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software.

[0047] According to embodiments of this disclosure, a data backtracking method, medium, apparatus, and computing device are proposed.

[0048] In this article, it is important to understand the following terms:

[0049] Kafka is an open-source real-time message queue from the Apache Software Foundation, consisting of a producer, a broker, and a consumer.

[0050] A Kafka topic is a message subject in Kafka, used to uniquely identify a message queue.

[0051] The Kafka offset is a long integer that measures the number of messages recorded in Kafka. The offset increases by 1 for each message written.

[0052] A Kafka time index is an index that associates the time a message was written to Kafka with its offset. It can be used to retrieve the offset corresponding to a specified time.

[0053] Flink is a distributed real-time data processing framework open-sourced by the Apache Software Foundation. Users can write different data processing logic as Flink operators, submit them to the Flink cluster, generate Flink tasks with a certain topology, process data in a streaming manner, and obtain the final result.

[0054] In Flink, operators are logical units that process data. An operator can have multiple instances, and these instances perform the same processing on the data. Different operators form a directed acyclic graph that processes the data together.

[0055] In Flink, input operators (i.e., source operators) are used to retrieve data from external systems (such as Kafka).

[0056] In Flink, output operators (i.e., sink operators) are used to output data to external systems (such as Kafka).

[0057] Transform operators in Flink are internal operators used by Flink to perform data mapping, filtering, and transformation.

[0058] Parallelism of operators in Flink: Operators in Flink can be configured with parallelism to generate multiple instances and process data in parallel.

[0059] Data exchange strategies between Flink operators can be configured between upstream and downstream Flink operators to determine which instance of the downstream operator the data is distributed to. Common exchange strategies include broadcast strategies, which broadcast data generated by upstream operators to all instances of downstream operators; and rebalance strategies, which send data generated by upstream operators to downstream operators in a round-robin manner to ensure load balancing downstream.

[0060] Model training is the process of solving for the parameters of an algorithm model using existing data.

[0061] Offline training is the process of training a model using all historical data (usually from N days to T-1 days, where T represents the current day).

[0062] Incremental learning is the process of updating an offline model by receiving a batch of new samples based on the offline model obtained through offline training.

[0063] Online learning is a special case of incremental learning, where the offline model is updated every time a sample is obtained, resulting in higher real-time performance.

[0064] Features are the input variables of the model, such as gender being male or city being city A.

[0065] A sample is a specific instance of data; a sample contains multiple features.

[0066] Feature admission is a mechanism that controls whether features can be included in the model. If features that appear less frequently are included in the model training, the training results will be unreliable. During the model training process, these features that appear less frequently need to be filtered out.

[0067] The dictionary is used to record the statistical results of the frequency of features; features that appear less frequently will be filtered out.

[0068] Data backtracking, which means that after the offline model training is completed, the local real-time dictionary needs to be replaced with the dictionary trained by the offline model. At the same time, data is consumed again from the end point of the offline model data to generate real-time sample data, thereby achieving the effect of updating the model based on the offline model.

[0069] Redis (Remote Dictionary Server) is an open-source distributed memory cache that can efficiently store key-value pair data.

[0070] The Redis INCR operation increments the value stored in the Redis key by 1.

[0071] Hadoop Distributed File System (HDFS) is an open-source distributed file system that can store massive amounts of data and can read files based on file paths.

[0072] Furthermore, the number of any elements in the accompanying drawings is for illustrative purposes only and not for limitation, and any naming is for distinction only and has no limiting meaning.

[0073] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0074] The principles and spirit of this disclosure will be explained in detail below with reference to several representative embodiments. Invention Overview

[0076] The inventors have discovered that currently, when performing data backtracking on real-time samples, the Flink task is typically restarted manually. When restarting the Flink task, the startup mode is specified as timestamp mode, and the desired backtracking timestamp is specified. After restarting, the Flink task can resume consuming data from the specified backtracking timestamp, generating real-time samples, and updating the offline model based on these samples, thus achieving the purpose of data backtracking. However, this method of manually restarting the Flink task is inefficient, and the time required to stop and start the Flink task is long, failing to guarantee real-time performance. Furthermore, the backtracked data is not separated from the data before backtracking, making subsequent consumption inconvenient.

[0077] To address the aforementioned issues, this disclosure provides a data backtracking method, medium, apparatus, and computing device. When data backtracking is required based on the configuration information corresponding to the offline model, existing consumer threads are automatically stopped and new consumer threads are created through data retrieval operators in multiple Flink operators. The new consumer threads acquire target data based on the data backtracking start time, thereby generating sample data for updating the offline model. This achieves automated real-time data backtracking, solving the need for periodically loading offline vocabulary and backtracking data during the real-time feature admission process of online learning. It can greatly improve the efficiency of data backtracking, thereby enhancing the real-time performance and maintainability of offline models in online learning.

[0078] Application Scenarios Overview

[0079] First refer to Figure 1 Examples of application scenarios for the solutions provided in this disclosure are given. Figure 1 This is a schematic diagram of an application scenario provided by an embodiment of the present disclosure, such as... Figure 1As shown, this application scenario can include: server cluster 11, server cluster 12, and terminal device 13. Server cluster 11 includes multiple servers 111 and a storage device 112, with servers 111 storing data through the storage device 112. Server cluster 12 includes multiple servers 121 and a storage device 122, with servers 121 storing data through the storage device 122. Terminal device 13 can be a tablet, laptop, desktop computer, or smart home appliance, etc. Server 111 is used to obtain an offline model through offline training. The offline model is used to predict the conversion rate and click-through rate of advertisements displayed on terminal device 13. During offline training, data is retrieved from storage device 112, and the generated data is stored in storage device 112. Furthermore, during offline training, server 111 and terminal device 13 communicate via a wireless or wired network. A distributed data processing system is deployed on server 121, running multiple Flink operators. Data backtracking is performed through these Flink operators to generate sample data for updating the offline model. Server 121 and server 111 communicate via a wireless or wired network. After updating the offline model based on sample data, the server 111 can more accurately predict the conversion rate and click-through rate of the advertisements displayed on the terminal device 13.

[0080] It should be noted that, Figure 1 This is merely a schematic diagram illustrating one application scenario provided by an embodiment of this disclosure; this embodiment does not necessarily represent an application scenario. Figure 1 The included equipment is not limited, nor is it restricted. Figure 1 The positional relationships between the devices are defined.

[0081] Exemplary methods

[0082] The following is combined Figure 1 Application scenarios, refer to Figure 2 This document describes a data backtracking method according to exemplary embodiments of the present disclosure. It should be noted that the above application scenarios are shown only to facilitate understanding of the spirit and principles of the present disclosure, and the embodiments of the present disclosure are not limited in any way. Rather, the embodiments of the present disclosure can be applied to any applicable scenario.

[0083] First, the data backtracking method is introduced through specific examples.

[0084] Figure 2The flowchart illustrates a data backtracking method provided in one embodiment of this disclosure, applied to a distributed data processing system. This system runs multiple Flink operators, including a data fetch operator, which performs streaming processing to implement Flink tasks. The method described in this embodiment can be applied to a computing device, such as a server or server cluster. Figure 2 As shown, the method in this embodiment includes:

[0085] S201. Obtain the configuration information corresponding to the offline model. The configuration information includes the data backtracking start time and whether backtracking is enabled.

[0086] In this embodiment, the configuration information can be obtained in several ways. For example, when the offline model training is complete, the configuration information can be updated by calling the Application Programming Interface (API) of the configuration backend. Then, the configuration information corresponding to the offline model can be obtained by listening to the configuration information update messages from the configuration backend. Alternatively, the configuration information corresponding to the offline model can be obtained when the Flink task starts. Or, the configuration information can be obtained periodically. The configuration information includes a data backtracking start time, which specifies the time point from which backtracking begins, such as 0:00 on the current day. The configuration information also includes a backtracking flag to indicate whether backtracking is needed. When the offline model has just finished running, the backtracking flag is set to "backtracking" (e.g., "true"), indicating that backtracking is needed to generate sample data for updating the offline model. This sample data can be understood as real-time sample data. After backtracking is complete, the configuration backend API can be called to update the flag to "no backtracking" (e.g., "false"), indicating that backtracking is no longer needed to avoid repeated backtracking. The configuration information may also include other information, which can be found in subsequent embodiments.

[0087] S202. If the backtracking flag is set to backtracking, the data retrieval operator stops the existing consumer thread and creates a new consumer thread. The new consumer thread retrieves the target data based on the data backtracking start time.

[0088] In this step, if the backtracking flag in the configuration information is set to "backtracking," it indicates that backtracking is required. The data fetch operator (e.g., represented by the KafkaOperator operator) stops existing consumer threads and creates new ones. It then begins consuming data from the data backtracking start time specified in the configuration information, i.e., retrieving the target data. For example, the data fetch operator can retrieve a preset amount of target data every 100ms.

[0089] S203. Based on the target data, generate sample data for updating the offline model.

[0090] In this step, after obtaining the target data, for example, the target data and configuration information can be sent to the Flink operator downstream of the data fetch operator. The Flink operator downstream of the data fetch operator generates sample data for updating the offline model. This sample data can be understood as real-time sample data, and the offline model can be updated based on the real-time sample data. For details on how to generate sample data for updating the offline model based on the target data, please refer to the subsequent embodiments, which will not be repeated here.

[0091] The data backtracking method provided in this disclosure obtains the configuration information corresponding to the offline model. This configuration information includes the data backtracking start time and a backtracking flag. If the backtracking flag is set to "backtracking," the data fetch operator stops the existing consumer thread and creates a new consumer thread. The new consumer thread retrieves target data based on the data backtracking start time. Based on the target data, sample data for updating the offline model is generated. Because this disclosure stops the existing consumer thread and creates a new one when the backtracking flag in the configuration information is set to "backtracking," data backtracking is performed without manually restarting the Flink task. This significantly improves the efficiency of data backtracking, thereby enhancing the real-time performance and maintainability of the offline model's online learning.

[0092] Figure 3 This is a flowchart illustrating a data backtracking method provided in another embodiment of this disclosure. Based on the above embodiments, the multiple Flink operators may further include input operators (i.e., Source operators), feature processing operators (e.g., represented by FeatureOperator operators), topic filtering operators (e.g., represented by TopicSwitchOperator operators), and output operators (i.e., Sink operators). This disclosure further illustrates the data backtracking method based on multiple different Flink operators. Figure 3 As shown, the method in this embodiment of the disclosure may include:

[0093] In this embodiment of the disclosure, Figure 2 Step S201 may further include the following step S301:

[0094] S301. In response to the start of a Flink task or the completion of offline model training, input operators are used to obtain the configuration information corresponding to the offline model, or input operators are used to periodically obtain the configuration information corresponding to the offline model.

[0095] It is understandable that when a Flink task starts, the configuration information corresponding to the offline model can be obtained through input operators; or, during the running of a Flink task, in response to the completion of offline model training, i.e., upon listening to the update message of configuration information from the configuration backend, the configuration information corresponding to the offline model can be obtained through input operators; or, during the running of a Flink task, the configuration information corresponding to the offline model can be obtained periodically through input operators to prevent anomalies caused by the loss of listening messages.

[0096] S302. Input operator determines whether the backtracking flag is backtracking; if so, input operator broadcasts configuration information, and data retrieval operator receives configuration information.

[0097] In this step, after the input operator obtains the configuration information, it can determine whether the backtracking flag included in the configuration information is set to backtracking. If the backtracking flag is set to backtracking, it indicates that data backtracking is required. In this case, the input operator broadcasts the configuration information, and the corresponding data retrieval operator receives the configuration information. If the backtracking flag is set to no backtracking, it indicates that data backtracking is not required, and the input operator will not broadcast the configuration information.

[0098] S303. The data retrieval operator determines whether to update the local configuration based on the configuration version in the configuration information and the local configuration version of the data retrieval operator; if it is determined to update the local configuration, then the local configuration is updated.

[0099] It's understandable that the configuration information can also include a configuration version, which could be an incrementing number indicating the newness of the configuration. For example, the training completion date of the offline model could be used as the configuration version, such as 20221226. Assuming the local configuration version of the data fetch operator is 20221225, by comparing the configuration version in the configuration information with the local configuration version of the data fetch operator, if the configuration version in the configuration information is determined to be higher, then the local configuration can be updated, for example, by updating the configuration version in the local configuration to the configuration version in the configuration information; if the data fetch operator determines that the local configuration version is higher, then the local configuration will not be updated. Figure 3 (Not shown in the image).

[0100] Optionally, if there are multiple data fetch operators, each data fetch operator, upon receiving configuration information broadcast by the input operator, can determine whether to update its local configuration based on the configuration version in the configuration information and the local configuration version of the data fetch operator.

[0101] S304. The data retrieval operator stops the existing consumer thread and creates a new consumer thread. The new consumer thread retrieves the target data based on the data backtracking start time.

[0102] For a detailed description of this step, please refer to [link / reference]. Figure 2 The relevant description of S202 in the illustrated embodiment will not be repeated here.

[0103] S305, The data retrieval operator sends the configuration information and target data to the feature processing operator.

[0104] For example, if there are multiple feature processing operators, the data retrieval operator can send configuration information and target data to the feature processing operators based on a rebalancing strategy to ensure load balancing of the feature processing operators.

[0105] S306. The feature processing operator determines whether to update the local configuration based on the configuration version in the configuration information and the local configuration version of the feature processing operator; if it is determined to update the local configuration, then the local configuration is updated.

[0106] For example, if the feature processing operator determines that the configuration version in the configuration information is higher by comparing the configuration version in the configuration information with the local configuration version of the feature processing operator, then it can determine to update the local configuration and update the local configuration accordingly; if the feature processing operator determines that the local configuration version is higher, then it will not update the local configuration. Figure 3 (Not shown in the image).

[0107] Optionally, there can be multiple feature processing operators, each storing its local configuration version through a cache. For example, assuming there are four feature processing operators, and the configuration version in the configuration information is relatively high, if the data fetch operator retrieves two target data entries for the first time, the data fetch operator will send the configuration information and target data to two of the four feature processing operators based on a rebalancing strategy. These two feature processing operators can update their local configurations according to the configuration information, while the remaining two feature processing operators do not update their local configurations. After the data fetch operator retrieves two target data entries for the second time, it will send the configuration information and the two newly retrieved target data entries to the remaining two feature processing operators based on the rebalancing strategy. The remaining two feature processing operators can then update their local configurations according to the configuration information.

[0108] The configuration information may also include the specified channel and the offline vocabulary address of the offline model. The downstream Flink operators of the feature processing operators include topic filtering operators and output operators. The topic filtering operators include a first channel and a second channel. Therefore, in this embodiment of the disclosure, Figure 2 Step S203 may further include the following six steps: S307 to S312:

[0109] S307. The feature processing operator extracts features from the target data and determines whether the features are in the local vocabulary of the feature processing operator. The local vocabulary is the offline vocabulary obtained based on the address of the offline vocabulary.

[0110] It is understandable that after updating its local configuration, the feature processing operator retrieves the offline vocabulary based on the offline vocabulary address, reloads the offline vocabulary, and replaces the local vocabulary. Then, the feature processing operator extracts features from the target data and determines whether those features are within its local vocabulary.

[0111] S308. If present, the feature processing operator determines the feature as sample data and sends the sample data and configuration information to the downstream topic filtering operator.

[0112] For example, after determining that the features of the target data are within the local vocabulary, the feature processing operator can identify the features as sample data and send the sample data and configuration information to the downstream topic filtering operator. For example, if there are multiple topic filtering operators, the feature processing operator can send the sample data and configuration information to the downstream topic filtering operators based on a rebalancing strategy to ensure load balancing of the topic filtering operators.

[0113] S309. If not, the feature processing operator increments the frequency corresponding to the feature by 1 to determine if the frequency is greater than the threshold. If the frequency is greater than the threshold, the feature is determined to be sample data, and the sample data and configuration information are sent to the downstream topic filtering operator. Accordingly, the topic filtering operator receives the sample data and configuration information.

[0114] For example, a threshold is used to indicate how many times a feature must appear to be considered as sample data and sent to the downstream topic filtering operator. For instance, a threshold of 100 means that a feature must appear more than 100 times to be considered as sample data, i.e., for updating the offline model. The threshold can be pre-configured in the configuration information or a fixed value can be used. If the threshold is included in the configuration information, the feature processing operator retrieves the threshold from the configuration information. After determining that the feature is not in the local vocabulary, it increments the frequency of the feature by 1 and stores it in Redis. By querying Redis, it obtains the total real-time frequency of the feature and determines whether the frequency is greater than 100. If the frequency is greater than 100, the feature is determined to be sample data, and the sample data and configuration information are sent to the downstream topic filtering operator. If the frequency is less than or equal to 100, the feature is not considered as sample data, i.e., the sample data and configuration information are not sent to the downstream topic filtering operator.

[0115] S310. The topic filtering operator determines whether to update the local configuration based on the configuration version in the configuration information and the local configuration version of the topic filtering operator. If it is determined to update the local configuration, the local configuration is updated, and the sequence number corresponding to the instance in the topic filtering operator that has updated the local configuration is recorded based on the configuration version in the configuration information.

[0116] For example, the topic filtering operator compares the configuration version in the configuration information with the topic filtering operator's local configuration version. If the configuration version in the configuration information is higher, then the local configuration can be updated. If the topic filtering operator determines that the local configuration version is higher, then the local configuration is not updated. Figure 3 (Not shown in the image).

[0117] For example, the topic filtering operator updates the configuration version in the local configuration to the configuration version in the configuration information, and records the sequence number of the instance that updated the local configuration in Redis. Specifically, for example, the configuration version is used as the key, and the value is a set, where each element in the set is the sequence number of the instance that updated the local configuration. It can be understood that each topic filtering operator adds an element to this set after receiving the updated configuration information.

[0118] S311 The topic filtering operator determines the target channel based on the specified channel in the configuration information, and sends the sample data to the output operator corresponding to the target channel through the target channel.

[0119] The target channel is either the first channel or the second channel, while the specified channel is the channel that is switched when the offline model training is completed.

[0120] In this step, the specified channel in the configuration information is used to isolate the real-time sample data before and after data backtracking. Two specified channels can be set, i.e., two output topics, represented by Topic A (Topic_A) and Topic B (Topic_B) respectively. The topic is switched during each data backtracking to ensure the isolation of sample data before and after backtracking. The topic filtering operator includes a first channel and a second channel. Assuming that the first channel is also represented by Topic_A and the second channel is also represented by Topic_B, and the specified channel in the configuration information is Topic_A, the topic filtering operator can compare the first and second channels with the specified channel in the received configuration information to determine that the target channel is Topic_A, i.e., the first channel is the target channel.

[0121] After determining the target channel, the topic filtering operator can send sample data to the output operator corresponding to the target channel, such as the output operator corresponding to Topic_A.

[0122] S312, Output operator outputs sample data.

[0123] S313. When the number of sequence numbers and the parallelism of the topic filtering operator are the same, update the backtracking flag in the configuration information to "no backtracking".

[0124] For example, the parallelism of the topic filtering operator can be set as needed, and this disclosure does not limit this. Based on the example of step S310, the input operator polls Redis. If the number of elements in the set (i.e., the number of indices) is consistent with the parallelism of the topic filtering operator, it is determined that the current data backtracking switch has been completed. The API can be called to notify the configuration backend. The configuration backend sets the current data backtracking to complete and updates the backtracking flag in the configuration information to no backtracking, no longer checking Redis, indicating that no further backtracking is needed.

[0125] For example, suppose the data fetching operator retrieves 3 target data entries every 100ms, the topic filtering operator has a parallelism of 10, and the configuration version in the configuration information is relatively high. After the data retrieval operator first obtains 3 target data items, it sends these 3 target data items and configuration information to the downstream feature processing operators (assuming there are 10 feature processing operators in total). Three of these feature processing operators update their local configurations according to the configuration information and then send the extracted target data features and configuration information to the three downstream topic filtering operators. Assuming 5 topic filtering operators contain the first channel Topic_A, and 5 topic filtering operators contain the second channel Topic_B, and the specified channel in the configuration information is Topic_A, the 3 topic filtering operators containing Topic_A receive the features sent by the feature processing operators, and the 3 topic filtering operators containing Topic_B also receive the features sent by the feature processing operators. Correspondingly, the Redis set contains the indices of 6 instances of the topic filtering operators. At this point, the input operator polls Redis and determines that the number of instance indices for the topic filtering operators is 6, which is different from the parallelism of 10 for the topic filtering operators. Therefore, it is determined that the current data backtracking switch has not been completed. After the second data retrieval operator obtains 3 target data items, based on the above method, the final Redis set contains the sequence number of 10 topic filtering operator instances, which is the same as the parallelism of the topic filtering operator 10. Then, the backtracking flag in the input operator update configuration information is set to no backtracking, which means that the current data backtracking switch is completed.

[0126] It should be noted that the embodiments disclosed herein do not limit the order in which steps S311 and S313 are executed.

[0127] The data backtracking method provided in this disclosure, when the configuration information includes a backtracking flag indicating backtracking, stops the existing consumption thread through the data retrieval operator and creates a new consumption thread to obtain target data based on the data backtracking start time. Then, it outputs sample data generated based on the target data through feature processing operators, topic filtering operators, and output operators. The sample data is used to update the offline model, completing the data backtracking. This data backtracking is performed without manually restarting the Flink task, which greatly improves the efficiency of data backtracking and enhances the real-time performance and maintainability of the offline model's online learning. In addition, by using the topic filtering operator to determine the target channel according to the specified channel in the configuration information, it is possible to ensure that the sample data before and after the data backtracking are stored in different topics, which facilitates subsequent problem tracking and subsequent data consumption.

[0128] Based on the above embodiments, Figure 4 A flowchart of a data backtracking method provided in yet another embodiment of this disclosure is shown below. Figure 4 As shown, the method in this embodiment of the disclosure may include:

[0129] S401. In response to the start of a Flink task, the input operator obtains the configuration information corresponding to the offline model from the configuration backend via the API interface.

[0130] It is understood that the embodiments disclosed herein are illustrated by taking the example of obtaining configuration information by inputting operators when a Flink task starts.

[0131] S402. When the input operator determines whether the backtracking flag in the configuration information is set to backtracking, it broadcasts the configuration information; correspondingly, multiple data retrieval operators receive the configuration information.

[0132] S403. When the data pull operator determines that the configuration version in the configuration information is higher than the local configuration version, it updates the local configuration, stops the existing consumer thread, and creates a new consumer thread. The new consumer thread consumes data from the original Kafka sample Topic based on the data backtracking start time in the configuration information, i.e., it retrieves the target data.

[0133] S404. The data retrieval operator sends configuration information and target data to multiple feature processing operators based on the rebalancing strategy.

[0134] S405. After receiving the configuration information and target data, the feature processing operator updates the local configuration when it determines that the configuration version in the configuration information is higher than the local configuration version. Based on the offline dictionary HDFS address in the configuration information, it obtains the offline dictionary and reloads the offline dictionary to replace the local dictionary.

[0135] S406. The feature processing operator extracts features from the target data and determines whether the features are in the local vocabulary of the feature processing operator. If they are, the feature processing operator determines the features as sample data. If they are not, the feature processing operator increments the frequency of the feature by 1, that is, counts the frequency of the feature and writes it into the real-time dictionary of Redis. Query Redis to determine whether the total real-time frequency of the feature is greater than the threshold. If the total real-time frequency is greater than the threshold, the feature is determined to be sample data.

[0136] S407. The feature processing operator sends the sample data and configuration information to the downstream topic filtering operator based on the rebalancing strategy, that is, the feature output is given to the topic filtering operator.

[0137] S408. After receiving configuration information and sample data, the topic filtering operator updates the local configuration when it determines that the configuration version in the configuration information is higher than the local configuration version. Based on the configuration version in the configuration information, it records the sequence number of the instance in Redis that has updated the local configuration in the topic filtering operator. It determines the target channel according to the specified channel in the configuration information. It sends the sample data to the output operator corresponding to the target channel through the target channel. The output operator outputs the sample data.

[0138] S409. When the input operator determines that the number of sequence numbers and the parallelism of the topic filtering operator are the same, it determines that the current data backtracking switch has been completed, calls the API to notify the configuration backend, the configuration backend sets the current data backtracking to completed, and updates the backtracking flag in the configuration information to non-backtracking, no longer checks Redis, indicating that no further backtracking is needed.

[0139] Exemplary device

[0140] Having introduced the medium of exemplary embodiments of this disclosure, the following references are made to... Figure 5 The data backtracking apparatus according to exemplary embodiments of this disclosure will be described. The apparatus of the exemplary embodiments of this disclosure can implement the various processes in the foregoing data backtracking method embodiments and achieve the same functions and effects.

[0141] Figure 5 This is a schematic diagram of a data backtracking device provided in one embodiment of the present disclosure. It is applied to a distributed data processing system, which runs multiple Flink operators, including a data fetch operator. These multiple Flink operators perform streaming processing to implement Flink tasks. Figure 5 As shown, the data backtracking device 500 of this embodiment includes: an acquisition module 501, a processing module 502, and a generation module 503. Wherein:

[0142] The acquisition module 501 is used to acquire the configuration information corresponding to the offline model. The configuration information includes the data backtracking start time and whether backtracking is enabled.

[0143] The processing module 502 is used to stop the existing consumer thread and create a new consumer thread if the backtracking mark is set to backtracking. The new consumer thread retrieves the target data based on the data backtracking start time.

[0144] The generation module 503 is used to generate sample data for updating the offline model based on the target data.

[0145] In one possible implementation, the multiple Flink operators further include a feature processing operator, and the configuration information also includes the address of the offline vocabulary of the offline model. The generation module 503 can be specifically used for: the feature processing operator extracting features from the target data and determining whether the features are in the local vocabulary of the feature processing operator, where the local vocabulary is the offline vocabulary obtained according to the address of the offline vocabulary; if they are, the feature processing operator determines that the features are sample data, sends the sample data and configuration information to the downstream Flink operator of the feature processing operator, and outputs the sample data through the downstream Flink operator; if they are not, the feature processing operator increments the frequency corresponding to the feature by 1 to determine whether the frequency is greater than a threshold; if the frequency is greater than the threshold, the feature is determined to be sample data, and the sample data and configuration information are sent to the downstream Flink operator of the feature processing operator, and outputs the sample data through the downstream Flink operator.

[0146] In one possible implementation, the multiple Flink operators further include a topic filtering operator and an output operator. The topic filtering operator includes a first channel and a second channel, and the configuration information also includes a specified channel. When the generation module 503 is used to output sample data through the downstream Flink operators, it can specifically be used as follows: the topic filtering operator determines a target channel based on the specified channel, where the target channel is either the first channel or the second channel, and the specified channel is the channel switched when the offline model training is completed; the topic filtering operator sends the sample data to the output operator corresponding to the target channel through the target channel; and the output operator outputs the sample data.

[0147] In one possible implementation, the multiple Flink operators also include an input operator, and the acquisition module 501 can be specifically used to: in response to the start of a Flink task or in response to the completion of offline model training, the input operator acquires the configuration information corresponding to the offline model, or the input operator acquires the configuration information corresponding to the offline model periodically.

[0148] In one possible implementation, the processing module 502 can also be used to: determine whether the backtracking flag is backtracking before the data retrieval operator stops the existing consumption thread if the backtracking flag is backtracking; if so, broadcast configuration information to the input operator and receive the configuration information.

[0149] In one possible implementation, the configuration information also includes a configuration version, and the processing module 502 can also be used to: before the data pull operator stops the existing consumption thread and creates a new consumption thread, the data pull operator determines whether to update the local configuration of the data pull operator based on the configuration version and the local configuration version of the data pull operator; if yes, then update the local configuration of the data pull operator; if no, then do not update the local configuration of the data pull operator.

[0150] In one possible implementation, the configuration information also includes a configuration version, and the generation module 503 can also be used to: before the feature processing operator extracts features from the target data, the feature processing operator determines whether to update its local configuration based on the configuration version and its local configuration version; if yes, the local configuration of the feature processing operator is updated, and the local vocabulary of the feature processing operator is replaced with an offline vocabulary; if no, the local configuration of the feature processing operator is not updated.

[0151] In one possible implementation, the configuration information further includes a configuration version, and the generation module 503 can also be used to: before the topic filtering operator determines the target channel according to the specified channel, the topic filtering operator determines whether to update the local configuration of the topic filtering operator based on the configuration version and the local configuration version of the topic filtering operator; if yes, then the local configuration of the topic filtering operator is updated, and the sequence number corresponding to the instance in the topic filtering operator whose local configuration has been updated is recorded based on the configuration version; if no, then the local configuration of the topic filtering operator is not updated.

[0152] In one possible implementation, the generation module 503 can also be used to: after updating the sequence number corresponding to the locally configured instance in the topic filtering operator, the input operator updates the backtracking flag to non-backtracking when it is determined that the number of sequence numbers is the same as the parallelism of the topic filtering operator.

[0153] The apparatus of this disclosure can be used to execute the data backtracking method in any of the above method embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0154] Exemplary media

[0155] After introducing the methods of exemplary embodiments of this disclosure, the following references are made. Figure 6 The storage medium of the exemplary embodiments of this disclosure will be described.

[0156] Figure 6 This is a schematic diagram of a storage medium provided according to an embodiment of the present disclosure. (See reference...) Figure 6 As shown, the storage medium 600 stores a program product for implementing the above-described method according to embodiments of the present disclosure. This program product may be a portable compact disc read-only memory (CD-ROM) and includes program code, and can run on a terminal device, such as a personal computer. However, the program product of the present disclosure is not limited thereto.

[0157] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0158] A readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying readable program code. This propagated data signal may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium.

[0159] Program code for performing the operations disclosed herein can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can execute entirely on the user's computing device, partially on the user's computing device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing devices can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN).

[0160] Exemplary computing device

[0161] Having described the methods, media, and apparatus of exemplary embodiments of this disclosure, the following references... Figure 7 A computing device according to an exemplary embodiment of the present disclosure will be described.

[0162] Figure 7The computing device 700 shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments disclosed herein.

[0163] Figure 7 This is a schematic diagram of the structure of a computing device provided in an embodiment of the present disclosure, as shown below. Figure 7 As shown, the computing device 700 is manifested in the form of a general-purpose computing device. The components of the computing device 700 may include, but are not limited to: at least one processing unit 701, at least one storage unit 702, and a bus 703 connecting different system components (including the processing unit 701 and the storage unit 702). For example, the processing unit 701 may specifically be a processor, and the storage unit 702 stores computer execution instructions. The processing unit 701 executes the computer execution instructions stored in the storage unit 702 to implement the aforementioned data backtracking method.

[0164] The 703 bus includes a data bus, a control bus, and an address bus.

[0165] Storage unit 702 may include readable media in the form of volatile memory, such as random access memory (RAM) 7021 and / or cache memory 7022, and may further include readable media in the form of non-volatile memory, such as read-only memory (ROM) 7023.

[0166] Storage unit 702 may also include a program / utility 7025 having a set (at least one) program module 7024, such program module 7024 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0167] The computing device 700 can also communicate with one or more external devices 704 (e.g., keyboard, pointing device, etc.). This communication can be performed via the input / output (I / O) interface 705. Furthermore, the computing device 700 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via a network adapter 706. Figure 7 As shown, network adapter 706 communicates with other modules of computing device 700 via bus 703. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with computing device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0168] It should be noted that although several units / modules or sub-units / modules of the data backtracking device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0169] Furthermore, although the operations of the methods disclosed herein are described in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the operations shown must be performed to achieve the desired result. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0170] While the spirit and principles of this disclosure have been described with reference to several specific embodiments, it should be understood that this disclosure is not limited to the disclosed specific embodiments, and the division of aspects does not imply that features in these aspects cannot be combined for benefit; such division is merely for convenience of expression. This disclosure is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims.

Claims

1. A data backtracking method applied to a distributed data processing system, wherein multiple Flink operators run in the distributed data processing system, the multiple Flink operators including a data fetch operator, a feature processing operator, a topic filtering operator, and an output operator, the multiple Flink operators stream processing to implement Flink tasks, the topic filtering operator including a first channel and a second channel, and the data backtracking method comprising: Obtain the configuration information corresponding to the offline model. The configuration information includes the data backtracking start time, whether backtracking is enabled, the offline vocabulary address of the offline model, and the specified channel. If the backtracking flag is set to backtracking, the data retrieval operator stops the existing consumption thread and creates a new consumption thread. The new consumption thread retrieves the target data based on the data backtracking start time. The feature processing operator extracts features from the target data and determines whether the features are in the local vocabulary of the feature processing operator. The local vocabulary is an offline vocabulary obtained according to the address of the offline vocabulary. If present, the feature processing operator determines that the feature is sample data, sends the sample data and the configuration information to the downstream Flink operator of the feature processing operator, and outputs the sample data through the downstream Flink operator; If not, the feature processing operator increments the frequency corresponding to the feature by 1 to determine if the frequency is greater than a threshold; if the frequency is greater than the threshold, the feature is determined to be the sample data, and the sample data and the configuration information are sent to the downstream Flink operator of the feature processing operator. The topic filtering operator determines the target channel according to the specified channel, which is either the first channel or the second channel. The specified channel is the channel switched when the offline model training is completed. The topic filtering operator sends the sample data to the output operator corresponding to the target channel through the target channel; the output operator outputs sample data for updating the offline model.

2. The data backtracking method according to claim 1, wherein the plurality of Flink operators further includes an input operator, and the step of obtaining the configuration information corresponding to the offline model includes: In response to the start of the Flink task or the completion of offline model training, the input operator obtains the configuration information corresponding to the offline model, or the input operator periodically obtains the configuration information corresponding to the offline model.

3. The data backtracking method according to claim 2, further comprising, before the data fetch operator stops the existing consumption thread if the backtracking flag is set to backtracking: The input operator determines whether the backtracking flag is a backtracking flag; If so, the input operator broadcasts the configuration information, and the data retrieval operator receives the configuration information.

4. The data backtracking method according to any one of claims 1 to 3, wherein the configuration information further includes a configuration version, and before the data retrieval operator stops the existing consumption thread and creates a new consumption thread, it further includes: The data pull operator determines whether to update the local configuration of the data pull operator based on the configuration version and the local configuration version of the data pull operator; If so, update the local configuration of the data fetch operator; If not, the local configuration of the data fetch operator will not be updated.

5. The data backtracking method according to claim 1, wherein the configuration information further includes a configuration version, and before the feature processing operator extracts features from the target data, it further includes: The feature processing operator determines whether to update its local configuration based on the configuration version and the local configuration version of the feature processing operator. If so, update the local configuration of the feature processing operator and replace the local vocabulary of the feature processing operator with the offline vocabulary; If not, the local configuration of the feature processing operator will not be updated.

6. The data backtracking method according to claim 3, wherein the configuration information further includes a configuration version, and before the topic filtering operator determines the target channel based on the specified channel, it further includes: The topic filtering operator determines whether to update the local configuration of the topic filtering operator based on the configuration version and the local configuration version of the topic filtering operator; If so, update the local configuration of the topic filtering operator, and based on the configuration version, record the sequence number of the instance in the topic filtering operator that has updated the local configuration; If not, the local configuration of the topic filtering operator will not be updated.

7. The data backtracking method according to claim 6, after recording the sequence number corresponding to the instance whose local configuration has been updated in the topic filtering operator, further includes: When the input operator determines that the number of the sequence number is the same as the parallelism of the topic filtering operator, it updates the backtracking flag to "no backtracking".

8. A data backtracking device, applied to a distributed data processing system, wherein multiple Flink operators run in the distributed data processing system, the multiple Flink operators including a data fetch operator, a feature processing operator, a topic filtering operator, and an output operator, the multiple Flink operators stream processing to implement Flink tasks, the topic filtering operator including a first channel and a second channel, the data backtracking device comprising: The acquisition module is used to acquire the configuration information corresponding to the offline model. The configuration information includes the data backtracking start time, whether backtracking is enabled, the offline vocabulary address of the offline model, and the specified channel. The processing module is configured to, if the backtracking flag is set to backtracking, stop the existing consumption thread and create a new consumption thread, wherein the new consumption thread acquires the target data based on the data backtracking start time; A generation module is used to extract features from the target data by the feature processing operator and determine whether the features are in the local vocabulary of the feature processing operator, wherein the local vocabulary is an offline vocabulary obtained according to the address of the offline vocabulary. If present, the feature processing operator determines that the feature is sample data, sends the sample data and the configuration information to the downstream Flink operator of the feature processing operator, and outputs the sample data through the downstream Flink operator; If not, the feature processing operator increments the frequency corresponding to the feature by 1 to determine if the frequency is greater than a threshold; if the frequency is greater than the threshold, the feature is determined to be the sample data, and the sample data and the configuration information are sent to the downstream Flink operator of the feature processing operator. The topic filtering operator determines the target channel according to the specified channel, which is either the first channel or the second channel. The specified channel is the channel switched when the offline model training is completed. The topic filtering operator sends the sample data to the output operator corresponding to the target channel through the target channel; the output operator outputs sample data for updating the offline model.

9. The data backtracking device according to claim 8, wherein the plurality of Flink operators further includes an input operator, and the acquisition module is specifically used for: In response to the start of the Flink task or the completion of offline model training, the input operator obtains the configuration information corresponding to the offline model, or the input operator periodically obtains the configuration information corresponding to the offline model.

10. The data backtracking device according to claim 9, wherein the processing module is further configured to: Before the data fetch operator stops the existing consumption thread if the backtracking flag is set to backtracking, the input operator determines whether the backtracking flag is set to backtracking; if so, the input operator broadcasts the configuration information, and the data fetch operator receives the configuration information.

11. The data backtracking device according to any one of claims 8 to 10, wherein the configuration information further includes a configuration version, and the processing module is further configured to: Before the data fetch operator stops the existing consumer thread and creates a new consumer thread, the data fetch operator determines whether to update the local configuration of the data fetch operator based on the configuration version and the local configuration version of the data fetch operator. If so, update the local configuration of the data fetch operator; If not, the local configuration of the data fetch operator will not be updated.

12. The data backtracking device according to claim 8, wherein the configuration information further includes a configuration version, and the generation module is further configured to: Before the feature processing operator extracts features from the target data, the feature processing operator determines whether to update its local configuration based on the configuration version and the local configuration version of the feature processing operator. If so, update the local configuration of the feature processing operator and replace the local vocabulary of the feature processing operator with the offline vocabulary; If not, the local configuration of the feature processing operator will not be updated.

13. The data backtracking device according to claim 10, wherein the configuration information further includes a configuration version, and the generation module is further configured to: Before the topic filtering operator determines the target channel based on the specified channel, the topic filtering operator determines whether to update the local configuration of the topic filtering operator based on the configuration version and the local configuration version of the topic filtering operator; If so, update the local configuration of the topic filtering operator, and based on the configuration version, record the sequence number of the instance in the topic filtering operator that has updated the local configuration; If not, the local configuration of the topic filtering operator will not be updated.

14. The data backtracking device according to claim 13, wherein the generation module is further configured to: After updating the local configuration instance number in the topic filtering operator, the input operator updates the backtracking flag to no backtracking when it determines that the number of the number of the number of the number of the topic filtering operator is the same as the parallelism of the topic filtering operator.

15. A computing device, comprising: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the data backtracking method as described in any one of claims 1 to 7.

16. A storage medium storing computer program instructions, which, when executed, implement the data backtracking method as described in any one of claims 1 to 7.

17. A computer program product comprising a computer program that, when executed by a processor, implements the data backtracking method as described in any one of claims 1 to 7.

Images