Methods, apparatus, electronic devices and storage media for streaming data backtracking

Abstract

This invention discloses a method, apparatus, electronic device, and storage medium for streaming data backtracking. The method includes: placing acquired streaming data into a message queue and adding a time sequence tag to the streaming data; determining the offset information of the streaming data based on the time sequence tag; determining the streaming data to be migrated from the message queue and determining an incremental data file based on the streaming data to be migrated; wherein the incremental data file is used to record the streaming data to be migrated; updating the offset information of the streaming data to be migrated according to the storage path of the incremental data file, and determining the updated offset information of the streaming data to be migrated; determining the streaming data to be backtracked based on the updated offset information and the unupdated offset information of the remaining streaming data in the message queue. This technical solution can effectively support large-scale streaming data backtracking, improve the quality of streaming data backtracking, and better meet the backtracking needs of large-scale streaming data in practical applications.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, electronic device, and storage medium for streaming data backtracking. Background Technology

[0002] With the advent of the big data era, enterprises have increasingly higher requirements for the timeliness of data processing, and more and more scenarios are adopting real-time stream processing to improve data processing efficiency. Stream processing refers to a processing method specifically for streaming data. In production and testing environments, it is often necessary to backtrack streaming data over a certain historical period to resolve issues such as streaming data anomalies. Due to the characteristics of streaming data—its burstiness, disorder, and massive scale—backtracking is often difficult. In real-time stream processing, data is not stored on the local machine; it is processed immediately upon arrival. In this case, large-scale streaming data backtracking cannot be supported, resulting in poor backtracking quality and failing to meet the backtracking needs of large-scale streaming data in practical applications. Summary of the Invention

[0003] This invention provides a streaming data backtracking method, apparatus, electronic device, and storage medium, which can effectively support large-scale streaming data backtracking, improve the quality of streaming data backtracking, and better meet the backtracking needs of large-scale streaming data in practical applications.

[0004] According to one aspect of the present invention, a streaming data backtracking method is provided, the method comprising:

[0005] The acquired streaming data is placed into a message queue, and a time sequence tag is added to the streaming data; wherein, the time sequence tag is used to characterize the acquisition order of the streaming data;

[0006] Based on the time sequence label of the streaming data, the offset information of the streaming data is determined; wherein, the offset information is used to describe the time sequence label corresponding to the acquisition time of the streaming data;

[0007] The process involves determining the stream data to be migrated from the message queue and then determining an incremental data file based on the stream data to be migrated; wherein the incremental data file is used to record the stream data to be migrated.

[0008] Based on the storage path of the incremental data file, the offset information of the stream data to be migrated is updated to determine the updated offset information of the stream data to be migrated.

[0009] Based on the updated offset information and the unupdated offset information of the remaining stream data in the message queue, the stream data to be traced back is determined.

[0010] According to another aspect of the present invention, a streaming data backtracking device is provided, comprising:

[0011] A time sequence tag adding module is used to put the acquired streaming data into a message queue and add time sequence tags to the streaming data; wherein, the time sequence tags are used to characterize the acquisition order of the streaming data;

[0012] The offset information determination module is used to determine the offset information of the streaming data based on the time sequence label of the streaming data; wherein, the offset information is used to describe the time sequence label corresponding to the acquisition time of the streaming data;

[0013] An incremental data file determination module is used to determine the stream data to be migrated from the message queue and determine an incremental data file based on the stream data to be migrated; wherein, the incremental data file is used to record the stream data to be migrated;

[0014] The offset information determination module is used to update the offset information of the stream data to be migrated according to the storage path of the incremental data file, and determine the updated offset information of the stream data to be migrated.

[0015] The module for determining the stream data to be traced back is used to determine the stream data to be traced back based on the updated offset information and the unupdated offset information of the remaining stream data in the message queue.

[0016] According to another aspect of the present invention, a streaming data backtracking electronic device is provided, the electronic device comprising:

[0017] At least one processor; and

[0018] A memory communicatively connected to the at least one processor; wherein,

[0019] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the streaming data backtracking method according to any embodiment of the present invention.

[0020] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the streaming data backtracking method according to any embodiment of the present invention.

[0021] The technical solution of this invention involves placing the acquired streaming data into a message queue and adding time sequence tags to the streaming data; wherein, the time sequence tags are used to characterize the acquisition order of the streaming data; determining the offset information of the streaming data based on the time sequence tags; wherein, the offset information is used to describe the time sequence tags corresponding to the acquisition time of the streaming data; determining the streaming data to be migrated from the message queue, and determining an incremental data file based on the streaming data to be migrated; wherein, the incremental data file is used to record the streaming data to be migrated; updating the offset information of the streaming data to be migrated according to the storage path of the incremental data file, and determining the updated offset information of the streaming data to be migrated; determining the streaming data to be backtracked based on the updated offset information and the unupdated offset information of the remaining streaming data in the message queue. This technical solution can effectively support large-scale streaming data backtracking, improve the quality of streaming data backtracking, and better meet the backtracking needs of large-scale streaming data in practical applications.

[0022] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart of a streaming data backtracking method provided in Embodiment 1 of the present invention;

[0025] Figure 2 This is a flowchart of a streaming data backtracking method provided in Embodiment 2 of the present invention;

[0026] Figure 3 This is a schematic diagram of the structure of a streaming data backtracking device according to Embodiment 3 of the present invention;

[0027] Figure 4 This is a schematic diagram of the structure of an electronic device that implements a streaming data backtracking method according to an embodiment of the present invention. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first," "second," "target," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] Example 1

[0031] Figure 1 This is a flowchart of a streaming data backtracking method provided in Embodiment 1 of the present invention. This embodiment is applicable to the situation of fast backtracking of large-scale streaming data. The method can be executed by a streaming data backtracking device, which can be implemented in hardware and / or software. The streaming data backtracking device can be configured in an electronic device with data processing capabilities. Figure 1 As shown, the method includes:

[0032] S110, the acquired streaming data is placed into a message queue, and a time sequence label is added to the streaming data; wherein, the time sequence label is used to characterize the acquisition order of the streaming data.

[0033] Streaming data refers to data continuously generated from thousands of data sources, characterized by high throughput and low latency. Generally, streaming data can be viewed as a dynamic data set that grows infinitely over time. Streaming data has wide applications in various fields, such as network monitoring, sensor networks, aerospace, meteorological monitoring and control, and financial services; it is suitable for multiple different application scenarios, such as real-time monitoring, dynamic tracking, and data operations. A message queue is a queue used to store messages, where messages follow a first-in, first-out (FIFO) principle. Time sequence labels can be used to characterize the order in which streaming data is acquired.

[0034] In this embodiment, when streaming data is detected, the streaming data is first acquired and stored in a message queue, and a corresponding time sequence tag is added to the streaming data. For example, an auto-incrementing sequence number can be dynamically added as the time sequence tag of the streaming data according to the order in which the streaming data enters the message queue. It can be understood that the larger the value of the time sequence tag, the later the corresponding streaming data entered the message queue.

[0035] S120, determine the offset information of the streaming data based on the time sequence label of the streaming data; wherein, the offset information is used to describe the time sequence label corresponding to the time of streaming data acquisition.

[0036] The offset information can be used to describe the time sequence label corresponding to the streaming data acquisition time. Specifically, the offset information can include the streaming data acquisition time and the time sequence label. For example, the offset information of streaming data can be represented as <streaming data acquisition time, streaming data time sequence label>.

[0037] S130, determine the stream data to be migrated from the message queue, and determine the incremental data file based on the stream data to be migrated; wherein, the incremental data file is used to record the stream data to be migrated.

[0038] The stream data to be migrated refers to the stream data waiting to be migrated in the message queue. Incremental data files can be used to record this stream data. It should be noted that because message queues have limited storage space, they can typically only cache real-time stream data within a certain time range (e.g., 7 days). Therefore, it is necessary to monitor the stream data caching time in the message queue in real time and migrate and store stream data that is about to exceed the message queue caching time range (e.g., stream data older than 6 days) to avoid data loss. Stream data that is about to exceed the message queue caching time range can be considered as stream data to be migrated. Furthermore, the stream data to be migrated corresponding to each offset can be written into an incremental data file.

[0039] S140, based on the storage path of the incremental data file, update the offset information of the data to be migrated and determine the updated offset information of the data to be migrated.

[0040] Among them, updating offset information can refer to the new offset information obtained after updating the original offset information of the data to be migrated.

[0041] In this embodiment, the offset information of the stream data to be migrated can be updated according to the storage path of the incremental data file to determine the updated offset information of the stream data to be migrated. Optionally, updating the offset information of the stream data to be migrated according to the storage path of the incremental data file includes: keeping the stream data acquisition time in the offset information of the stream data to be migrated unchanged, and replacing the time sequence label in the offset information of the stream data to be migrated with the storage path of the incremental data file to update the offset information of the stream data to be migrated. For example, the updated offset information of the stream data to be migrated can be represented as <acquisition time of the stream data to be migrated, storage path of the incremental data file>.

[0042] This solution allows for updating the offset information of the stream data to be migrated based on the storage path of the incremental data file. This enables the stream data to be migrated to be retrieved quickly, accurately, and conveniently by using the storage path in the updated offset information.

[0043] S150, determine the stream data to be backtracked based on the updated offset information and the unupdated offset information of the remaining stream data in the message queue.

[0044] Here, "remaining stream data" can refer to stream data in the message queue other than the stream data to be migrated. "Unupdated offset information" refers to offset information that has not been updated, i.e., the offset information corresponding to the remaining stream data in the message queue. "Stream data to be backtracked" refers to stream data waiting to be backtracked.

[0045] In this embodiment, optionally, determining the stream data to be backtracked based on the updated offset information and the unupdated offset information of the remaining stream data in the message queue includes: matching the backtracking time of the stream data to be backtracked based on the stream data acquisition time in the updated offset information and the stream data acquisition time in the unupdated offset information; determining the target offset information of the stream data to be backtracked based on the matching result; and determining the stream data to be backtracked based on the target offset information.

[0046] The backtracking time can be a single point in time (e.g., 00:00 on October 1, 2022) or a time period (e.g., 08:00 to 10:00 on October 1, 2022). This embodiment does not impose any limitations on this and can determine it according to actual application requirements. The target offset information can refer to the offset information corresponding to the stream data to be backtracked.

[0047] In this embodiment, firstly, based on the backtracking time of the stream data to be backtracked, the stream data acquisition time in the updated offset information and the unupdated offset information is compared one by one to determine the stream data acquisition time that matches the backtracking time of the stream data to be backtracked. Then, the offset information corresponding to the successfully matched stream data acquisition time is determined as the target offset information of the stream data to be backtracked. Subsequently, the stream data to be backtracked can be determined based on the target offset information. It should be noted that the target offset information has one of the following three cases: (1) the target offset information only contains updated offset information; (2) the target offset information only contains unupdated offset information; (3) the target offset information contains both updated offset information and unupdated offset information.

[0048] In this embodiment, optionally, determining the stream data to be traced back based on the target offset information includes: if the target offset information is updated offset information, determining the target incremental data file based on the target offset information, and determining the stream data to be traced back based on the target incremental data file; if the target offset information is not updated offset information, determining the target stream data from the message queue based on the target offset information, and determining the stream data to be traced back based on the target stream data; if the target offset information includes both updated offset information and not updated offset information, determining the target incremental data file based on the updated offset information in the target offset information, determining the stream data to be traced back based on the target incremental data file, and determining the target stream data from the message queue based on the not updated offset information in the target offset information, and determining the stream data to be traced back based on the target stream data.

[0049] The target incremental data file can refer to the incremental data file corresponding to the target offset information. The target stream data can refer to the stream data corresponding to the target offset information.

[0050] In this embodiment, the stream data to be backtracked can be determined based on the composition of the target offset information. Specifically, if the target offset information only contains updated offset information, the target incremental data file (i.e., the incremental data file corresponding to the successfully matched updated offset information) can be determined based on the target offset information (i.e., the successfully matched updated offset information), and then the stream data in the target incremental data file can be determined as the stream data to be backtracked. If the target offset information only contains non-updated offset information, the target stream data (i.e., the stream data corresponding to the successfully matched non-updated offset information) can be determined from the message queue based on the target offset information (i.e., the successfully matched non-updated offset information), and the target stream data can be determined as the stream data to be backtracked. If the target offset information includes both updated and non-updated offset information, the stream data to be backtracked can be determined through two methods based on the updated and non-updated offset information in the target offset information, respectively. The processes for determining the stream data to be backtracked based on the updated offset information and the non-updated offset information can be referred to the determination processes in the two cases described above, and will not be repeated here.

[0051] This solution, through this configuration, enables rapid backtracking of short-term stream data in the message queue based on the unupdated offset information in the target offset information, and simultaneously enables rapid backtracking of long-term stream data in the incremental data file based on the updated offset information in the target offset information, thus better achieving rapid backtracking of large-scale stream data.

[0052] The technical solution of this invention involves placing the acquired streaming data into a message queue and adding time sequence tags to the streaming data; wherein, the time sequence tags are used to characterize the acquisition order of the streaming data; determining the offset information of the streaming data based on the time sequence tags; wherein, the offset information is used to describe the time sequence tags corresponding to the acquisition time of the streaming data; determining the streaming data to be migrated from the message queue, and determining an incremental data file based on the streaming data to be migrated; wherein, the incremental data file is used to record the streaming data to be migrated; updating the offset information of the streaming data to be migrated according to the storage path of the incremental data file, and determining the updated offset information of the streaming data to be migrated; determining the streaming data to be backtracked based on the updated offset information and the unupdated offset information of the remaining streaming data in the message queue. This technical solution can effectively support large-scale streaming data backtracking, improve the quality of streaming data backtracking, and better meet the backtracking needs of large-scale streaming data in practical applications.

[0053] Example 2

[0054] Figure 2This is a flowchart of a streaming data backtracking method provided in Embodiment 2 of the present invention. This embodiment is based on the above embodiment and optimized. Specifically, the optimization is as follows: determining the offset information of the streaming data according to the time sequence label of the streaming data includes: obtaining the streaming data with the largest time sequence label from the message queue as candidate streaming data based on a preset time interval; and determining the candidate offset information of the candidate streaming data according to the acquisition time of the candidate streaming data and the time sequence label of the candidate streaming data.

[0055] like Figure 2 As shown, the method in this embodiment specifically includes the following steps:

[0056] S210, the acquired streaming data is placed into a message queue, and a time sequence label is added to the streaming data; wherein, the time sequence label is used to characterize the acquisition order of the streaming data.

[0057] S220: Based on a preset time interval, retrieve the stream data with the largest time sequence label from the message queue as candidate stream data.

[0058] The preset time interval can refer to a pre-defined time interval for determining candidate stream data. For example, the preset time interval can be set to 1 minute. Candidate stream data can refer to stream data in the message queue waiting to have its offset information determined.

[0059] S230, determine the candidate offset information of the candidate stream data based on the acquisition time of the candidate stream data and the time sequence label of the candidate stream data.

[0060] Here, candidate offset information refers to the offset information corresponding to candidate stream data. For example, candidate offset information can be represented as <acquisition time of candidate stream data, time sequence label of candidate stream data>. It should be noted that in this embodiment, offset information is not determined for all stream data in the message queue; only the offset information corresponding to candidate stream data (i.e., candidate offset information) needs to be determined.

[0061] S240, determine the stream data to be migrated from the message queue, and determine the incremental data file based on the stream data to be migrated; wherein, the incremental data file is used to record the stream data to be migrated.

[0062] In this embodiment, optionally, determining the stream data to be migrated from the message queue and determining the incremental data file based on the stream data to be migrated includes: determining the stream data to be migrated from the message queue based on a preset expiration time; determining the offset information of the stream data to be migrated based on candidate offset information; and determining the incremental data file based on the offset information of the stream data to be migrated.

[0063] The preset expiration time can refer to a pre-defined expiration time for streaming data, which can be used to represent a time threshold that is about to exceed the maximum value of the message queue cache time range. For example, the preset expiration time can be set to the day before the maximum value of the message queue cache time range.

[0064] In this embodiment, stream data that has exceeded a preset expiration time in the message queue can be identified as stream data to be migrated. Then, candidate stream data is searched in the stream data to be migrated, and the candidate offset information corresponding to the candidate stream data in the stream data to be migrated is determined as the offset information of the stream data to be migrated. Then, the incremental data file is determined based on the offset information of the stream data to be migrated.

[0065] This solution, through this configuration, can record streaming data that is about to exceed the message queue cache time range using incremental data files, effectively avoiding the loss of data that is about to expire in the message queue, and better supporting the backtracking needs of large-scale streaming data.

[0066] In this embodiment, optionally, determining the incremental data file based on the offset information of the stream data to be migrated includes: sorting the offset information of the stream data to be migrated according to the stream data acquisition time in the offset information of the stream data to be migrated; and based on the sorting result, determining the incremental data file corresponding to the previous offset information according to the stream data to be migrated corresponding to the previous offset information and the stream data to be migrated before the stream data to be migrated corresponding to the subsequent offset information adjacent to the previous offset information.

[0067] The preceding and following offset information can be used to characterize the order of offset information in the sorting result. In this embodiment, firstly, the offset information of the stream data to be migrated is sorted according to the stream data acquisition time in the offset information, resulting in a sorting result of the offset information of the stream data to be migrated arranged in chronological order. Then, based on the stream data to be migrated corresponding to the preceding offset information in the sorting result, and the stream data to be migrated before the stream data corresponding to the following offset information adjacent to the preceding offset information, the incremental data file corresponding to the preceding offset information is determined.

[0068] It should be noted that each offset of the data to be migrated needs to generate a corresponding incremental data file, and each incremental data file records the data to be migrated from the current offset to the next offset.

[0069] This solution reduces the number of incremental data files by writing the migration stream data between the current offset information and the next offset information into an incremental data file, which facilitates the storage and management of incremental data files.

[0070] S250, based on the storage path of the incremental data file, update the offset information of the data to be migrated and determine the updated offset information of the data to be migrated.

[0071] S260, determine the stream data to be backtracked based on the updated offset information and the unupdated offset information of the remaining stream data in the message queue.

[0072] In this embodiment, if the backtracking time is a single point in time and there is no corresponding offset information, the closest streaming data acquisition time before that point can be searched from both the updated and unupdated offset information, and this time is used as the starting point for updating the backtracking time. Similarly, the closest streaming data acquisition time after that point can be searched from both the updated and unupdated offset information, and this time is used as the ending point for updating the backtracking time. Here, the updated backtracking time can refer to a new backtracking time obtained after expanding the time range. Then, based on the offset information matching the starting and ending points of the updated backtracking time, the streaming data to be backtracked is determined. It should be noted that the updated backtracking time includes the original backtracking time point; that is, the backtracking data corresponding to the updated backtracking time includes all the streaming data required for the original backtracking time point, ensuring that the streaming data backtracking requirements are fully met.

[0073] If the backtracking time is a time period, and there is no corresponding offset information for the start and / or end times of the backtracking time, then the closest stream data acquisition time before the start time can be searched from the updated and unupdated offset information, and this time can be used as the update start time for updating the backtracking time. Similarly, the closest stream data acquisition time after the end time can be searched from the updated and unupdated offset information, and this time can be used as the update end time for updating the backtracking time. Here, the update start time can refer to the start time corresponding to the update backtracking time, and the update end time can refer to the end time corresponding to the update backtracking time. Then, the stream data to be backtracked is determined based on the offset information matching the update start time and / or update end time. It should be noted that the updated backtracking time includes the entire time range of the original backtracking time; that is, the backtracking data corresponding to the updated backtracking time includes all the stream data required for the original backtracking time, ensuring that the stream data backtracking requirements are fully met.

[0074] The technical solution of this invention retrieves the stream data with the largest time sequence tag from the message queue as candidate stream data based on a preset time interval; and determines the candidate offset information of the candidate stream data based on the retrieval time and the time sequence tag of the candidate stream data. This technical solution, by determining the candidate stream data and the corresponding candidate offset information based on a preset time interval, can effectively support large-scale stream data backtracking, improve the quality of stream data backtracking, reduce the number of incremental data files, and facilitate the storage and management of incremental data files, thereby better meeting the backtracking needs of large-scale stream data in practical applications.

[0075] Example 3

[0076] Figure 3 This is a schematic diagram of a streaming data backtracking device provided in Embodiment 3 of the present invention. This device can execute the streaming data backtracking method provided in any embodiment of the present invention, and possesses the corresponding functional modules and beneficial effects of the method execution. For example... Figure 3 As shown, the device includes:

[0077] The time sequence tag adding module 310 is used to put the acquired streaming data into a message queue and add time sequence tags to the streaming data; wherein, the time sequence tags are used to characterize the acquisition order of the streaming data;

[0078] The offset information determination module 320 is used to determine the offset information of the streaming data based on the time sequence label of the streaming data; wherein, the offset information is used to describe the time sequence label corresponding to the acquisition time of the streaming data;

[0079] The incremental data file determination module 330 is used to determine the stream data to be migrated from the message queue and determine the incremental data file based on the stream data to be migrated; wherein, the incremental data file is used to record the stream data to be migrated;

[0080] The offset information determination module 340 is used to update the offset information of the stream data to be migrated according to the storage path of the incremental data file, and determine the updated offset information of the stream data to be migrated.

[0081] The stream data to be traced back module 350 is used to determine the stream data to be traced back based on the updated offset information and the unupdated offset information of the remaining stream data in the message queue.

[0082] Optionally, the offset information determination module 320 is specifically used for:

[0083] Based on a preset time interval, the stream data with the largest time sequence label is obtained from the message queue as candidate stream data;

[0084] Based on the acquisition time of the candidate stream data and the time sequence label of the candidate stream data, the candidate offset information of the candidate stream data is determined.

[0085] Optionally, the incremental data file determination module 330 includes:

[0086] The data to be migrated is determined by a data determination unit, which determines the data to be migrated from the message queue based on a preset expiration time.

[0087] An offset information determination unit is used to determine the offset information of the stream data to be migrated based on the candidate offset information;

[0088] An incremental data file determination unit is used to determine an incremental data file based on the offset information of the data to be migrated.

[0089] Optionally, the incremental data file determination unit is specifically used for:

[0090] Based on the data acquisition time in the offset information of the data to be migrated, sort the offset information of the data to be migrated.

[0091] Based on the sorting results, the incremental data file corresponding to the preceding offset information is determined according to the data to be migrated corresponding to the preceding offset information and the data to be migrated before the data to be migrated corresponding to the following offset information adjacent to the preceding offset information.

[0092] Optionally, the update offset information determination module 340 is specifically used for:

[0093] The stream data acquisition time in the offset information of the stream data to be migrated is kept unchanged, while the time sequence label in the offset information of the stream data to be migrated is replaced with the storage path of the incremental data file to update the offset information of the stream data to be migrated.

[0094] Optionally, the data to be traced back stream determination module 350 includes:

[0095] The backtracking time matching unit is used to match the backtracking time of the stream data to be backtracked based on the stream data acquisition time in the updated offset information and the stream data acquisition time in the unupdated offset information.

[0096] The target offset information determination unit is used to determine the target offset information of the stream data to be traced back based on the matching result;

[0097] The data to be traced back is determined by a data determination unit, which determines the data to be traced back based on the target offset information.

[0098] Optionally, the data to be traced back stream determination unit is specifically used for:

[0099] If the target offset information is update offset information, determine the target incremental data file based on the target offset information, and determine the stream data to be traced back based on the target incremental data file;

[0100] If the target offset information is not updated offset information, the target stream data is determined from the message queue based on the target offset information, and the stream data to be traced is determined based on the target stream data;

[0101] If the target offset information includes both updated offset information and non-updated offset information, the target incremental data file is determined based on the updated offset information in the target offset information, the stream data to be backtracked is determined based on the target incremental data file, and the target stream data is determined from the message queue based on the non-updated offset information in the target offset information, and the stream data to be backtracked is determined based on the target stream data.

[0102] The streaming data backtracking device provided in this embodiment of the invention can execute a streaming data backtracking method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0103] Example 4

[0104] Figure 4 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0105] like Figure 4As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0106] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0107] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as streaming data backtracking methods.

[0108] In some embodiments, the streaming data backtracking method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the streaming data backtracking method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the streaming data backtracking method by any other suitable means (e.g., by means of firmware).

[0109] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0110] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0111] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0112] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0113] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0114] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0115] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0116] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A streaming data backtracking method, characterized in that, The method includes: The acquired streaming data is placed into a message queue, and a time sequence tag is added to the streaming data; wherein, the time sequence tag is used to characterize the acquisition order of the streaming data; Based on the time sequence label of the streaming data, the offset information of the streaming data is determined; wherein, the offset information is used to describe the time sequence label corresponding to the acquisition time of the streaming data; The process involves determining the stream data to be migrated from the message queue and then determining an incremental data file based on the stream data to be migrated; wherein the incremental data file is used to record the stream data to be migrated. Keep the stream data acquisition time in the offset information of the stream data to be migrated unchanged, and replace the time sequence label in the offset information of the stream data to be migrated with the storage path of the incremental data file to update the offset information of the stream data to be migrated and determine the updated offset information of the stream data to be migrated. The backtracking time of the stream data to be backtracked is matched based on the stream data acquisition time in the updated offset information and the stream data acquisition time in the unupdated offset information of the remaining stream data in the message queue. The target offset information of the stream data to be traced back is determined based on the matching results; If the target offset information is update offset information, determine the target incremental data file based on the target offset information, and determine the stream data to be traced back based on the target incremental data file; If the target offset information is not updated offset information, the target stream data is determined from the message queue based on the target offset information, and the stream data to be traced is determined based on the target stream data; If the target offset information includes both updated offset information and non-updated offset information, the target incremental data file is determined based on the updated offset information in the target offset information, the stream data to be backtracked is determined based on the target incremental data file, and the target stream data is determined from the message queue based on the non-updated offset information in the target offset information, and the stream data to be backtracked is determined based on the target stream data.

2. The method according to claim 1, characterized in that, Based on the time sequence label of the streaming data, determine the offset information of the streaming data, including: Based on a preset time interval, the stream data with the largest time sequence label is obtained from the message queue as candidate stream data; Based on the acquisition time of the candidate stream data and the time sequence label of the candidate stream data, the candidate offset information of the candidate stream data is determined.

3. The method according to claim 2, characterized in that, Determining the stream data to be migrated from the message queue, and determining the incremental data file based on the stream data to be migrated, includes: The data to be migrated is determined from the message queue based on a preset expiration time; The offset information of the stream data to be migrated is determined based on the candidate offset information; The incremental data file is determined based on the offset information of the stream data to be migrated.

4. The method according to claim 3, characterized in that, The incremental data file is determined based on the offset information of the stream data to be migrated, including: Based on the data acquisition time in the offset information of the data to be migrated, sort the offset information of the data to be migrated. Based on the sorting results, the incremental data file corresponding to the preceding offset information is determined according to the data to be migrated corresponding to the preceding offset information and the data to be migrated before the data to be migrated corresponding to the following offset information adjacent to the preceding offset information.

5. A streaming data backtracking device, characterized in that, The device includes: A time sequence tag adding module is used to put the acquired streaming data into a message queue and add time sequence tags to the streaming data; wherein, the time sequence tags are used to characterize the acquisition order of the streaming data; The offset information determination module is used to determine the offset information of the streaming data based on the time sequence label of the streaming data; wherein, the offset information is used to describe the time sequence label corresponding to the acquisition time of the streaming data; An incremental data file determination module is used to determine the stream data to be migrated from the message queue and determine an incremental data file based on the stream data to be migrated; wherein, the incremental data file is used to record the stream data to be migrated; The offset information determination module is used to keep the stream data acquisition time in the offset information of the stream data to be migrated unchanged, and at the same time replace the time sequence tag in the offset information of the stream data to be migrated with the storage path of the incremental data file to update the offset information of the stream data to be migrated and determine the updated offset information of the stream data to be migrated. The module for determining backtracking stream data is used for: The backtracking time of the stream data to be backtracked is matched based on the stream data acquisition time in the updated offset information and the stream data acquisition time in the unupdated offset information of the remaining stream data in the message queue. The target offset information of the stream data to be traced back is determined based on the matching results; If the target offset information is update offset information, determine the target incremental data file based on the target offset information, and determine the stream data to be traced back based on the target incremental data file; If the target offset information is not updated offset information, the target stream data is determined from the message queue based on the target offset information, and the stream data to be traced is determined based on the target stream data; If the target offset information includes both updated offset information and non-updated offset information, the target incremental data file is determined based on the updated offset information in the target offset information, the stream data to be backtracked is determined based on the target incremental data file, and the target stream data is determined from the message queue based on the non-updated offset information in the target offset information, and the stream data to be backtracked is determined based on the target stream data.

6. A streaming data backtracking electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the streaming data backtracking method according to any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the streaming data backtracking method according to any one of claims 1-4.

Images