Data stream processing method and device, processor and electronic device

Abstract

The application provides a data stream processing method and device, a processor and an electronic device. In the case that a data stream is received upstream of a preset data processing engine, it is determined whether the data stream is a target data stream. In the case that the data stream is the target data stream, an asynchronous query operation is performed according to the target data stream. It is judged whether the asynchronous query operation is successfully performed, and in the case of successful execution, the target data stream is sent downstream of the preset data processing engine. In the case of failed execution, the target data stream is written into a replay sub-engine to perform a data stream replay operation. The technical problem of low execution efficiency of asynchronous query in the prior art Flink open source stream processing framework is solved, and the technical effect of improving the accuracy of asynchronous query is achieved by performing data replay processing on the data stream.

Description

Technical Field

[0001] This application relates to the field of data enhancement, and more specifically, to a data stream processing method and apparatus, a processor, and an electronic device. Background Technology

[0002] In real-time data processing of real-time data streams, data augmentation is inevitably required. Data augmentation involves retrieving data from external data storage media and adding the necessary data back into the original data. In this scenario, Flink natively provides an asynchronous query component—AsyncDataStream—to send asynchronous query requests. This component implements a Future waiting queue. Whenever the component receives a piece of data, it executes an asynchronous operation and places this operation into the Future. After the asynchronous operation completes, the result is placed into the Future, from which the component retrieves the result.

[0003] Currently, asynchronous queries in this asynchronous query component need to be implemented by the developers themselves. There's no unified interface standard for this logic, which can lead to inconsistent code across different developers, causing significant maintenance challenges later on.

[0004] Furthermore, the code logic for asynchronous requests corresponding to asynchronous query components in existing technologies is not reusable. Different data augmentation objects lack unified data retrieval and processing rules.

[0005] In real-time computing scenarios, external data may not be synchronized in a timely manner, causing asynchronous query failures.

[0006] Finally, there is no unified way to handle exceptions in the native components.

[0007] No effective solutions have yet been proposed for the aforementioned problems in the existing technology. Summary of the Invention

[0008] The main objective of this application is to provide a data stream processing method, apparatus, processor, and electronic device to solve the technical problem that asynchronous queries in the existing Flink open-source stream processing framework lack specific execution logic and do not handle exceptions after asynchronous queries fail, resulting in low execution efficiency of asynchronous queries.

[0009] According to one aspect of the present invention, a data stream processing method is provided, comprising: when a data stream is received upstream of a preset data processing engine, determining whether the data stream is a target data stream, wherein the target data stream is a data stream for which an asynchronous query operation is to be performed; if the data stream is a target data stream, performing an asynchronous query operation based on the target data stream; determining whether the asynchronous query operation is successfully executed, and if successful, sending the target data stream downstream of the preset data processing engine; if unsuccessful, controlling the target data stream to write to a replay sub-engine to perform a data stream replay operation, wherein the data stream replay operation is an operation to re-control the data stream to perform the asynchronous query operation.

[0010] Furthermore, when a data stream is received upstream of the preset data processing engine, determining whether the data stream is the target data stream includes: identifying preset keywords included in the data stream, wherein the preset keywords are set at preset character positions in the data stream; based on the preset keywords, finding instances corresponding to the preset keywords, and obtaining multiple attribute data contained in the data stream defined in the instances; parsing the current attribute data contained in the data stream, comparing the current attribute data with multiple attribute data, and determining whether some attribute data is missing in the data stream; if the data stream is missing some attribute data, determining that the data stream is the target data stream; if the current attribute data of the data stream is consistent with multiple attribute data, determining that the data stream is a non-target data stream.

[0011] Furthermore, based on preset keywords, the process involves finding instances corresponding to those keywords, including: acquiring multiple data streams and determining multiple keywords corresponding to each data stream, wherein each type of data stream corresponds one-to-one with a keyword; determining multiple attribute data contained in each data stream, and links to at least one external storage medium containing the multiple attribute data; generating a configuration file based on the keywords, multiple attribute data, and links to at least one external storage medium; instantiating the configuration file using JAVA reflection to generate instances corresponding to the data streams; obtaining multiple instances corresponding to multiple types of data streams, and saving these instances to a preset hash map.

[0012] Furthermore, when the data stream is the target data stream, an asynchronous query operation is performed based on the target data stream, including: determining a preset keyword contained in the data stream, and finding an instance corresponding to the preset keyword based on the preset keyword, wherein the preset keyword is set at a preset character position in the data stream; determining the missing attribute data corresponding to the data stream based on multiple attribute data contained in the data stream as defined in the instance, wherein the missing attribute data is at least one missing attribute data in the data stream; finding a link to the external storage medium corresponding to the missing attribute data based on the instance; and retrieving the missing attribute data from the external storage medium based on the link to the external storage medium.

[0013] Furthermore, it determines whether the asynchronous query operation was executed successfully, and if it was executed successfully, sends the target data stream to the downstream of the preset data processing engine, including: if the missing attribute data is successfully obtained from the external storage medium, it controls the missing attribute data to be written into the target data stream and writes the preset label into the target data stream; and sends the target data stream containing the missing attribute data and the preset label to the downstream of the preset data processing engine.

[0014] Furthermore, in the event of execution failure, after controlling the target data stream to write to the replay sub-engine to perform the data stream replay operation, the method includes: after determining that the target data stream has failed to perform the data stream replay operation, writing the target data stream to the replay sub-engine again; determining the cumulative number of times the target data stream has been written to the replay sub-engine, and if the cumulative number is greater than a preset number, sending the target data stream to the downstream of the preset data processing engine.

[0015] According to another aspect of the present invention, a data stream processing apparatus is also provided, comprising: a first determining unit, configured to determine whether a data stream is a target data stream when a data stream is received upstream of a preset data processing engine, wherein the target data stream is a data stream for which an asynchronous query operation is to be performed; a first executing unit, configured to execute an asynchronous query operation based on the target data stream when the data stream is the target data stream; a first judging unit, configured to judge whether the asynchronous query operation is successfully executed, and if the execution is successful, to send the target data stream downstream of the preset data processing engine; and a first controlling unit, configured to control the target data stream to be written into a replay sub-engine to perform a data stream replay operation if the execution fails, wherein the data stream replay operation is an operation to re-control the data stream to perform the asynchronous query operation.

[0016] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored program, wherein the program executes a data stream processing method.

[0017] According to another aspect of the present invention, a processor is also provided, which is used to run a program, wherein the program executes a data stream processing method during runtime.

[0018] According to another aspect of the present invention, an electronic device is also provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include a processing method for performing a data stream.

[0019] In this embodiment of the invention, when a data stream is received upstream of a preset data processing engine, it is determined whether the data stream is a target data stream, wherein the target data stream is the data stream for which an asynchronous query operation is to be performed. If the data stream is the target data stream, the asynchronous query operation is performed based on the target data stream. It is then determined whether the asynchronous query operation is successfully executed, and if successful, the target data stream is sent downstream of the preset data processing engine. If the execution fails, the target data stream is controlled to be written to a replay sub-engine to perform a data stream replay operation, wherein the data stream replay operation is an operation to re-control the data stream to execute the asynchronous query operation. This solves the technical problem in the prior art where asynchronous queries in the Flink open-source stream processing framework lack specific execution logic and do not handle exceptions after asynchronous query failures, resulting in low execution efficiency of asynchronous queries. By performing data replay processing on asynchronous queries, the technical effect of improving the accuracy of asynchronous queries is achieved. Attached Figure Description

[0020] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings:

[0021] Figure 1 This is a flowchart of a data stream processing method provided according to an embodiment of this application;

[0022] Figure 2 This is a schematic diagram of a data stream processing method provided in an embodiment of this application;

[0023] Figure 3 This is a schematic diagram of a data stream processing apparatus provided according to an embodiment of this application. Detailed Implementation

[0024] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0025] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0026] It should be noted that the terms "first," "first," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such use of these terms can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0027] It should be understood that when an element (such as a layer, film, region, or substrate) is described as being "on" another element, the element may be directly on the other element, or there may be an intermediate element present. Furthermore, in the specification and claims, when an element is described as being "connected" to another element, the element may be "directly connected" to the other element, or "connected" to the other element via a third element.

[0028] For ease of description, the following explains some of the nouns or terms used in the embodiments of this application:

[0029] Flink: Apache Flink is an open-source stream processing framework developed by the Apache Software Foundation. Its core is a distributed streaming data stream engine written in Java and Scala. Flink executes arbitrary streaming data programs in a data-parallel and pipelined manner. Flink's pipelined runtime system can execute both batch and stream processing programs. Furthermore, Flink's runtime itself also supports the execution of iterative algorithms.

[0030] Data augmentation is a process of enhancing existing data by providing complete data. For example, in a business system's order table, only the customer's ID number is often stored, not their name. The customer's name is stored in the customer information table. During data processing, the customer's name is often extracted and stored in a summary table, which is an enhancement of the customer information.

[0031] As mentioned in the background section, the asynchronous query in the existing Flink open-source stream processing framework lacks specific execution logic and does not handle exceptions after asynchronous query failure, resulting in low execution efficiency of asynchronous queries. To solve the above problems, in a typical embodiment of this application, a data stream processing method and apparatus, processor, and electronic device are provided.

[0032] According to an embodiment of this application, a method for processing a data stream is provided.

[0033] Figure 1 This is a flowchart of a data stream processing method provided according to an embodiment of this application. For example... Figure 1 As shown, the method includes the following steps:

[0034] Step S101: When a data stream is received upstream of the preset data processing engine, determine whether the data stream is the target data stream, wherein the target data stream is the data stream for which an asynchronous query operation is to be performed;

[0035] Preferably, the preset data processing engine in this application is the Flink data stream engine. In this data stream engine, the Indictor interface is defined to check whether the data stream received from the upstream needs to look up relevant data from external storage.

[0036] Step S102: If the data stream is the target data stream, perform an asynchronous query operation based on the target data stream;

[0037] Step S103: Determine whether the asynchronous query operation was executed successfully, and if it was executed successfully, send the target data stream to the downstream of the preset data processing engine;

[0038] Step S104: In the event of execution failure, control the target data stream to write to the replay sub-engine to perform a data stream replay operation, wherein the data stream replay operation is an operation to re-control the data stream to perform an asynchronous query operation.

[0039] As mentioned above, this application also defines a data stream replay operation, which replays the data stream in the event that the asynchronous query fails to execute on the target data stream, in order to maximize the chances that the asynchronous query operation can be successfully executed on the data stream.

[0040] In this application, a data stream processing method as described above is provided, and a replay function is defined to mitigate the failure of asynchronous queries.

[0041] In one optional embodiment, when a data stream is received upstream of a preset data processing engine, determining whether the data stream is a target data stream includes: determining preset keywords included in the data stream, wherein the preset keywords are set at preset character positions in the data stream; based on the preset keywords, finding an instance corresponding to the preset keywords, and obtaining multiple attribute data contained in the data stream defined in the instance; parsing the current attribute data contained in the data stream, comparing the current attribute data with multiple attribute data, and determining whether some attribute data is missing in the data stream; if the data stream is missing some attribute data, determining that the data stream is a target data stream; if the current attribute data of the data stream is consistent with multiple attribute data, determining that the data stream is a non-target data stream.

[0042] As described above, this application defines an Indictor interface, which defines three methods: isCertainInpayType (determines whether it is an asynchronous query), queryDataFromDb (defines how to query data from the database), and hasInpayData (determines whether data has been retrieved from the database).

[0043] The implementation class of the Indicator will be configured in a YAML configuration file, as follows:

[0044]

[0045] When a Flink task starts, it looks up the configuration file, finds this configuration, and then uses Java reflection to instantiate the implementation class of the Indicator. Finally, it stores the Indicator in a ConcurrentHashMap, where the key is asych1 and asych2, and the value is an instance of the implementation class.

[0046] Each time a data stream is received, the corresponding keyword is determined, and an instance corresponding to the keyword is found through keyword matching. The instance defines the attribute data contained in this type of data stream. The attribute data contained in the current data stream is compared with the attribute data defined in the instance to determine whether the current data stream is a data stream with missing data. If the data stream is a data stream with missing data, it is the target data stream for which an asynchronous query operation needs to be performed. For example, the current data stream should contain ID and name. If the current data stream only contains ID and does not contain name, then the data stream is determined to be the target data stream for which an asynchronous query operation needs to be performed.

[0047] As mentioned above, the configurability of the Indicator avoids modifications to the original code. It is implemented based on Flink's native AsyncDataStream. It integrates seamlessly with the Flink framework, and its interface-based abstraction of asynchronous queries effectively improves development efficiency.

[0048] Furthermore, each type of data stream corresponds one-to-one with a keyword; the multiple attribute data contained in each data stream are determined, along with links to at least one external storage medium where each attribute data resides; a configuration file is generated based on the keyword, the multiple attribute data, and the links to at least one external storage medium; the configuration file is instantiated using Java reflection to generate instances corresponding to the data streams; multiple instances corresponding to various data stream types are obtained and saved to a preset hash map. The preset hash map is a ConcurrentHashMap. By defining the keyword corresponding to each data stream and configuring the instance corresponding to the data stream, each instance of each data stream defines the attribute data that should be included in the data stream and the links to the external storage medium where each attribute data resides, thus defining how to query data from the database.

[0049] In one optional embodiment, when the data stream is a target data stream, an asynchronous query operation is performed based on the target data stream, including: determining a preset keyword contained in the data stream, and finding an instance corresponding to the preset keyword, wherein the preset keyword is set at a preset character position in the data stream; determining the missing attribute data corresponding to the data stream based on multiple attribute data contained in the data stream as defined in the instance, wherein the missing attribute data is at least one missing attribute data in the data stream; finding a link to the external storage medium corresponding to the missing attribute data based on the instance; and retrieving the missing attribute data from the external storage medium based on the link to the external storage medium. For example, if the data stream should contain two attribute data, ID and name, and the currently missing attribute data is name, the external storage medium where the data corresponding to name is located can be determined through the corresponding instance. When performing an asynchronous query, the data corresponding to name is retrieved from the external storage medium corresponding to the name data.

[0050] In one optional embodiment, determining whether the asynchronous query operation was successfully executed, and if successful, sending the target data stream to the downstream of the preset data processing engine, includes: if missing attribute data is successfully obtained from the external storage medium, controlling the missing attribute data to be written into the target data stream, and writing a preset tag into the target data stream; sending the target data stream containing the missing attribute data and the preset tag to the downstream of the preset data processing engine. After the asynchronous query operation is successful, the found name data is written into the data stream to enhance the current data stream into a complete data stream, and then the complete data stream is sent to the downstream of the engine.

[0051] In the above process, after writing the found name into the data stream, the label corresponding to the Indicator is written into the message to indicate that the message has asynchronously retrieved the relevant data, and finally the message is sent downstream.

[0052] In one optional embodiment, in the event of execution failure, controlling the target data stream to write to the replay sub-engine to perform a data stream replay operation includes: controlling the target data stream to write to the replay window, determining the cumulative number of data streams written to the replay window, and if the cumulative number is greater than a preset number, controlling the target data stream to write to the replay sub-engine to perform a data stream replay operation; or, determining the replay time interval corresponding to the replay window, and after the time interval has elapsed, controlling the target data stream to write to the replay sub-engine to perform a data stream replay operation.

[0053] As described above, this application defines two triggering conditions for replaying data streams corresponding to the replay window. One is that the cumulative number of data streams in the replay window reaches a preset number, and the other is that the window calculation is triggered when the time interval of the maximum window calculation is exceeded. The window calculation sends the messages in the window to the FlinkKafakaProducer operator. This operator writes the messages that need to be replayed into the replay message engine topic. In the replay topic, the data stream enters the asynchronous data retrieval logic.

[0054] It should be noted that this application also provides an alternative solution for the data stream replay operation, which is to write the message to be replayed into RocketMQ and use RocketMQ's delayed sending function to achieve the effect of delaying and waiting for external data. However, this method requires the introduction of RocketMQ, which increases the fixed cost of the server and the operation and maintenance cost compared with the data replay function.

[0055] In an optional embodiment, in the event of execution failure, after controlling the target data stream to write to the replay sub-engine to perform the data stream replay operation, the method includes: after determining that the target data stream has failed to perform the data stream replay operation, writing the target data stream to the replay sub-engine again; determining the cumulative number of times the target data stream has been written to the replay sub-engine, and if the cumulative number is greater than a preset number, sending the target data stream to the downstream of a preset data processing engine.

[0056] As described above, by defining the total number of times the data stream can be replayed, the upper limit of the data stream replay is limited. After exceeding the upper limit, the data stream is considered unable to successfully execute the asynchronous query operation, and the data stream can be allowed to proceed. Through the above replay operation, the success rate of asynchronous queries of the data stream is maximized.

[0057] Figure 2 This is a schematic diagram of a data stream processing method provided in an embodiment of this application, such as... Figure 2As shown, in the asynchronous query phase, an asynchronous query is performed from the DB database for the current data stream. After the query, the success of the asynchronous query is checked. If successful, subsequent processing is performed on the data stream; if it fails, the data stream replay process is executed. Figure 2 The data stream processing method shown uses a replay function to mitigate asynchronous query failures, provides unified handling for abnormal situations, and ensures that the saved information is completely stored in the log.

[0058] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0059] This application also provides a data stream processing apparatus. It should be noted that the data stream processing apparatus of this application can be used to execute the data stream processing method provided in this application. The following describes the data stream processing apparatus provided in this application.

[0060] Figure 3 This is a schematic diagram of a data stream processing apparatus according to an embodiment of this application. Figure 3 As shown, the device includes: a first determining unit 301, used to determine whether a data stream is a target data stream when a data stream is received upstream of a preset data processing engine, wherein the target data stream is a data stream for which an asynchronous query operation is to be performed; a first executing unit 302, used to execute an asynchronous query operation based on the target data stream when the data stream is the target data stream; a first judging unit 303, used to judge whether the asynchronous query operation is successfully executed, and if the execution is successful, to send the target data stream downstream of the preset data processing engine; and a first controlling unit 304, used to control the target data stream to be written into a replay sub-engine to perform a data stream replay operation if the execution fails, wherein the data stream replay operation is an operation to re-control the data stream to perform the asynchronous query operation.

[0061] In an optional embodiment, the first determining unit 301 includes: a first determining subunit, configured to determine a preset keyword included in the data stream, wherein the preset keyword is set at a preset character position in the data stream; a first searching subunit, configured to search for an instance corresponding to the preset keyword based on the preset keyword, and obtain multiple attribute data contained in the data stream defined in the instance; a parsing subunit, configured to parse the current attribute data contained in the data stream, compare the current attribute data with multiple attribute data, and determine whether some attribute data is missing in the data stream; a second determining subunit, configured to determine the data stream as a target data stream if the data stream is missing some attribute data; and a third determining subunit, configured to determine the data stream as a non-target data stream if the current attribute data of the data stream is consistent with multiple attribute data.

[0062] In one optional embodiment, the lookup subunit includes: a first determining module, configured to acquire multiple data streams and determine multiple keywords corresponding to the multiple data streams, wherein each type of data stream corresponds one-to-one with a keyword; a second determining module, configured to determine multiple attribute data contained in each data stream, and a link to at least one external storage medium where the multiple attribute data is located; a first generating module, configured to generate a configuration file based on the keywords, multiple attribute data, and the link to at least one external storage medium; a second generating module, configured to instantiate the configuration file using JAVA reflection to generate instances corresponding to the data streams; and a saving module, configured to obtain multiple instances corresponding to the multiple types of data streams and save the multiple instances to a preset hash map.

[0063] In an optional embodiment, the first execution unit 302302 includes: a fourth determining subunit, configured to determine a preset keyword contained in the data stream, and search for an instance corresponding to the preset keyword based on the preset keyword, wherein the preset keyword is set at a preset character position in the data stream; a fifth determining subunit, configured to determine missing attribute data corresponding to the data stream based on multiple attribute data contained in the data stream as defined in the instance, wherein the missing attribute data is at least one missing attribute data in the data stream; a second searching subunit, configured to search for a link to an external storage medium corresponding to the missing attribute data based on the instance; and an obtaining subunit, configured to obtain the missing attribute data from the external storage medium based on the link to the external storage medium.

[0064] In an optional embodiment, the first determination unit 303 includes: a first control subunit, configured to control the missing attribute data to be written into a target data stream and to write a preset tag into the target data stream when the missing attribute data is successfully obtained from an external storage medium; and a sending subunit, configured to send the target data stream containing the missing attribute data and the preset tag to the downstream of a preset data processing engine.

[0065] In one optional embodiment, the first control unit 304 includes: a second control subunit, configured to control the target data stream to be written to the playback window, determine the cumulative number of data streams written to the playback window, and, if the cumulative number is greater than a preset number, control the target data stream to be written to the playback sub-engine to perform a data stream playback operation; or, a sixth determining subunit, configured to determine the playback time interval corresponding to the playback window, and, after the time interval has elapsed, control the target data stream to be written to the playback sub-engine to perform a data stream playback operation.

[0066] In one optional embodiment, the device includes: a writing unit, configured to, in the event of execution failure, control the target data stream to be written to the replay sub-engine after performing the data stream replay operation, and after determining that the target data stream replay operation has failed, write the target data stream to the replay sub-engine again; and a sending unit, configured to determine the cumulative number of times the target data stream has been written to the replay sub-engine, and if the cumulative number of times is greater than a preset number, send the target data stream to the downstream of a preset data processing engine.

[0067] A data stream processing device includes a processor and a memory. The first determining unit 301 and the like are all stored in the memory as program units, and the processor executes the program units stored in the memory to realize the corresponding functions.

[0068] The processor contains a kernel, which retrieves the corresponding program unit from memory. One or more kernels can be configured. By adjusting kernel parameters, the technical problem in existing Flink open-source stream processing frameworks—the lack of specific execution logic for asynchronous queries and the absence of exception handling after a failed asynchronous query—can be addressed, resulting in low execution efficiency.

[0069] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0070] This invention provides a computer-readable storage medium storing a program that, when executed by a processor, implements a data stream processing method.

[0071] This invention provides a processor for running a program, wherein the program executes a data stream processing method during runtime.

[0072] This invention provides an apparatus including a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs at least the following steps: If a data stream is received upstream of a preset data processing engine, determine whether the data stream is a target data stream, wherein the target data stream is a data stream for which an asynchronous query operation is to be performed; if the data stream is the target data stream, perform an asynchronous query operation based on the target data stream; determine whether the asynchronous query operation was successfully executed, and if successful, send the target data stream downstream of the preset data processing engine; if the execution fails, control the target data stream to be written into a replay sub-engine to perform a data stream replay operation, wherein the data stream replay operation is an operation to re-control the data stream to perform the asynchronous query operation.

[0073] Optionally, when a data stream is received upstream of the preset data processing engine, determining whether the data stream is the target data stream includes: determining preset keywords included in the data stream, wherein the preset keywords are set at preset character positions in the data stream; based on the preset keywords, finding instances corresponding to the preset keywords, and obtaining multiple attribute data contained in the data stream defined in the instances; parsing the current attribute data contained in the data stream, comparing the current attribute data with multiple attribute data, and determining whether some attribute data is missing in the data stream; if the data stream is missing some attribute data, determining that the data stream is the target data stream; if the current attribute data of the data stream is consistent with multiple attribute data, determining that the data stream is a non-target data stream.

[0074] Optionally, based on preset keywords, the process involves finding instances corresponding to those preset keywords, including: acquiring multiple data streams and determining multiple keywords corresponding to the data streams, wherein each type of data stream corresponds one-to-one with a keyword; determining multiple attribute data contained in each data stream, and links to at least one external storage medium containing the attribute data; generating a configuration file based on the keywords, the multiple attribute data, and the links to at least one external storage medium; instantiating the configuration file using JAVA reflection to generate instances corresponding to the data streams; obtaining multiple instances corresponding to the multiple types of data streams, and saving the multiple instances to a preset hash map.

[0075] Optionally, when the data stream is the target data stream, an asynchronous query operation is performed based on the target data stream, including: determining a preset keyword contained in the data stream, and finding an instance corresponding to the preset keyword based on the preset keyword, wherein the preset keyword is set at a preset character position in the data stream; determining the missing attribute data corresponding to the data stream based on multiple attribute data contained in the data stream as defined in the instance, wherein the missing attribute data is at least one missing attribute data in the data stream; finding a link to the external storage medium corresponding to the missing attribute data based on the instance; and retrieving the missing attribute data from the external storage medium based on the link to the external storage medium.

[0076] Optionally, determine whether the asynchronous query operation was executed successfully, and if it was executed successfully, send the target data stream to the downstream of the preset data processing engine, including: if the missing attribute data is successfully obtained from the external storage medium, control the missing attribute data to be written into the target data stream and write the preset label into the target data stream; send the target data stream containing the missing attribute data and the preset label to the downstream of the preset data processing engine.

[0077] Optionally, in the event of execution failure, after controlling the target data stream to write to the replay sub-engine to perform the data stream replay operation, the method includes: after determining that the target data stream replay operation has failed, writing the target data stream to the replay sub-engine again; determining the cumulative number of times the target data stream has been written to the replay sub-engine, and if the cumulative number is greater than a preset number, sending the target data stream downstream to a preset data processing engine. The device in this document can be a server, PC, PAD, mobile phone, etc.

[0078] This application also provides a computer program product, which, when executed on a data stream processing device, is suitable for executing an initialization program having at least the following method steps: upon receiving a data stream upstream of a preset data processing engine, determining whether the data stream is a target data stream, wherein the target data stream is a data stream for which an asynchronous query operation is to be performed; if the data stream is the target data stream, performing an asynchronous query operation based on the target data stream; determining whether the asynchronous query operation was successfully executed, and if successful, sending the target data stream downstream of the preset data processing engine; if the execution fails, controlling the target data stream to write to a replay sub-engine to perform a data stream replay operation, wherein the data stream replay operation is an operation to re-control the data stream to perform the asynchronous query operation.

[0079] Optionally, when a data stream is received upstream of the preset data processing engine, determining whether the data stream is the target data stream includes: determining preset keywords included in the data stream, wherein the preset keywords are set at preset character positions in the data stream; based on the preset keywords, finding instances corresponding to the preset keywords, and obtaining multiple attribute data contained in the data stream defined in the instances; parsing the current attribute data contained in the data stream, comparing the current attribute data with multiple attribute data, and determining whether some attribute data is missing in the data stream; if the data stream is missing some attribute data, determining that the data stream is the target data stream; if the current attribute data of the data stream is consistent with multiple attribute data, determining that the data stream is a non-target data stream.

[0080] Optionally, based on preset keywords, the process involves finding instances corresponding to those preset keywords, including: acquiring multiple data streams and determining multiple keywords corresponding to the data streams, wherein each type of data stream corresponds one-to-one with a keyword; determining multiple attribute data contained in each data stream, and links to at least one external storage medium containing the attribute data; generating a configuration file based on the keywords, the multiple attribute data, and the links to at least one external storage medium; instantiating the configuration file using JAVA reflection to generate instances corresponding to the data streams; obtaining multiple instances corresponding to the multiple types of data streams, and saving the multiple instances to a preset hash map.

[0081] Optionally, when the data stream is the target data stream, an asynchronous query operation is performed based on the target data stream, including: determining a preset keyword contained in the data stream, and finding an instance corresponding to the preset keyword based on the preset keyword, wherein the preset keyword is set at a preset character position in the data stream; determining the missing attribute data corresponding to the data stream based on multiple attribute data contained in the data stream as defined in the instance, wherein the missing attribute data is at least one missing attribute data in the data stream; finding a link to the external storage medium corresponding to the missing attribute data based on the instance; and retrieving the missing attribute data from the external storage medium based on the link to the external storage medium.

[0082] Optionally, determine whether the asynchronous query operation was executed successfully, and if it was executed successfully, send the target data stream to the downstream of the preset data processing engine, including: if the missing attribute data is successfully obtained from the external storage medium, control the missing attribute data to be written into the target data stream and write the preset label into the target data stream; send the target data stream containing the missing attribute data and the preset label to the downstream of the preset data processing engine.

[0083] Optionally, in the event of execution failure, after controlling the target data stream to write to the replay sub-engine to perform the data stream replay operation, the method includes: after determining that the target data stream has failed to perform the data stream replay operation, writing the target data stream to the replay sub-engine again; determining the cumulative number of times the target data stream has been written to the replay sub-engine, and if the cumulative number is greater than a preset number, sending the target data stream to the downstream of the preset data processing engine.

[0084] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0085] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some interfaces; indirect couplings or communication connections between units or modules may be electrical or other forms.

[0086] The units described as separate components may or may not be physically separate. Similarly, the components shown as units may or may not be physical units; they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0087] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0088] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0089] As can be seen from the above description, the embodiments of this application achieve the following technical effects:

[0090] 1) Based on Flink, a configurable indicator is defined that can be seamlessly integrated with the Flink framework. At the same time, through the interface abstraction of asynchronous queries, development efficiency can be effectively improved and modifications to the original code can be avoided.

[0091] 2) The replay function mitigates asynchronous query failures. A unified approach to handling exceptions ensures that saved information is completely stored in the logs.

[0092] The above are merely preferred embodiments of this application and are not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for processing data streams, characterized in that, include: When a data stream is received upstream of the preset data processing engine, it is determined whether the data stream is the target data stream, wherein the target data stream is the data stream for which an asynchronous query operation is to be performed; If the data stream is the target data stream, the asynchronous query operation is performed based on the target data stream; Determine whether the asynchronous query operation was executed successfully, and if it was executed successfully, send the target data stream to the downstream of the preset data processing engine; In the event of execution failure, the target data stream is controlled to write to the replay sub-engine to perform a data stream replay operation, wherein the data stream replay operation is an operation to re-control the data stream to execute the asynchronous query operation. When a data stream is received upstream of the preset data processing engine, determining whether the data stream is the target data stream includes: Determine the preset keywords included in the data stream, wherein the preset keywords are set at preset character positions in the data stream; Based on the preset keyword, find the instance corresponding to the preset keyword, and obtain multiple attribute data contained in the data stream defined in the instance; The current attribute data contained in the data stream is parsed, and the current attribute data is compared with multiple attribute data to determine whether some attribute data is missing in the data stream. In the case where the attribute data is missing from the data stream, the data stream is determined to be the target data stream; If the current attribute data of the data stream is consistent with multiple attribute data, the data stream is determined to be a non-target data stream.

2. The method according to claim 1, characterized in that, Based on the preset keyword, find instances corresponding to the preset keyword, including: Acquire multiple data streams and determine multiple keywords corresponding to the multiple data streams, wherein each type of data stream corresponds one-to-one with the keyword; Identify the plurality of attribute data contained in each of the data streams, and the link to at least one external storage medium where the plurality of attribute data resides; A configuration file is generated based on the keywords, multiple attribute data, and a link to at least one external storage medium; The configuration file is instantiated using Java reflection to generate an instance corresponding to the data stream. Multiple instances corresponding to the various types of data streams are obtained, and the multiple instances are saved to a preset hash graph.

3. The method according to claim 1, characterized in that, When the data stream is the target data stream, the asynchronous query operation is performed based on the target data stream, including: Determine the preset keywords contained in the data stream, and find instances corresponding to the preset keywords based on the preset keywords, wherein the preset keywords are set at preset character positions in the data stream; Based on the multiple attribute data contained in the data stream as defined in the example, the missing attribute data corresponding to the data stream is determined, wherein the missing attribute data is at least one of the attribute data missing in the data stream; Based on the example described, locate the link to the external storage medium corresponding to the missing attribute data; Based on the link to the external storage medium, the missing attribute data is obtained from the external storage medium.

4. The method according to claim 3, characterized in that, Determining whether the asynchronous query operation was successfully executed, and if successful, sending the target data stream to the downstream of the preset data processing engine, includes: If the missing attribute data is successfully obtained from the external storage medium, the missing attribute data is written to the target data stream, and a preset tag is written to the target data stream. The target data stream, containing the missing attribute data and the preset label, is sent downstream of the preset data processing engine.

5. The method according to claim 1, characterized in that, In the event of execution failure, the target data stream is controlled to write to the replay sub-engine to perform a data stream replay operation, including: The target data stream is controlled to be written to the replay window. The cumulative number of data streams written to the replay window is determined. If the cumulative number is greater than a preset number, the target data stream is controlled to be written to the replay sub-engine to perform the data stream replay operation; or, The replay time interval corresponding to the replay window is determined. After the time interval has elapsed, the target data stream is controlled to be written to the replay sub-engine to perform the data stream replay operation.

6. The method according to claim 1, characterized in that, In the event of execution failure, after controlling the target data stream to write to the replay sub-engine to perform a data stream replay operation, the method includes: After determining that the target data stream failed to perform the data stream replay operation, the target data stream is written to the replay sub-engine again; The cumulative number of times the target data stream is written into the replay sub-engine is determined. If the cumulative number is greater than a preset number, the target data stream is sent to the downstream of the preset data processing engine.

7. A data stream processing apparatus, characterized in that, include: The first determining unit is used to determine whether the data stream is a target data stream when a data stream is received upstream of a preset data processing engine, wherein the target data stream is a data stream for which an asynchronous query operation is to be performed; The first execution unit is configured to execute the asynchronous query operation based on the target data stream when the data stream is the target data stream; The first judgment unit is used to determine whether the asynchronous query operation is executed successfully, and if it is executed successfully, to send the target data stream to the downstream of the preset data processing engine; The first control unit is configured to, in the event of execution failure, control the target data stream to be written to the replay sub-engine to perform a data stream replay operation, wherein the data stream replay operation is an operation to re-control the data stream to execute the asynchronous query operation. The first determining unit includes: a first determining subunit, used to determine a preset keyword included in the data stream, wherein the preset keyword is set at a preset character position in the data stream; a first searching subunit, used to search for an instance corresponding to the preset keyword based on the preset keyword, and obtain multiple attribute data contained in the data stream defined in the instance; a parsing subunit, used to parse the current attribute data contained in the data stream, compare the current attribute data with multiple attribute data, and determine whether some attribute data is missing in the data stream; a second determining subunit, used to determine that the data stream is a target data stream when some attribute data is missing in the data stream; and a third determining subunit, used to determine that the data stream is a non-target data stream when the current attribute data of the data stream is consistent with multiple attribute data.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein the program executes a data stream processing method according to any one of claims 1 to 6.

9. A processor, characterized in that, The processor is used to run a program, wherein the program executes a data stream processing method according to any one of claims 1 to 6.

10. An electronic device, characterized in that, include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including a method for processing a data stream as described in any one of claims 1 to 6.

Images