A data processing method, apparatus and system

Abstract

This invention discloses a data processing method, apparatus, and system. By receiving a data stream and determining its data attribute information and corresponding configuration information, the method queries a data cache based on the data attribute information and configuration information to determine the matching progress of the data stream. Data matching is then performed according to the matching progress, and upon successful matching, a message corresponding to the data stream is sent to a target message queue. Therefore, this embodiment can improve data processing efficiency while achieving real-time data processing by caching historical matching results of behavioral data and performing data matching processing based on the current matching progress.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and more specifically, to a data processing method, apparatus, and system. Background Technology

[0002] With the rapid development of the internet, various computer devices are indispensable in daily life and work. Therefore, responding to and judging users' real-time behavior is crucial. Among related technologies, Flink-Cep was used to implement a complex business logic based on watermarks. All events are sorted based on timestamps, and delayed events are not processed, resulting in low data processing efficiency. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a data processing method, apparatus and system to perform data matching processing based on the current matching progress by caching historical matching results of behavioral data, thereby improving data processing efficiency while achieving real-time data processing.

[0004] In a first aspect, embodiments of the present invention provide a data processing method, the method comprising:

[0005] Receive data stream;

[0006] Determine the data attribute information and corresponding configuration information of the data stream;

[0007] The matching progress corresponding to the data stream is determined by querying the data cache based on the data attribute information and the configuration information.

[0008] Data matching is performed according to the matching progress.

[0009] In response to a successful data match, the message corresponding to the data stream is sent to the target message queue.

[0010] In a second aspect, embodiments of the present invention provide a data processing system, the system comprising:

[0011] The database is configured to store data rules;

[0012] The data caching module is configured to cache the matching progress of the data stream;

[0013] The data processing module is configured to query the database and the data caching module based on the received data stream to determine the matching progress corresponding to the data behavior stream, perform data matching based on the matching progress, and send the message corresponding to the successfully matched data stream to the target message queue.

[0014] Thirdly, embodiments of the present invention provide a data processing apparatus, the apparatus comprising:

[0015] The receiving unit is configured to receive data streams;

[0016] The information determination unit is configured to determine the data attribute information and corresponding configuration information of the data stream;

[0017] The progress query unit is configured to query the data cache based on the data attribute information and the configuration information to determine the matching progress corresponding to the data stream.

[0018] A matching unit is configured to perform data matching according to the matching progress.

[0019] The sending unit is configured to send the message corresponding to the data stream to the target message queue in response to a successful data match.

[0020] Fourthly, embodiments of the present invention provide an electronic device, including a memory and a processor, wherein the memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method described above.

[0021] Fifthly, embodiments of the present invention provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described above.

[0022] Sixthly, embodiments of the present invention provide a computer program product that, when run on a computer, causes the computer to perform the method described above.

[0023] This embodiment of the invention receives a data stream, determines its data attribute information and corresponding configuration information, queries the data cache based on the data attribute information and configuration information, determines the matching progress of the data stream, performs data matching according to the matching progress, and sends the message corresponding to the data stream to the target message queue in response to successful data matching. Therefore, this embodiment can improve data processing efficiency while achieving real-time data processing by caching historical matching results of behavioral data and performing data matching processing based on the current matching progress. Attached Figure Description

[0024] The above and other objects, features and advantages of the present invention will become clearer from the following description of embodiments of the invention with reference to the accompanying drawings, in which:

[0025] Figure 1 This is a schematic diagram of the data processing system according to an embodiment of the present invention;

[0026] Figure 2This is a flowchart of a data processing method according to an embodiment of the present invention;

[0027] Figure 3 This is a flowchart of another data processing method according to an embodiment of the present invention;

[0028] Figure 4 This is a schematic diagram of a data processing device according to an embodiment of the present invention;

[0029] Figure 5 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0030] The present invention is described below based on embodiments, but the invention is not limited to these embodiments. In the detailed description of the invention below, certain specific details are described in detail. Those skilled in the art will fully understand the invention even without these details. To avoid obscuring the essence of the invention, well-known methods, processes, flows, elements, and circuits are not described in detail.

[0031] Furthermore, those skilled in the art should understand that the accompanying drawings provided herein are for illustrative purposes only and are not necessarily drawn to scale.

[0032] Unless the context explicitly requires it, words such as "including" or "contains" in the instruction manual should be interpreted as including rather than exclusive or exhaustive; that is, meaning "including but not limited to".

[0033] In the description of this invention, it should be understood that the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance. Furthermore, in the description of this invention, unless otherwise stated, "a plurality of" means two or more.

[0034] In related technologies, Flink-Cep, proposed by Flink, is commonly used. This involves matching one or more data streams composed of simple events according to certain rules to output complex events that satisfy those rules. However, this technology currently only supports the Java language package and cannot be widely applied to various languages. Furthermore, in terms of time sequence processing, Flink-Cep implements a complex business logic based on watermarks, sorting all events based on timestamps and neglecting to process delayed events. Therefore, existing Flink-Cep-based methods are logically complex and have low data processing efficiency. This embodiment proposes a data processing system based on an NFA (Non-Deterministic Finite State Machine) and utilizing data caching (e.g., Redis caching) to achieve real-time data processing. Specifically, this embodiment can determine the time window based on the data cache expiration time and handle delayed events by caching historical matching results. In other words, this embodiment can continue matching based on the historical matching progress of the data stream without re-matching the data stream, improving data processing efficiency and reducing the complexity of the system's algorithm logic.

[0035] Figure 1 This is a schematic diagram of a data processing system according to an embodiment of the present invention. Figure 1 As shown, the data processing system 1 in this embodiment includes a data processing module 11, a data caching module 12, and a database 13.

[0036] In this embodiment, database 13 is configured to store data rules. In this embodiment, data rules for each behavioral data stream are pre-set via a visual interface and stored in database 13. Optionally, database 13 can be a NoSQL database. It should be understood that this embodiment does not limit the database used.

[0037] In one optional implementation, the data rule includes a rule identifier (rule name or rule code identifier, etc.) and a rule triggering condition. Optionally, the stored field format is shown in Table (1). It should be understood that this embodiment does not limit this.

[0038] Table (1)

[0039] key val

[0040] Wherein, key is the rule name, and val is the rule configuration information, including rule triggering conditions, target message queue, etc. Data caching module 12 is configured to cache the matching progress of data streams. A data stream needs to satisfy at least one data rule before it can enter the target message queue for processing. In this embodiment, data caching module 12 caches the matching progress of data streams. For example, when data stream A is received, it is parsed to determine that it includes three data rules: a, b, and c. Data rule matching is performed on data stream A. If data rules a and b match successfully, but data rule c fails to match, the matching progress of data stream A (i.e., data stream A has matched data rule b) is cached in data caching module 12. Optionally, data caching module 12 can use Redis caching. It should be understood that this embodiment does not limit the data caching module used.

[0041] The data processing module 11 is configured to query the database 12 and the data caching module 13 based on the received data stream to determine the matching progress corresponding to the data behavior stream, perform data matching according to the corresponding matching progress, and send the message corresponding to the successfully matched data stream to the target message queue.

[0042] In one optional implementation, the data processing module 11 performs data cleaning and parsing on the received data stream. Optionally, for the received data stream, the data processing module 11 parses the data stream to determine its data attribute information. If the data attribute information of the data stream includes a user identifier and a rule identifier, then the data stream is determined to be a specified data stream. If the data stream is not a specified data stream, that is, its data attribute information does not include a user identifier or a rule identifier, then the data stream is deleted. After determining that the data stream is a specified data stream, the data processing module 11 queries the database 13 based on the rule identifier in the data attribute information of the data stream. If the rule configuration information corresponding to the rule identifier is not found in the database 13, then the data stream is deleted. If the rule configuration information corresponding to the rule identifier is found in the database 13, then the data caching module 12 is queried based on the user identifier and the rule identifier to query the matching progress corresponding to the data stream. If the data caching module 12 caches the corresponding matching progress, then data matching continues based on the matching node corresponding to the matching progress, and the matching progress is updated. After the matching is completed, the message corresponding to the data stream is sent to the target message queue. If the corresponding matching progress is not cached in the data caching module 12, that is, the data stream is a new data stream, then data matching is performed from the initial matching node, the matching progress is updated, and after the matching is completed, the message corresponding to the data stream is sent to the target message queue.

[0043] This invention stores data rules in a database, caches the matching progress of data streams using a data caching module, and parses and cleans the received data streams using a data processing module. It filters and selects data streams according to specified data attributes, queries corresponding data rules and matching progress based on the data attributes of the specified data streams, performs data matching, updates the matching progress in the data caching module, and sends the corresponding message to the target message queue after successful matching. Therefore, this embodiment can achieve real-time data processing based on a simple real-time data stream processing system, improving data processing efficiency. Furthermore, by caching the matching progress of a data stream using the data caching module, matching operations can continue based on the historical matching progress of the data stream, handling delayed events and further improving data processing efficiency.

[0044] Figure 2 This is a flowchart of a data processing method according to an embodiment of the present invention. Figure 2 As shown, the data processing method of this embodiment includes the following steps:

[0045] Step S110: Receive data stream.

[0046] Step S120: Determine the data attribute information and corresponding configuration information of the data stream. In an optional implementation, step S120 includes: parsing the data stream to determine its data attribute information, and determining whether the data stream is a specified data stream based on its data attribute information. Wherein, if the data attribute information of the data stream includes a user identifier and a rule identifier, then the data stream is a specified data stream. Optionally, the data attribute information also includes supplementary information for the data stream, and the field format of the specified data stream is shown in Table (2).

[0047] Table (2)

[0048] event_code uid extra_info

[0049] Here, `event_code` is the rule identifier, `uid` is the user identifier, and `extra_info` contains other supplementary information about the data stream. Optionally, the data attribute information may also include the target message queue.

[0050] In one optional implementation, the data processing method of this embodiment further includes:

[0051] The system receives data rules from each data stream and stores each data rule in its corresponding database. Each data rule includes a rule identifier and a rule triggering condition.

[0052] In an optional implementation, step S120 further includes: responding to the data attribute information of the data stream satisfying a predetermined condition, searching for the corresponding configuration information based on the data attribute information. Optionally, the predetermined condition is that the data attribute information includes a user identifier and a rule identifier. That is, responding to the received data stream being a specified data stream, the configuration information of the data stream is searched based on the data attribute information of the data stream. Optionally, responding to the data attribute information of the data stream including a user identifier and a rule identifier, the configuration information corresponding to each data rule in the data stream is searched in the corresponding database based on the rule identifier. The configuration information of the data rule includes a rule identifier, rule triggering conditions, and a target message queue, etc. Optionally, the data attribute information may also include a target message queue.

[0053] In this embodiment, if the data attribute information of the data stream does not include a user identifier or a rule identifier, or if the configuration information of the corresponding data rule cannot be found based on the rule identifier corresponding to the data stream, that is, the data stream is not the required data stream, then the data stream is deleted.

[0054] Step S130: Query the data cache based on the data attribute information and configuration information of the data stream to determine the matching progress corresponding to the data stream.

[0055] In one optional implementation, the data attribute information responding to the data stream includes user identifier and rule identifier, and the configuration information corresponding to the rule identifier of the data stream is queried. Then, the data cache is queried based on the data attribute information and configuration information of the data stream to determine the matching progress corresponding to the data stream.

[0056] In one optional implementation, step S130 includes: generating a target query identifier based on data attribute information and configuration information, and querying the data cache based on the target query identifier to determine the matching progress. Optionally, the target query identifier can be generated based on a user identifier and a rule identifier. For example, assuming the user identifier is D62086 and the rule identifier is Tc, the target query identifier could be D62086_Tc. It should be understood that in other optional implementations, the target query identifier can also be generated based on a combination of the user identifier, rule identifier, and other information. This embodiment does not limit the method of generating the target query identifier.

[0057] In one alternative implementation, the initial matching node is determined as the current matching node in response to the data cache not containing the matching progress corresponding to the target query identifier. In other words, if the currently received data stream is a new data stream and no rule matching has been performed previously, the initial matching node is determined as the current matching node.

[0058] For example, suppose the currently received data stream A includes data rules a, b, and c. The configuration information of data rules a, b, and c is retrieved from the database. A target query identifier x is generated based on the user identifier in data stream A and data rules a, b, and c. The data cache is then queried based on target query identifier x. If it is determined that the data cache does not store the matching progress of target query identifier x, data matching is performed based on the information of the data stream and the configuration information of data rules a, b, and c. If the matching fails during the matching process, the matching progress of data stream A is updated in the data cache. Therefore, this embodiment can process data in real time.

[0059] In one optional implementation, in response to the matching progress corresponding to the target query identifier contained in the data cache, the historical matching progress corresponding to the target query identifier is determined, and the current matching node is determined based on the historical matching progress. That is, if the currently received data stream is not a new data stream but has previously undergone data rule matching, then matching continues based on the previous historical matching progress and the data attribute information and rule configuration information of the currently received data stream, thereby improving data processing efficiency.

[0060] For example, suppose the currently received data stream A includes data rules a, b, and c. The database is queried to retrieve the configuration information of data rules a, b, and c. A target query identifier x is generated based on the user identifier in data stream A and data rules a, b, and c. The data cache is then queried based on target query identifier x. If the historical matching progress corresponding to target query identifier x is determined to be that data rules a and b have been matched, then the current matching node is determined to be the matching of data rule c. Data matching is performed based on the information of the data stream and the configuration information of data rule c, eliminating the need to match data rules a and b again. This reduces the amount of data computation and improves data processing efficiency.

[0061] Step S140: Perform data matching according to the matching progress.

[0062] In one optional implementation, step S140 includes: performing data matching based on the current matching node, the data attribute information of the data stream, and the configuration information of the corresponding data rules, and updating the data matching progress. As described above, the current matching node is determined based on the matching progress cached in the data cache, and data matching is performed based on the data attribute information of the data stream and the configuration information of the corresponding data rules. If data matching fails, the matching progress of the data stream in the data cache is updated; if data matching succeeds, the matching progress of the data stream in the data cache is deleted.

[0063] In step S150, in response to successful data matching, the message corresponding to the data stream is sent to the target message queue. That is, in response to all data rules in the received data stream being successfully matched, the message corresponding to that data stream is sent to the target message queue for queuing and processing in order to respond to that message.

[0064] This invention, through storing data rules and the matching progress of data streams, parses and cleans received data streams, filters specified data streams, queries corresponding data rules and matching progress based on the data attribute information of the specified data streams for data matching, updates the matching progress in the data caching module, and sends the message corresponding to the data stream to the target message queue after successful data matching. Therefore, this embodiment can achieve real-time data processing based on a simple real-time data stream processing system, improving data processing message efficiency. Furthermore, by caching the matching progress of a data stream through the data caching module, matching operations can continue based on the historical matching progress of the data stream, enabling the handling of delayed events and further improving data processing efficiency.

[0065] Figure 3 This is a flowchart of another data processing method according to an embodiment of the present invention. Figure 3 As shown, the data processing method in this embodiment includes the following steps:

[0066] Step S210: Receive data stream.

[0067] Step S220: Parse the data stream to determine its data attribute information, and determine whether the data stream is a specified data stream based on the data attribute information. If the data attribute information of the data stream includes a user identifier and a rule identifier, then the data stream is a specified data stream. Optionally, the data attribute information may also include supplementary information about the data stream.

[0068] Step S230: Determine whether the data attribute information meets predetermined conditions. In this embodiment, if the data attribute information meets the predetermined conditions, step S240 is executed; if the data attribute information does not meet the predetermined conditions, step S260 is executed, i.e., the data stream is deleted. Optionally, the predetermined conditions are that the data attribute information includes a user identifier and a rule identifier.

[0069] Step S240: In response to the data attribute information of the data stream meeting predetermined conditions, the corresponding database is queried based on the data attribute information. The database stores configuration information for each data rule, including the rule identifier and triggering conditions of the data rule. Optionally, the corresponding database is queried based on the rule identifier in the data attribute information.

[0070] Step S250: Determine whether the configuration information of each data rule in the data stream has been obtained from the database. In this embodiment, in response to the query of the database and the configuration information corresponding to each rule identifier in the data attribute information, step S270 is executed; in response to the query of the database and the configuration information corresponding to each rule identifier in the data attribute information not being obtained, step S260 is executed, that is, the data stream is deleted.

[0071] Step S260: Delete the data stream. In this embodiment, if the data attribute information of the data stream does not include a user identifier or a rule identifier, or if the configuration information of the corresponding data rule cannot be found in the database based on the rule identifier corresponding to the data stream, that is, if the data stream is not the required data stream, then the data stream is deleted.

[0072] Step S270: Generate a target query identifier based on the data attribute information and configuration information of the data stream. Optionally, the target query identifier can be generated based on the user identifier and rule identifier. For example, assuming the user identifier is D62086 and the rule identifier is Tc, the target query identifier could be D62086_Tc. It should be understood that in other optional implementations, the target query identifier can also be generated based on a combination of the user identifier, rule identifier, and other information. This embodiment does not limit the method of generating the target query identifier.

[0073] Step S280: Query the data cache based on the target query identifier. The data cache stores the matching progress of each non-new data stream.

[0074] Step S290: Determine whether the matching progress corresponding to the target query identifier is stored in the data cache. In this embodiment, if the matching progress corresponding to the target query identifier is stored in the data cache, step S2B0 is executed; if the matching progress corresponding to the target query identifier is not stored in the data cache, step S2A0 is executed.

[0075] In step S2A0, in response to the fact that the matching progress corresponding to the target query identifier is not stored in the data cache, that is, the data stream is a new data stream, the initial matching node is determined as the current matching node.

[0076] Step S2B0: In response to the fact that the data cache stores the matching progress corresponding to the target query identifier, that is, the data stream is not a new data stream, the current matching node is determined according to the historical matching progress corresponding to the target query identifier.

[0077] Step S2C0 involves performing data matching based on the current matching node, the data attribute information of the data stream, and the configuration information of the corresponding data rules, and updating the data matching progress. As described above, the current matching node is determined based on the matching progress cached in the data cache, and data matching is performed based on the data attribute information of the data stream and the configuration information of the corresponding data rules. If data matching fails, the matching progress of the data stream in the data cache is updated; if data matching succeeds, the matching progress of the data stream in the data cache is deleted.

[0078] In step S2D0, in response to successful data matching, the message corresponding to the data stream is sent to the target message queue. That is, in response to all data rules in the received data stream being successfully matched, the message corresponding to that data stream is sent to the target message queue for queuing and processing in order to respond to that message.

[0079] This invention, through storing data rules and the matching progress of data streams, parses and cleans received data streams, filters specified data streams, queries corresponding data rules and matching progress based on the data attribute information of the specified data streams for data matching, updates the matching progress in the data caching module, and sends the message corresponding to the data stream to the target message queue after successful data matching. Therefore, this embodiment can achieve real-time data processing based on a simple real-time data stream processing system, improving data processing message efficiency. Furthermore, by caching the matching progress of a data stream through the data caching module, matching operations can continue based on the historical matching progress of the data stream, enabling the handling of delayed events and further improving data processing efficiency.

[0080] The following example illustrates the rule of "sending behavior data to the t_open message queue when a user in region D performs a QR code scanning behavior".

[0081] First, the configuration information of the rules for scanning behavior entered through the visual interface is received and stored in the database. The rule name is t_qrcose, and the rule trigger condition is that when a user in region D scans a code, the current behavior data is sent to the message queue of t_open. The rules are stored in the database as shown in Table (3).

[0082] Table (3)

[0083] key t_qrcose val Pattern.begin("s1").where("channel==1130015&&city==2").sinkTopic("t_open")

[0084] Table (3) shows the rule configuration information for the data rule t_qrcose. The rule is that the channel is 1130015 and the city is 2. After these two conditions are met, the data should be sent to the message queue of t_open. The channel represents the scanning channel, and city 2 is region D. That is, users who have performed scanning behavior in region D can send messages to the message queue of t_open.

[0085] In this embodiment, after receiving the data stream, the data stream is parsed, cleaned, and matched according to rules. The message corresponding to the successfully matched data stream is sent to the message queue of t_open.

[0086] The regulations stipulate that the data flow should include user identifier, rule identifier, and other supplementary information of the data flow. The definition of the data flow corresponding to the data rule t_qrcose is shown in Table (4).

[0087] Table (4)

[0088] field name Explanation Example event_code The name corresponding to each rule flow t_qrcose uid User unique identifier D62086 extra_info Additional Information {"channel":1130015,"city":2,"price":100}

[0089] This data stream represents user ID D62086, the behavior rule name is t_qrcose, and the behavior data translates to: In region D, a QR code scanning behavior occurred, with a price of 100 yuan.

[0090] In this embodiment, a query is performed in the database based on the rule name t_qrcose, and the configuration information corresponding to the rule name t_qrcose is found in the database, as shown in Table (3). Then, a target query identifier D62086_t_qrcose is generated based on the user identifier and the rule identifier. The matching progress corresponding to the target query identifier D62086_t_qrcose is not found in the Redis cache, that is, the data stream is a new data stream. Then, the information of the data stream is matched with the configuration information corresponding to the rule name t_qrcose. It is found that the requirements of channel is 1130015 and city is 2 are met. Therefore, it can be determined that the data stream is the data stream generated by the user scanning code in area D, that is, it meets the corresponding rule. The message corresponding to the data stream is sent to the target message queue t_open. The data processing process of other embodiments is similar to this, and will not be described again here.

[0091] This invention, through storing data rules and the matching progress of data streams, parses and cleans received data streams, filters specified data streams, queries corresponding data rules and matching progress based on the data attribute information of the specified data streams for data matching, updates the matching progress in the data caching module, and sends the message corresponding to the data stream to the target message queue after successful data matching. Therefore, this embodiment can achieve real-time data processing based on a simple real-time data stream processing system, improving data processing message efficiency. Furthermore, by caching the matching progress of a data stream through the data caching module, matching operations can continue based on the historical matching progress of the data stream, enabling the handling of delayed events and further improving data processing efficiency.

[0092] Figure 4 This is a schematic diagram of a data processing apparatus according to an embodiment of the present invention. Figure 4 As shown, the data processing device 4 in this embodiment includes a receiving unit 41, an information determining unit 42, a progress query unit 43, a matching unit 44, and a sending unit 45.

[0093] In this embodiment, the receiving unit 41 is configured to receive a data stream. The information determining unit 42 is configured to determine the data attribute information and corresponding configuration information of the data stream. The progress query unit 43 is configured to query the data cache based on the data attribute information and the configuration information to determine the matching progress corresponding to the data stream. The matching unit 44 is configured to perform data matching based on the matching progress. The sending unit 45 is configured to send the message corresponding to the data stream to the target message queue in response to successful data matching.

[0094] In one optional implementation, the information determination unit 42 includes a first information acquisition subunit. The first information acquisition subunit is configured to parse the data stream and acquire the data attribute information.

[0095] In one optional implementation, the information determination unit 42 includes a second information acquisition subunit. The second information acquisition subunit is configured to look up the corresponding configuration information based on the data attribute information in response to the data attribute information satisfying a predetermined condition. Optionally, the predetermined condition is that the data attribute information includes a user identifier and a rule identifier.

[0096] In one alternative implementation, the second information acquisition subunit is further configured to search for configuration information corresponding to each data rule in the data stream in the corresponding database according to the rule identifier.

[0097] In one optional implementation, the progress query unit 43 includes an identifier generation subunit and a progress query subunit. The identifier generation subunit is configured to generate a target query identifier based on the data attribute information and the configuration information. The progress query subunit is configured to query the data cache based on the target query identifier to determine the matching progress.

[0098] In one optional implementation, the progress query subunit includes a historical progress determination module and a first matching node determination module. The historical progress determination module is configured to determine the historical matching progress in response to the data cache containing the matching progress corresponding to the target query identifier. The first matching node determination module is configured to determine the current matching node based on the historical matching progress.

[0099] In one optional implementation, the progress query subunit includes a second matching node determination module. The second matching node determination module is configured to determine the initial matching node as the current matching node in response to the data cache not containing the matching progress corresponding to the target query identifier.

[0100] In an alternative implementation, the matching unit 44 is further configured to perform data matching based on the current matching node, the data attribute information, and the configuration information, and update the data matching progress.

[0101] In one optional implementation, the data processing apparatus of this embodiment further includes a rule acquisition unit and a storage unit. The rule acquisition unit is configured to receive data rules from each data stream, wherein the data rules include rule identifiers and rule triggering conditions. The storage unit is configured to store each of the data rules in a corresponding database.

[0102] This invention, through storing data rules and the matching progress of data streams, parses and cleans received data streams, filters specified data streams, queries corresponding data rules and matching progress based on the data attribute information of the specified data streams for data matching, updates the matching progress in the data caching module, and sends the message corresponding to the data stream to the target message queue after successful data matching. Therefore, this embodiment can achieve real-time data processing based on a simple real-time data stream processing system, improving data processing message efficiency. Furthermore, by caching the matching progress of a data stream through the data caching module, matching operations can continue based on the historical matching progress of the data stream, enabling the handling of delayed events and further improving data processing efficiency.

[0103] Figure 5 This is a schematic diagram of an electronic device according to an embodiment of the present invention. (For example...) Figure 5As shown, electronic device 5 is a general-purpose data processing device, which includes a general-purpose computer hardware structure, including at least a processor 51 and a memory 52. ​​The processor 51 and memory 52 are connected via a bus 53. The memory 52 is adapted to store instructions or programs executable by the processor 51. The processor 51 can be a standalone microprocessor or a collection of one or more microprocessors. Thus, the processor 51 executes the instructions stored in the memory 52 to perform the method flow of the embodiments of the present invention as described above, thereby realizing data processing and control of other devices. The bus 53 connects the aforementioned components together, and also connects the aforementioned components to a display controller 54, a display device, and an input / output (I / O) device 55. The input / output (I / O) device 55 can be a mouse, keyboard, modem, network interface, touch input device, motion-sensing input device, printer, and other devices known in the art. Typically, the input / output device 55 is connected to the system via an input / output (I / O) controller 55.

[0104] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus (devices), or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0105] This application is described with reference to flowchart illustrations of methods, apparatus (devices), and computer program products according to embodiments of this application. It should be understood that each step in the flowchart can be implemented by computer program instructions.

[0106] These computer program instructions may be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including an instruction means, the implementation process of which is described in the instruction means. Figure 1 The function specified in one or more processes.

[0107] These computer program instructions may also be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing device, produce instructions for implementing processes. Figure 1 A device for a function specified in one or more processes.

[0108] Another embodiment of the present invention relates to a non-volatile storage medium for storing a computer-readable program for use by a computer to execute some or all of the above-described method embodiments.

[0109] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program specifying the relevant hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0110] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. For those skilled in the art, the present invention can be modified and varied in various ways. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of protection of the present invention.

Claims

1. A data processing method, characterized in that, The method includes: Receive data stream; Determine the data attribute information and corresponding configuration information of the data stream; The matching progress corresponding to the data stream is determined by querying the data cache based on the data attribute information and the configuration information. Data matching is performed according to the matching progress. In response to a successful data match, the message corresponding to the data stream is sent to the target message queue; Specifically, determining the matching progress of the data stream by querying the data cache based on the data attribute information and the configuration information includes: Generate a target query identifier based on the data attribute information and the configuration information; The data cache is queried based on the target query identifier to determine the matching progress; Specifically, querying the data cache based on the target query identifier to determine the matching progress includes: In response to the data cache containing the matching progress corresponding to the target query identifier, the historical matching progress is determined; The current matching node is determined based on the historical matching progress. The historical matching progress includes data rules that have already been matched.

2. The method according to claim 1, characterized in that, Determining the data attribute information of the data stream includes: The data stream is parsed to obtain the data attribute information.

3. The method according to claim 1 or 2, characterized in that, Determining the configuration information corresponding to the data stream includes: In response to the data attribute information meeting predetermined conditions, the corresponding configuration information is retrieved based on the data attribute information.

4. The method according to claim 3, characterized in that, The predetermined condition is that the data attribute information includes user identifier and rule identifier.

5. The method according to claim 4, characterized in that, Finding the corresponding configuration information based on the data attribute information includes: Based on the rule identifier, search the corresponding database for the configuration information of each data rule in the data stream.

6. The method according to claim 1, characterized in that, Querying the data cache based on the target query identifier to determine the matching progress includes: In response to the fact that the data cache does not contain the matching progress corresponding to the target query identifier, the initial matching node is determined as the current matching node.

7. The method according to claim 1 or 6, characterized in that, Data matching based on the matching progress includes: Data matching is performed based on the current matching node, the data attribute information, and the configuration information, and the data matching progress is updated.

8. The method according to claim 1, characterized in that, The method further includes: Receive data rules from each data stream, wherein the data rules include rule identifiers and rule triggering conditions; Each of the aforementioned data rules is stored in its corresponding database.

9. A data processing system, characterized in that, The system includes: The database is configured to store data rules; The data caching module is configured to cache the matching progress of the data stream; The data processing module is configured to query the database and the data caching module based on the received data stream to determine the matching progress corresponding to the data stream, perform data matching based on the matching progress, and send the message corresponding to the successfully matched data stream to the target message queue. The data processing module is used for: Generate a target query identifier based on data attribute information and configuration information; The data cache is queried based on the target query identifier to determine the matching progress; Specifically, querying the data cache based on the target query identifier to determine the matching progress includes: In response to the data cache containing the matching progress corresponding to the target query identifier, the historical matching progress is determined; The current matching node is determined based on the historical matching progress. The historical matching progress includes data rules that have already been matched.

10. A data processing apparatus, characterized in that, The device includes: The receiving unit is configured to receive data streams; The information determination unit is configured to determine the data attribute information and corresponding configuration information of the data stream; The progress query unit is configured to query the data cache based on the data attribute information and the configuration information to determine the matching progress corresponding to the data stream. A matching unit is configured to perform data matching according to the matching progress. The sending unit is configured to send the message corresponding to the data stream to the target message queue in response to a successful data match. The progress query unit is used for: Generate a target query identifier based on the data attribute information and the configuration information; The data cache is queried based on the target query identifier to determine the matching progress; Specifically, querying the data cache based on the target query identifier to determine the matching progress includes: In response to the data cache containing the matching progress corresponding to the target query identifier, the historical matching progress is determined; The current matching node is determined based on the historical matching progress. The historical matching progress includes data rules that have already been matched.

11. An electronic device comprising a memory and a processor, characterized in that, The memory is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method as described in any one of claims 1-8.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1-8.

13. A computer program product, characterized in that, When the computer program product is run on a computer, it causes the computer to perform the method as described in any one of claims 1-8.

Images