Data Processing Method, System, Computer Device, and Storage Medium

Abstract

This application relates to a data processing method, apparatus, computer device, and storage medium. The method includes: obtaining a data stream transmitted by a data generation end, and storing each event message data included in the data stream in a first message queue; classifying and performing business theme mapping on the event message data in the first message queue based on an event model to determine event message data belonging to the same business theme; storing the event message data belonging to the same business theme in a second message queue of the business theme, where the second message queue is used to respond to a data subscription request of the data consumption end for the data of the business theme of the second message queue. Using this method ensures data timeliness and improves data processing efficiency.

Description

Technical Field

[0001] The present application relates to the technical field of message data processing, and particularly to a data processing method, apparatus, computer device, and storage medium. Background Art

[0002] With the development of Internet technology, under the trend of explosive growth of data information, various business application systems (also known as data consumer terminals) are developed according to business requirements. Furthermore, business data is transmitted from the data generation end to each business application system for data application, and data needs to be processed during the data transmission process.

[0003] In traditional data processing methods, it is necessary to batch-divide the data stream at the data generation end according to a preset dimension. After the data transmission channel receives the data of the same batch, data processing operations such as business theme classification are performed on the data of the same batch, and then the data of the same business theme domain after processing is transmitted to the corresponding downstream business application system.

[0004] However, the method of processing data in batches requires waiting for the data of the same batch to be received before data processing can be performed. In the face of real-time data, the method of batch data processing affects the timeliness of data. Summary of the Invention

[0005] Based on this, it is necessary to provide a data processing method, apparatus, computer device, and storage medium for the above technical problems.

[0006] A data processing method, which is applied to a data processing system, and the method includes:

[0007] Obtain the data stream transmitted by the data generation end, and store each event message data included in the data stream in the first message queue;

[0008] Classify and perform business theme mapping on the event message data in the first message queue based on an event model to determine the event message data belonging to the same business theme;

[0009] Store the event message data belonging to the same business theme in the second message queue of the business theme, and the second message queue is used to respond to the data subscription request of the data consumer terminal for the data of the business theme of the second message queue.

[0010] In one embodiment, the obtaining the data stream transmitted by the data generation end and storing each event message data included in the data stream in the first message queue includes:

[0011] Receive the data stream of the data generation end, and send the data stream to the data processing system by the data generation end calling the event message management interface;

[0012] Read the message header of each piece of message data in the data stream, and filter out the message data whose message header contains the target field from the message data to obtain event message data;

[0013] Store each piece of the event message data in the first message queue.

[0014] In one embodiment, classifying and performing business theme mapping on the event message data in the first message queue based on an event model to determine the event message data belonging to the same business theme includes:

[0015] Obtain an event model, which is a data recognition model with business theme attributes obtained according to a preset modeling method;

[0016] According to each event model, identify and read event message data in the first message queue, map the event message to the business theme to which the event model belongs, and determine the event message data belonging to the same business theme.

[0017] In one embodiment, storing the event message data belonging to the same business theme in the second message queue of the business theme includes:

[0018] According to the target event rule, perform processing and integration on the event message data belonging to the same business theme to obtain event data;

[0019] Add an event coding identifier to the event data to obtain target event data, and store the target event data in the second message queue of the business theme to which it belongs.

[0020] In one embodiment, the second message queue is used to respond to a data subscription request of the data consumer for the business theme of the second message queue, including:

[0021] Call the event message management interface to listen for the data subscription request of the data consumer; the data subscription request carries a target business theme identifier;

[0022] When the data subscription request sent by the data consumer is monitored, according to the target business theme identifier, determine a target second message queue in each second message queue, read the event message data in the target second message queue, and send the event message data in the target second message queue to the data consumer.

[0023] In one embodiment, the method further includes:

[0024] Obtain the event message data in the first message queue and each of the second message queues according to a preset period;

[0025] Store the event message data in each of the message queues in a target partition of a distributed file database;

[0026] Check the encoding order of the event coding identifiers carried in the event message data in each of the target partitions, and determine whether the encoding order of the event coding identifiers meets a preset coding condition;

[0027] If the encoding order of the event coding identifiers does not meet the preset coding condition, output a data missing prompt message.

[0028] A data processing system, the data processing system includes: a data transmission channel and a data processing engine,

[0029] The data transmission channel is used to obtain the data stream transmitted by the data generation end, store each event message data included in the data stream in the first message queue, and implement the transmission of the event message data;

[0030] The data processing engine is used to classify and map the business theme of the event message data in the first message queue based on an event model, and determine the event message data belonging to the same business theme;

[0031] The data processing engine is further used to store the event message data belonging to the same business theme in the second message queue of the business theme to which it belongs, and the second message queue is used to respond to the data subscription request of the data consumer end for the business theme of the second message queue.

[0032] In one embodiment, the data processing engine is further used to obtain an event model, and the event model is a data recognition model with business theme attributes obtained according to a preset modeling method;

[0033] According to each of the event models, identify and read the event message data in the first message queue, map the event message to the business theme to which the event model belongs, and determine the event message data belonging to the same business theme.

[0034] A data processing device, the device includes:

[0035] An acquisition module, configured to acquire the data stream transmitted by the data generation end, and store each event message data included in the data stream in the first message queue;

[0036] A classification module, configured to classify the event message data in the first message queue and perform business theme mapping based on an event model, and determine the event message data belonging to the same business theme;

[0037] A storage module, configured to store the event message data belonging to the same business theme into a second message queue of the corresponding business theme, where the second message queue is used to respond to a data subscription request of a data consumer for the business theme of the second message queue.

[0038] A computer device includes a memory and a processor. The memory stores a computer program. When the processor executes the computer program, the following steps are implemented:

[0039] Obtain a data stream transmitted by a data generation end, and store each piece of event message data included in the data stream in a first message queue;

[0040] Classify the event message data in the first message queue and perform business theme mapping based on an event model, and determine the event message data belonging to the same business theme;

[0041] Store the event message data belonging to the same business theme into a second message queue of the corresponding business theme, where the second message queue is used to respond to a data subscription request of a data consumer for the business theme of the second message queue.

[0042] A computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the following steps are implemented:

[0043] Obtain a data stream transmitted by a data generation end, and store each piece of event message data included in the data stream in a first message queue;

[0044] Classify the event message data in the first message queue and perform business theme mapping based on an event model, and determine the event message data belonging to the same business theme;

[0045] Store the event message data belonging to the same business theme into a second message queue of the corresponding business theme, where the second message queue is used to respond to a data subscription request of a data consumer for the business theme of the second message queue.

[0046] The above data processing method, apparatus, computer device, and storage medium. The data processing system obtains a data stream transmitted by a data generation end, stores each event message data included in the data stream in a first message queue; classifies and maps business topics for the event message data in the first message queue based on an event model to determine the event message data belonging to the same business topic; stores the event message data belonging to the same business topic in a second message queue, and instructs a data consumption end to subscribe to data from a corresponding second consumption queue according to a target business topic. By adopting the above method, through the asynchronous transmission mode of the first message queue and the second message queue, the event message data is classified and processed in a timely manner during the asynchronous transmission process to supply the data consumption end application, ensuring data timeliness and improving data processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 It is a schematic flowchart of the data processing method in an embodiment;

[0048] Figure 2 It is a schematic flowchart of the steps of transmitting event message data through a data transmission channel in an embodiment;

[0049] Figure 3 It is a schematic flowchart of the steps of determining the business topic to which event message data belongs in an embodiment;

[0050] Figure 4 It is a schematic flowchart of the steps of classifying and storing event message data in an embodiment;

[0051] Figure 5 It is a schematic flowchart of the steps of listening for a data subscription request from a data consumption end in an embodiment;

[0052] Figure 6 It is a schematic flowchart of the steps of verifying data consistency of a data generation end in an embodiment;

[0053] Figure 7 It is an example flowchart of the data processing method in an embodiment;

[0054] Figure 8 It is a schematic functional structure diagram of a data processing system in an embodiment;

[0055] Figure 9 It is a structural block diagram of a data processing apparatus in an embodiment;

[0056] Figure 10 It is an internal structure diagram of a computer device in an embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0057] In order to make the objectives, technical solutions and advantages of this application clearer, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not used to limit this application.

[0058] In one embodiment, as Figure 1 shown, a data processing method is provided. This data processing method is applied to a data processing system. Among them, the data processing system can be deployed on a server or on a terminal device. This method can also be deployed in a system including a terminal device and a server and implemented through the interaction between the terminal device and the server. The embodiments of this application do not make limitations. In this embodiment, the method includes the following steps:

[0059] Step 101, obtain the data stream transmitted by the data generation end, and store each event message data included in the data stream in the first message queue.

[0060] In implementation, the data generation end accesses the data processing system according to a pre-developed access policy, and transmits the generated data stream to the data processing system by calling a unified API (Application Programming Interface, also known as an event message management interface). The data processing system obtains the data stream transmitted by the data generation end, and stores each event message data included in the data stream in the first message queue. Among them, the event message data is obtained by the data generation end performing format processing on the original data of the data generation end according to the data format specified in the interface rules of the API. The data processing system includes a data transmission channel and a data processing engine. Then, through the interface call rule, the data transmission channel in the data processing system transmits the data stream stored in the first message queue transmitted by the data generation end to the data processing engine.

[0061] Step 102, classify and map the business topics of the event message data in the first message queue based on the event model, and determine the event message data belonging to the same business topic.

[0062] In implementation, the data processing system pre-stores event models of multiple different business scenarios and different business topics (topics). The data processing engine in the data processing system performs real-time classification and business topic mapping on the event message data in the first message queue in the data processing system according to the business topic to which each event model belongs, and determines the event message data belonging to the same business topic among them.

[0063] Step 103, store the event message data belonging to the same business topic in the second message queue of the business topic to which it belongs.

[0064] Among them, the second message queue is used to respond to the data subscription request of the data consumer for the business topic of the second message queue

[0065] In implementation, the second message queue can also be called a topic message queue, that is, the second message queue has a business topic attribute. Furthermore, the data processing engine of the data processing system, based on the data pulling logic of the event model, pulls and stores the event message data belonging to the same business topic into the second message queue corresponding to the business topic to which it belongs. Thus, the data consumer can subscribe to data in the second message queue of the business topic to which it belongs according to the business topic

[0066] In the above data processing method, the data processing system obtains the data stream transmitted by the data generation end, and stores each event message data included in the data stream in the first message queue. And classify and map the business topics of the event message data in the first message queue based on the event model to determine the event message data belonging to the same business topic. Then, the data processing system stores the event message data belonging to the same business topic into the second message queue of the business topic to which it belongs. Among them, the second message queue is used to respond to the data subscription request of the data consumer for the business topic of the second message queue. By adopting this method, through the asynchronous transmission method of the first message queue and the second message queue, the event message data is classified and processed in a timely manner during the asynchronous transmission process to supply the data consumer application, ensuring data timeliness and improving data processing efficiency

[0067] In one embodiment, as Figure 2 shown, the data stream of the data generation end includes various types of message data. Therefore, before data processing, the data in the data stream of the data generation end is screened first. The specific processing process of step 101 includes

[0068] Step 201, receive the data stream of the data generation end, where the data stream is sent to the data processing system by the data generation end calling the event message management interface

[0069] In implementation, the data generation end can be a system developed based on the Java development language. It can call a standard jar package (that is, the standard jar package is an event message transmission component) and transmit the data stream to the data transmission channel of the data processing system by calling the event message management interface API (unified data transmission interface)

[0070] Optionally, the data generation end can also be a system in a non-Java development language. It only needs to adapt to the data processing system and automatically develop an event message sending component when the data generation end is connected to the data processing system. The embodiments of the present application do not make limitations

[0071] Step 202: Read the message headers of each piece of message data in the data stream, and filter out the message data whose message headers contain the target field from the message data to obtain event message data.

[0072] In implementation, in order to call a unified API for data transmission, the data generation end needs to perform pre-formatting processing on the data to be transmitted generated by itself to generate event message data. Specifically, the data generation end processes the data to be transmitted into a standard event message body, and the data format of the standard event message body is JSON (JavaScript Object Notation) data format, which includes a common information header and a service body. Among them, the common information header includes basic event information, and the service body contains specific service information of the data generation end system. After generating the event message data, the data processing system adds a target field to the message header of each event message data, and this target field is used to uniquely identify the event message data. Then the data processing system reads the message headers of each piece of message data in the data stream after receiving the data stream, filters out the message data whose message headers contain the target field from the message data to obtain event message data. Filter other message data in the data stream, for example, sop messages, soap messages, 8583 messages, etc.

[0073] Optionally, in addition to filtering the data stream transmitted by the data generation end, the data processing system can also perform data processing operations such as conversion, deduplication, and enrichment. The embodiments of the present application do not make limitations.

[0074] Step 203: Store each event message data in the first message queue.

[0075] In implementation, a first message queue is preset in the data transmission channel of the data processing system, and the first message queue receives and stores the data stream of the data generation end. Therefore, the data processing system stores each piece of event message data obtained by screening in the first message queue. Then, through the data transmission channel of the data processing system, the data is transmitted to the data processing engine in the form of the first message queue.

[0076] In this embodiment, based on the event message data in the standard format generated by the data generation end, unified interface access to the data processing system is realized, reducing the intrusion of multiple interface developments on the data processing system, reducing the development cycle. At the same time, through the data transmission channel in the data processing system, the data message data of the data generation end is filtered and screened and processed in real time, improving the timely transmission and timely processing of event message data.

[0077] In one embodiment, as Figure 3 shown, the specific processing process of step 102 includes:

[0078] Step 301, obtain an event model.

[0079] The event model is a data recognition model with business theme attributes obtained according to a preset modeling method.

[0080] In implementation, the data processing system pre - constructs and stores multiple event models. Specifically, the data processing system analyzes the initial data (or called sample data) sent by the data generation end according to modeling methodologies (such as BIAN banking architecture network methodology and DDD (domain drive design) methodology), determines the data ownership relationships such as data sources, logics, data transmission destinations, etc. required for business scenarios under a certain business theme, defines business events according to the data ownership relationships, and then constructs an event model that can be used for data identification and classification from the business events. Therefore, the event model has business theme attributes. Among them, the business events that can be defined in each business scenario under the financial business theme include: related persons, contracts, products and servers, marketing, finance, channels, geographical resources, collaborative interactions, etc. business events. Therefore, each event model generated by the corresponding business event belongs to the event model in the financial field. Based on this, the data processing system obtains the event models of each business theme stored in advance, so as to classify the business theme to which the data belongs according to the event model.

[0081] Step 302, according to each event model, identify and read event message data in the first message queue, map the event messages to the business theme to which the event model belongs, and determine the event message data belonging to the same business theme.

[0082] In implementation, the data processing system pulls data in the first message queue according to each obtained event model. Specifically, the data processing engine in the data processing system identifies and reads the event message data in the first message queue according to the data attribute information contained in each event model, determines the event message data belonging to the same business theme, and performs data aggregation processing on the event message data.

[0083] In this embodiment, by pre - constructing an event model, real - time classification processing of the event message data in the first message queue is realized, improving data processing timeliness and data processing messages.

[0084] In one embodiment, as Figure 4 shown, the specific processing process of step 103 includes:

[0085] Step 401, according to the target event rule, perform processing of processing and integration on the event message data belonging to the same business theme to obtain event data.

[0086] In implementation, each event message data is business event data under a certain business scenario of the business theme to which it belongs. The data processing engine in the data processing system processes and integrates the event message data belonging to the same business theme in the same second message queue according to the target event rules of the data consumer end known in advance, and obtains the integrated event data.

[0087] Step 402: Add an event coding identifier to the event data to obtain the target event data, and store the target event data in the second message queue of the business theme to which it belongs.

[0088] In implementation, the data processing engine in the data processing system adds an event coding identifier to the integrated event data to obtain the target event data, and stores the target event data in the second message queue of the business theme to which it belongs, so that the data consumer end can obtain the target event data for data consumption in this second message queue according to the application requirements of the target event data. Among them, the event coding identifier includes: the node number of the event creator, the event code, the global event tracking number, etc., which are not limited in the embodiments of the present application.

[0089] In this embodiment, after the first message queue receives the event message data from the data generation end and stores the classified event message data in the second message queue respectively, further processing and integration of the event message data are implemented in the second message queue, realizing the asynchronous transmission of event messages and the real-time processing and statistics of multi-dimensional index analysis, improving the timeliness of data processing and data processing efficiency.

[0090] In one embodiment, as Figure 5 shown, the second message queue is used to respond to the data subscription request of the data consumer end for the business theme of the second message queue. The specific processing steps for the data consumer end to subscribe to data include:

[0091] Step 501: Call the event message management interface to listen to the data subscription request of the data consumer end. Among them, the target business theme identifier is carried in the data subscription request.

[0092] In implementation, the downstream data consumer end will send a data subscription request to the data processing system according to the data application requirements. The data processing system calls the unified event message management interface (API) to listen to the data subscription request of the data consumer end in real time. Among them, the target business theme identifier is carried in the data subscription request of the data consumer end, and the business theme to which the data required by the downstream data consumer end belongs can be identified based on this target business theme identifier.

[0093] Step 502: When a data subscription request sent by the data consumer end is monitored, based on the target service topic identifier, determine the target second message queue in each second message queue, read the event message data in the target second message queue, and send the event message data in the target second message queue to the data consumer end.

[0094] In implementation, when a data subscription request sent by the data consumer end is monitored, the data processing engine in the data processing system locates and queries the second message queue of the target service topic in each second message queue according to the target service topic identifier carried in the data subscription request, and determines the target second message queue. Then, the data processing engine reads the event message data in the target second message queue and sends the read event message data to the data consumer end for data consumption by the data consumer end.

[0095] In this embodiment, the data processing system calls a unified event message management interface to monitor the data subscription request of the data consumer end, and manages asynchronous event-driven SLA (Service-Level Agreement) / QoS (Quality of Service) and event exception handling according to the data subscription request, so as to implement functions and processes such as collection and query of data exception error reporting.

[0096] In one embodiment, as Figure 6 shown, the method further includes:

[0097] Step 601: Obtain the event message data in the first message queue and each second message queue according to a preset period.

[0098] Step 602: Store the event message data in each message queue in the target partition of the distributed file database.

[0099] In implementation, the data processing system clears the data cache in the message queue according to a fixed period. Therefore, before clearing the cache, the data processing system needs to pre-obtain the event message data in the current first message queue and the second message queues belonging to each service topic. Then, store the event message data in each message queue in a distributed file system (HDFS, Hadoop Distributed System). The distributed file system is a distributed file database integrated on multiple server nodes. In this distributed file database, the database tables are partitioned according to the database's own algorithm rules to obtain each partition (also called a partition sub-table) in the database. Then, the data processing system stores the event message data in each message queue in the corresponding target partition for classified storage.

[0100] Step 603: Check the encoding order of the event coding identifiers carried in the event message data in each target partition, and determine whether the encoding order of the event coding identifiers meets the preset coding conditions.

[0101] In implementation, the data processing system reads the event message data in each target partition of the distributed file database, checks the encoding order of the event coding identifiers carried in the event message data in the same message queue stored in the target partition, and determines whether its encoding order meets the preset coding conditions. Among them, the coding conditions can be configured according to the transmission requirements of the event message, which are not limited in the embodiments of the present application. For example, the preset coding condition can be that there is no skip number in the encoding order, or the encoding order in the coding condition satisfies monotonicity, etc.

[0102] Step 604: If the encoding order of the event coding identifiers does not meet the preset coding conditions, output a data missing prompt message.

[0103] In implementation, if the encoding order of the event coding identifiers does not meet the preset coding conditions, it indicates that there is a lack of event message data included in the same event transmitted by the upstream data generation end. Then, the data processing system outputs a data missing prompt to the upstream data generation end, instructing the upstream data generation end to resend the missing data.

[0104] In this embodiment, by checking the encoding order of the event coding identifiers of the event message data, it is determined whether there is data missing, thereby implementing a data consistency (also known as event transaction consistency) guarantee mechanism (problem discovery, alarm prompt), and further implementing a mechanism for the ultimate consistency of data at the downstream data application end.

[0105] It should be understood that although Figures 1 to 6 the steps in the flowchart are shown in sequence according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise clearly stated in this article, the execution of these steps has no strict order limit, and these steps can be executed in other orders. Moreover, Figures 1 to 6 at least a part of the steps in

[0106] In one embodiment, as Figure 7 shown, an example of data processing is provided, and the specific data processing process is as follows:

[0107] There are two data transfer scenarios during data processing: Scenario 1, the transfer of simple event message data, and Scenario 2, the transfer of responsible event message data.

[0108] The specific steps in the process of event message data transfer are as follows:

[0109] Step 1, the data generation end generates a data stream containing event message data.

[0110] Step 2, the data generation end calls the event management API to perform a preliminary format check on the data in the data stream.

[0111] Step 3, transmit the data stream through the API to the data transfer channel of the data processing system.

[0112] Step 4, perform event routing after a preliminary division of the data stream in the data transfer channel of the data processing system.

[0113] Step 5, if it is a simple event transfer process, the data transfer channel sends the preliminary division result of the data stream to the corresponding data consumer end according to the event routing information.

[0114] Step 6, the data consumer end performs subsequent event processing based on the received initial division result of the data stream, and then the data transfer ends.

[0115] Step 5', if it is a complex event transfer, the data transfer channel stores the event message data in the form of a first message queue. At the same time, the data transfer channel sends the initial division result of the data stream in the first message queue to the data processing engine.

[0116] Step 6', the data processing engine stores the initial division result in different second message queues for different business topics, processes and integrates the data in each second message queue to obtain new integrated event data.

[0117] Step 7', the data processing engine calls the event management API to check the newly integrated event data after processing and integration.

[0118] Step 8', the data processing engine calls the event management API to send the processed and integrated data to the data transfer channel.

[0119] Step 9', perform event routing on the processed and integrated data in the data transfer channel of the data processing system.

[0120] Step 10', the data transfer channel sends the processed and integrated data to the corresponding data consumer end according to the event routing information.

[0121] Step 11', the data consumer end performs subsequent event processing based on the received processed and integrated data, and then the data transfer ends.

[0122] In one embodiment, a structural and functional schematic diagram of a data processing system (data processing platform) generated according to the above data processing method is as Figure 8 shown, where Figure 8 from bottom to top are a data source (i.e., a data generation end), an event platform (data processing system), and a downstream application system (i.e., a data consumption end). Among them, in the event platform, it includes: event (data) collection, event (data) storage, event (data) calculation, event (data) subscription functions, etc. for event management processes, and also provides various operation options for the event management functions included in the corresponding event management processes, enabling users to perform interactive operations based on the display operation interface provided by the device where the data processing system is located.

[0123] In one embodiment, a data processing system is provided. This data processing system is used to implement the above data processing method. The data processing system includes: a data transmission channel and a data processing engine, where

[0124] The data transmission channel is used to obtain the data stream transmitted by the data generation end, store each event message data included in the data stream in the first message queue, and implement the transmission of event message data;

[0125] The data processing engine is used to classify and map the business theme of the event message data in the first message queue based on the event model to determine the event message data belonging to the same business theme;

[0126] The data processing engine is also used to store the event message data belonging to the same business theme in the second message queue belonging to the business theme. The second message queue is used to respond to the data subscription request of the data consumption end for the data of the business theme of the second message queue.

[0127] In one embodiment, the data transmission channel is also used to receive the data stream of the data generation end. The data stream is sent by the data generation end to the data processing system by calling the event message management interface; read the message header of each message data in the data stream, filter the message data whose message header contains the target field from the message data to obtain the event message data; and store each event message data in the first message queue.

[0128] In one embodiment, the data processing engine is also used to obtain the event model. The event model is a data recognition model with business theme attributes obtained according to a preset modeling method;

[0129] According to each event model, identify and read the event message data in the first message queue, map the event message to the business theme to which the event model belongs, and determine the event message data belonging to the same business theme.

[0130] In one embodiment, the data processing engine is further configured to process and integrate the event message data belonging to the same business theme according to the target event rule to obtain event data;

[0131] Add an event coding identifier to the event data to obtain target event data, and store the target event data in the second message queue of the belonging business theme.

[0132] In one embodiment, the data processing engine is further configured to call the event message management interface to listen for the data subscription requests of the data consumer; the data subscription request carries the target business theme identifier;

[0133] When the data subscription request sent by the data consumer is listened to, according to the target business theme identifier, determine the target second message queue in each second message queue, read the event message data in the target second message queue, and send the event message data in the target second message queue to the data consumer.

[0134] In one embodiment, the data processing engine of the data processing system is further configured to obtain the event message data in the first message queue and each second message queue according to a preset period;

[0135] Store the event message data in each message queue in the target partition of the distributed file database;

[0136] Check the coding order of the event coding identifiers carried in the event message data in each target partition, and determine whether the coding order of the event coding identifiers meets the preset coding conditions;

[0137] If the coding order of the event coding identifiers does not meet the preset coding conditions, output a data missing prompt message.

[0138] In one embodiment, as Figure 9 shown, a data processing apparatus 900 is provided, including: an acquisition module 910, a classification module 920, and a storage module 930, where:

[0139] The acquisition module 910 is configured to acquire the data stream transmitted by the data generation end, and store each event message data included in the data stream in the first message queue;

[0140] The classification module 920 is configured to classify and map the business theme of the event message data in the first message queue based on the event model, and determine the event message data belonging to the same business theme;

[0141] The storage module 930 is configured to store the event message data belonging to the same business theme in the second message queue of the belonging business theme, and the second message queue is used to respond to the data subscription request of the data consumer for the business theme of the second message queue.

[0142] In one embodiment, the obtaining module 910 is specifically configured to receive the data stream from the data generation end, and the data stream is sent to the data processing system by the data generation end invoking the event message management interface;

[0143] Read the message header of each message data in the data stream, screen the message data whose message header contains the target field from the message data, and obtain the event message data;

[0144] Store each event message data in the first message queue.

[0145] In one embodiment, the classification module 920 is specifically configured to obtain an event model, and the event model is a data recognition model with business theme attributes obtained according to a preset modeling method;

[0146] According to each event model, identify and read the event message data in the first message queue, map the event message to the business theme to which the event model belongs, and determine the event message data belonging to the same business theme.

[0147] In one embodiment, the storage module 930 is specifically configured to process and integrate the event message data belonging to the same business theme according to the target event rule to obtain event data;

[0148] Add an event coding identifier to the event data to obtain the target event data, and store the target event data in the second message queue of the belonging business theme.

[0149] In one embodiment, the device 900 further includes a sending module, configured to listen for the data subscription request of the data consumption end by invoking the event message management interface; the data subscription request carries the target business theme identifier;

[0150] When the data subscription request sent by the data consumption end is monitored, according to the target business theme identifier, determine the target second message queue in each second message queue, read the event message data in the target second message queue, and send the event message data in the target second message queue to the data consumption end.

[0151] In one embodiment, the device 900 further includes:

[0152] An obtaining module, configured to obtain the event message data in the first message queue and each second message queue according to a preset period;

[0153] A storage module, configured to store the event message data in each message queue in the target partition of the distributed file database;

[0154] A checking module, configured to check the encoding order of the event coding identifiers carried in the event message data in each target partition, and determine whether the encoding order of the event coding identifiers meets a preset encoding condition;

[0155] An alarming module, configured to output a data missing prompt message if the encoding order of the event coding identifiers does not meet the preset encoding condition.

[0156] For the specific limitations of the data processing device 900, reference can be made to the limitations on the data processing method in the foregoing text, which will not be elaborated here. Each module in the above data processing device 900 can be implemented in whole or in part by software, hardware, and their combination. The above modules can be embedded in the processor in the computer device in hardware form or be independent of the processor, or be stored in the memory in the computer device in software form, so as to facilitate the processor to call and execute the operations corresponding to the above respective modules.

[0157] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as Figure 10 shown. The computer device includes a processor, a memory, and a network interface connected through a system bus. Wherein, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store event message data. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data processing method is implemented.

[0158] Those skilled in the art can understand that Figure 10 the structure shown in

[0159] is only a block diagram of some structures related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have a different component layout.

[0160] In one embodiment, a computer device is further provided, including a memory and a processor. A computer program is stored in the memory, and when the processor executes the computer program, the steps in the foregoing method embodiments are implemented.

[0161] Those of ordinary skill in the art can understand that all or part of the processes in the methods of the above embodiments can be completed by instructing relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above various methods. Among them, any reference to a memory, storage, database, or other medium used in the various embodiments provided in the present application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0162] The technical features of the above embodiments can be combined arbitrarily. For the sake of brevity of description, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered to be within the scope described in this specification.

[0163] The above-described embodiments merely represent several implementation manners of the present application. Their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present application, several modifications and improvements can still be made, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A data processing method, characterized in that, The method is applied to a data processing system, and the method includes: Obtain a data stream transmitted by a data generation end, and store each event message data included in the data stream in a first message queue; Obtain an event model, where the event model is a data recognition model with business theme attributes obtained according to a preset modeling method; according to each of the event models, identify and read event message data in the first message queue, map the event messages to the business theme to which the event model belongs, and determine the event message data belonging to the same business theme; According to target event rules, perform processing of processing and integration on the event message data belonging to the same business theme to obtain event data; add an event coding identifier to the event data to obtain target event data, and store the target event data in a second message queue belonging to the business theme, where the second message queue is used to respond to a data subscription request of a data consumption end for the business theme of the second message queue; the data subscription request carries a target business theme identifier; Among them, according to the target business theme identifier carried in the data subscription request, perform a positioning query on the second message queue of the target business theme in each of the second message queues to determine a target second message queue; send the event message data read in the target second message queue to the data consumption end for the data consumption end to perform data consumption; Among them, the steps of constructing the event model include: perform data analysis on the initial data sent by the data generation end, determine the data ownership relationship required for the business scenarios under each business theme, and construct the event model that can be used for data identification and classification according to the defined business events. The data ownership relationship includes data source, logic, and data transmission destination.

2. The method according to claim 1, wherein The obtaining the data stream transmitted by the data generation end and storing each event message data included in the data stream in the first message queue includes: Receive the data stream of the data generation end, where the data stream is sent by the data generation end to the data processing system by calling an event message management interface; Read the message header of each message data in the data stream, and filter the message data whose message header contains a target field from the message data to obtain event message data; Store each of the event message data in the first message queue.

3. The method according to claim 1, wherein The second message queue is used to respond to a data subscription request of a data consumption end for the business theme of the second message queue, including: Call an event message management interface to monitor the data subscription request of the data consumption end; the data subscription request carries a target business theme identifier; When the data subscription request sent by the data consumption end is monitored, according to the target business theme identifier, determine a target second message queue in each of the second message queues, read the event message data in the target second message queue, and send the event message data in the target second message queue to the data consumption end.

4. The method according to claim 1, wherein The method further includes: Obtain the event message data in the first message queue and each of the second message queues according to a preset period; Store the event message data in each of the message queues in a target partition of a distributed file database; Check the encoding order of the event coding identifiers carried in the event message data in each of the target partitions, and determine whether the encoding order of the event coding identifiers meets a preset coding condition; If the encoding order of the event coding identifiers does not meet the preset coding condition, output a data missing prompt message.

5. The method according to claim 1 or 4, characterized in that, The event coding identifier includes: an event creator node number, an event code, and a global event tracking number.

6. A data processing system, characterized in that, The data processing system includes: a data transmission channel and a data processing engine, The data transmission channel is used to obtain a data stream transmitted by a data generation end, store each event message data included in the data stream in a first message queue, and implement the transmission of the event message data; The data processing engine is used to obtain an event model, and the event model is a data recognition model with business theme attributes obtained according to a preset modeling method; according to each of the event models, identify and read event message data in the first message queue, map the event messages to the business theme to which the event model belongs, and determine the event message data belonging to the same business theme; the steps of constructing the event model include: performing data analysis on the initial data sent by the data generation end, determining the data ownership relationship required for the business scenarios under each business theme, and constructing the event model that can be used for data recognition and classification according to the defined business events. The data ownership relationship includes data source, logic, and data transmission destination; The data processing engine is further used to process and integrate the event message data belonging to the same business theme according to a target event rule to obtain event data; add an event coding identifier to the event data to obtain target event data, and store the target event data in a second message queue belonging to the business theme. The second message queue is used to respond to a data subscription request of a data consumer for the business theme of the second message queue; the data subscription request carries a target business theme identifier; The data processing engine is further used to perform a positioning query of the second message queue of the target business theme in each of the second message queues according to the target business theme identifier carried in the data subscription request to determine a target second message queue; send the event message data read in the target second message queue to the data consumer for the data consumer to consume data.

7. A data processing device, characterized in that, The device includes: An acquisition module, configured to acquire a data stream transmitted by a data generation end, and store each event message data included in the data stream in a first message queue; A classification module, configured to obtain an event model, where the event model is a data recognition model with business theme attributes obtained according to a preset modeling method; according to each event model, identify and read event message data in the first message queue, map the event messages to the business theme to which the event model belongs, and determine the event message data belonging to the same business theme; the steps of constructing the event model include: performing data analysis on the initial data sent by the data generation end, determining the data ownership relationship required for the business scenario under a certain business theme, and constructing an event model that can be used for data recognition and classification according to the defined business events, where the data ownership relationship includes data source, logic, and data transmission destination; A storage module, configured to process and integrate the event message data belonging to the same business theme according to the target event rule to obtain event data; add an event coding identifier to the event data to obtain target event data, and store the target event data in the second message queue of the business theme to which it belongs, where the second message queue is used to respond to the data subscription request of the data consumer for the data of the business theme of the second message queue; the data subscription request carries a target business theme identifier; Wherein, according to the target business theme identifier carried in the data subscription request, perform a positioning query on the second message queue of the target business theme in each second message queue to determine the target second message queue; send the event message data read in the target second message queue to the data consumer for the data consumer to consume data.

8. The device according to claim 7, wherein The device further includes an acquisition module, an inspection module, and an alarm module, where: The acquisition module is configured to obtain event message data in the first message queue and each second message queue according to a preset period; The storage module is configured to store the event message data in each message queue in a target partition of a distributed file database; The inspection module is configured to inspect the encoding order of the event coding identifiers carried in the event message data in each target partition, and determine whether the encoding order of the event coding identifiers meets a preset encoding condition; The alarm module is configured to output a data missing prompt message if the encoding order of the event coding identifiers does not meet the preset encoding condition.

9. A computer device, comprising a memory and a processor, the memory storing a computer program, characterized in that, When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, the steps of the method according to any one of claims 1 to 5 are implemented.

Images