A data processing method, apparatus, device, and storage medium based on message middleware

Abstract

This application provides a data processing method, apparatus, device, and storage medium based on message middleware. The method includes: obtaining a data collection object with an event sequence number according to a data collection request from a business party; writing the data collection object into the corresponding message topic of the message middleware according to the event sequence number and event type to obtain business-isolated message queue data; obtaining a target message from the message queue data through several preset consumer instances, and parsing the target message according to the event sequence number to generate a first data object; performing triple data processing on the first data object to generate a second data object; and sending the second data object to downstream devices according to preset routing rules. This application constructs an end-to-end data processing link for data collection, message middleware writing, distributed consumption, triple data processing, and routing rule-based distribution, effectively improving data processing efficiency and business decoupling capabilities.

Description

Technical Field

[0001] This application relates to the field of computer communication technology, and in particular to a data processing method, apparatus, device and storage medium based on message middleware. Background Technology

[0002] In digital operations scenarios, enterprise applications face the need for large-scale user behavior data collection and processing. This type of data is characterized by diverse behavior types (such as heartbeats, logins, sharing, access, clicks, registrations, etc.), massive data volume, high reliability requirements, and low business coupling. Furthermore, the collection layer should evolve rapidly independently of business iterations.

[0003] However, existing data acquisition and processing technologies typically use direct database writes or simple REST interfaces to directly store data in the database, which can easily lead to single points of failure under high concurrency conditions. Furthermore, data acquisition logic and business logic are often coupled in the same process, requiring the data acquisition module to be adjusted synchronously when the business system is updated. This increases development and maintenance costs and severely impacts the system's scalability and stability. Summary of the Invention

[0004] This application provides a data processing method, apparatus, device, and storage medium based on message middleware, which solves the problem that related technologies cannot guarantee the decoupling of data acquisition and business logic in high-concurrency scenarios.

[0005] The first aspect of this application provides a data processing method based on message middleware, the data processing method based on message middleware includes:

[0006] Obtain the collected data object with the event sequence number according to the data collection request from the business party;

[0007] The collected data object is written into the corresponding message topic of the message middleware according to the event sequence number and event type to obtain business-isolated message queue data;

[0008] The target message is obtained from the message queue data by a number of preset consumer instances, and the target message is parsed according to the event sequence number to generate a first data object;

[0009] The first data object undergoes triple data processing to generate the second data object;

[0010] The second data object is sent to the downstream device according to the preset routing rules.

[0011] Optionally, in the first implementation of the first aspect of this application, the step of writing the collected data object into the corresponding message topic of the message middleware according to the event sequence number and event type to obtain business-isolated message queue data includes:

[0012] Based on the event type field of the collected data object, the collected data object is classified and identified, and a topic identifier corresponding to the event type is generated;

[0013] By combining the event sequence number with the topic identification information, a message key value for writing to the message middleware is determined, and the target partition corresponding to the collected data object is determined based on the message key value;

[0014] The collected data object is written into the corresponding message topic in the message middleware based on the topic identifier, and the message key value is written into the target partition as the partitioning basis to generate business-isolated message queue data; wherein, the message topic includes several target partitions.

[0015] Optionally, in a second implementation of the first aspect of this application, the step of obtaining a target message from the message queue data through a preset number of consumer instances, and parsing the target message according to the event sequence number to generate a first data object includes:

[0016] The target message is obtained from the message queue data by using a number of preset consumer instances;

[0017] The order of the target message is checked according to the event sequence number corresponding to the target message, and the sorting result of the order check is stored in correspondence with the target message to generate a message set to be parsed;

[0018] The target messages in the message set to be parsed are deserialized, and parsed data is generated according to the field definitions in the message body.

[0019] The parsed data is associated with the event sequence number, and a first data object with a unified field format is generated in the memory buffer.

[0020] Optionally, in the third implementation of the first aspect of this application, the step of performing triple data processing on the first data object to generate the second data object includes:

[0021] The first data object is filled in with fields and anomalies are corrected to generate a cleaned data object;

[0022] Based on the user identifier of the cleaned data object, user profile attributes are obtained from an external data source, and the user profile attributes are merged with the cleaned data object to generate an enriched data object.

[0023] An idempotent key is generated based on the user identifier, event identifier, and time information in the enriched data object, and a second data object is obtained by performing a repeatability check on the idempotent key.

[0024] Optionally, in the fourth implementation of the first aspect of this application, the step of performing field completion and anomaly correction on the first data object to generate a clean data object includes:

[0025] Perform field integrity checks on the first data object based on the event sequence number and timestamp of the first data object;

[0026] When a missing field is detected, compensation information is retrieved from the message topic data based on the event sequence number, and the compensation information is merged with the first data object to generate a compensated third data object.

[0027] The time field of the third data object is converted to a time format to generate a time window label, and the time window label is written into the third data object to obtain intermediate data;

[0028] The device field of the intermediate data is encoded and replaced to generate a standardized device field, and the standardized device field is merged into the intermediate data to obtain extended data;

[0029] According to predefined range rules, the numerical fields in the extended data that are out of range are corrected to generate corrected numerical fields, and the corrected numerical fields are updated to the extended data to obtain a cleaned data object.

[0030] Optionally, in the fifth implementation of the first aspect of this application, the step of generating an idempotent key based on the user identifier, event identifier, and time information in the enriched data object, and obtaining a second data object by performing a repeatability check on the idempotent key, includes:

[0031] Based on the user identifier, event identifier, event occurrence time, and original event sequence number in the enriched data object, an idempotent key is generated;

[0032] The existence of the idempotent key is determined in the distributed idempotent index by performing a distributed check on the idempotent key.

[0033] If the idempotent key already exists, the enriched data object is recorded as a duplicate message and written to the duplicate record table.

[0034] If the idempotent key does not exist, the idempotent key is inserted into the distributed idempotent key to determine the enriched data object as the second data object.

[0035] Optionally, in a sixth implementation of the first aspect of this application, the step of sending the second data object to the downstream device according to a preset routing rule includes:

[0036] Based on the user identifier in the second data object, obtain the user profile tag corresponding to the user identifier, and use the user profile tag as the first routing condition;

[0037] A combined label is generated based on the event type and event occurrence time of the second data object, and the combined label is used as the second routing condition;

[0038] A load weight value is generated based on the current load status of the downstream devices, and the load weight value is used as a third routing condition;

[0039] The target downstream device of the second data object is determined by weighted matching of the first routing condition, the second routing condition, and the third routing condition.

[0040] A second aspect of this application provides a message middleware-based data processing apparatus, which is used to implement a message middleware-based data processing method. The message middleware-based data processing apparatus includes:

[0041] The acquisition module is used to obtain the collected data object with the event sequence number according to the data collection request from the business party;

[0042] The writing module is used to write the collected data object into the corresponding message topic of the message middleware according to the event sequence number and event type, so as to obtain the business-isolated message queue data.

[0043] The parsing module is used to obtain the target message from the message queue data through a number of preset consumer instances, and parse the target message according to the event sequence number to generate a first data object;

[0044] The processing module is used to perform triple data processing on the first data object to generate the second data object;

[0045] The sending module is used to send the second data object to the downstream device according to the preset routing rules.

[0046] A third aspect of this application provides an electronic device, including a memory and a processor, wherein the processor is configured to execute a computer program stored in the memory, and when the processor executes the computer program, it implements the steps of the message middleware-based data processing method provided in the first aspect of this application.

[0047] The fourth aspect of this application provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it implements the steps of the data processing method based on message middleware provided in the first aspect of this application.

[0048] In summary, the data processing method, apparatus, device, and storage medium based on message middleware provided in this application involves: obtaining a data collection object with an event sequence number based on the data collection request from the business party; writing the data collection object into the corresponding message topic of the message middleware according to the event sequence number and event type to obtain business-isolated message queue data; retrieving a target message from the message queue data through several preset consumer instances, parsing the target message according to the event sequence number to generate a first data object; performing triple data processing on the first data object to generate a second data object; and sending the second data object to downstream devices according to preset routing rules. Through the implementation of this application, an end-to-end data processing link is constructed for data collection, message middleware writing, distributed consumption, triple data processing, and routing rule-based distribution, effectively improving data processing efficiency and business decoupling capabilities. Attached Figure Description

[0049] Figure 1 A flowchart illustrating the data processing method based on message middleware provided in an embodiment of this application;

[0050] Figure 2 A schematic diagram of the program modules of a data processing device based on message middleware provided in an embodiment of this application;

[0051] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0052] To make the inventive objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0053] To address the issue of decoupling data acquisition from business logic in high-concurrency scenarios in related technologies, embodiments of this application provide a data processing method based on message middleware, such as... Figure 1This is a flowchart illustrating the message middleware-based data processing method provided in this embodiment. The message middleware-based data processing method includes the following steps:

[0054] Step 110: Obtain the data collection object with the event sequence number according to the data collection request from the business party.

[0055] Specifically, the unified data collection interface adopts a multi-protocol adaptation layer, including but not limited to SDK, HTTP, and gRPC interfaces. It uses asynchronous non-blocking I / O and connection pooling strategies to receive reports from Web, App, and mini-program. The receiving logic performs basic field verification and signature verification on the inbound packets. Then, it generates a globally unique event sequence number through a distributed ID generator (such as an ordered identifier based on timestamps and machine sequences or a combination of UUID and snowflake algorithm). This sequence number, along with the timestamp, client environment information, and metadata, is encapsulated into a data collection object and written to a local short-term buffer to support short-term retries, reporting quality monitoring, and mapping records of the original packets. At the same time, the reported fields are lightweight standardized, including field name mapping and preliminary time format specification for unified parsing by the backend.

[0056] Step 120: Write the collected data object into the corresponding message topic of the message middleware according to the event sequence number and event type to obtain the business-isolated message queue data.

[0057] Specifically, when writing collected data objects into the corresponding message topics of the message middleware based on event sequence numbers and event types to obtain business-isolated message queue data, message routing logic binds the collected data objects to their corresponding message topics, and distinguishes different business flows based on event types. The event sequence number serves as a message key to guide data partitioning, ensuring that events from the same user or of the same type can be stably written to the same partition, maintaining logical order and controllability. The division of message topics provides business isolation capabilities for subsequent processing; each type of event is stored in an independent topic, and different topics do not interfere with each other, avoiding the impact on overall throughput due to excessive pressure from a single business.

[0058] In one optional implementation of this embodiment, the step of writing the collected data object into the corresponding message topic of the message middleware according to the event sequence number and event type to obtain business-isolated message queue data includes: classifying and identifying the collected data object according to the event type field of the collected data object to generate a topic identifier corresponding to the event type; determining the message key value for writing to the message middleware by combining the event sequence number and the topic identifier information, and determining the target partition corresponding to the collected data object based on the message key value; writing the collected data object into the corresponding message topic in the message middleware based on the topic identifier, and writing the message key value as the partitioning basis into the target partition to generate business-isolated message queue data.

[0059] In this embodiment, a topic identifier can contain multiple target partitions, and a target partition can only belong to one topic identifier. During the data processing, the event type field in the collected data object is read and identified. The event type field describes the type of behavior triggered by the user in a specific application scenario. For example, in the same e-commerce platform application, a user may perform different behaviors such as browsing products, adding items to the shopping cart, or submitting an order. This field exists in the collected data object in a predefined encoded form. After obtaining this field, it is matched with an internally maintained event type mapping table. The mapping table describes the correspondence between event types and topic identifiers. The topic identifier is used to identify logical channels in the message middleware. Different topic identifiers are independent of each other and are used to distinguish different business data streams. Through the above matching process, the collected data object is assigned a topic identifier corresponding to its event type, thereby completing the classification and identification of the data object, enabling data from the same business scenario to be grouped into the same logical set. After the topic identifier is generated, the event sequence number and topic identifier information are combined to generate a message key value. The event sequence number is a globally unique identifier generated during the data acquisition phase, used to mark the order and uniqueness of a single event. The message key guides the data distribution strategy of the message middleware during internal storage. The combination process constructs a composite identifier by using the topic identifier as the high-order information and the event sequence number as the low-order information, ensuring stable distribution characteristics for data under the same topic while maintaining event uniqueness. A partition in the message middleware refers to a physical or logical sub-unit within a topic used for parallel storage and consumption; each partition is responsible for storing a portion of the data within the topic. By performing a deterministic mapping operation on the generated message key, the target partition where the acquired data object should be written is determined, ensuring that events with the same topic identifier and correlated sequence numbers are assigned to the same partition. After clarifying the topic identifier and target partition, the acquired data object is written to the message topic corresponding to the topic identifier in the message middleware. During the writing process, the topic identifier is used to locate the logical channel of the data, while the message key, as the partitioning basis, is written along with the data object, enabling the message middleware to store the data object in the aforementioned determined target partition based on this key value. Taking e-commerce platform applications as an example, the topic identifier corresponding to a product browsing event is used to distinguish it from the topic identifier corresponding to an order placement event. When browsing event data is written, it carries a message key value generated by combining the event sequence number and the topic identifier, thus being written to a designated partition under the browsing event topic. Order placement event data, on the other hand, is written to a separate topic and its partition. The two are isolated from each other in their storage structure and do not affect each other. Through this writing method, data of different event types form independent data sets in the message middleware, while data of the same event type maintains a controllable order and consistency within the partition.Understandably, the introduction of topic identifiers is used to achieve data isolation at the business level, while the determination of message key values ​​and partitions is used to achieve data distribution control at the storage level. The combination of the two enables the collected data objects to have both business differentiation and distribution determinism when writing to the message middleware, thereby generating business-isolated message queue data, which effectively improves the stability and controllability of data writing and subsequent processing in high-concurrency scenarios.

[0060] Step 130: Obtain the target message from the message queue data through several preset consumer instances, and parse the target message according to the event sequence number to generate the first data object.

[0061] Specifically, in the process of retrieving target messages from business-isolated message queues using several pre-defined consumer instances and parsing the target messages to generate the first data object based on event sequence numbers, a consumer group mechanism is used to pull messages from different partitions. Each consumer instance can independently parse and process the message set it is responsible for. The parsing process first verifies the order of messages based on event sequence numbers to avoid out-of-order delivery and loss. Then, the message body is deserialized, transforming the content originally stored in binary or specific encoding formats in the message middleware into a standardized data structure. The first data object is thus generated, possessing unified field naming and structure, covering key information such as user identifier, event attributes, and time, and is stored in a memory buffer awaiting further processing.

[0062] In one optional implementation of this embodiment, the steps of obtaining target messages from message queue data through a preset number of consumer instances, parsing the target messages according to event sequence numbers, and generating a first data object include: obtaining target messages from message queue data through a preset number of consumer instances; performing sequence checks on the target messages according to the event sequence numbers corresponding to the target messages, and storing the sorting results of the sequence checks in correspondence with the target messages to generate a set of messages to be parsed; performing deserialization operations on the target messages in the set of messages to be parsed, generating parsed data according to the field definitions in the message body; establishing a correspondence between the parsed data and the event sequence numbers, and generating a first data object with a unified field format in a memory buffer.

[0063] In this embodiment, when message queue data enters the consumption stage, several pre-defined consumer instances pull the target message from the message queue in parallel. Consumer instances are stateless processes deployed in a distributed environment. They work collaboratively as consumer groups, with each instance responsible for sequentially reading messages from a specific partition, thus covering the entire message queue's data flow. Since the partition routing key ensures that related events are concentrated in the same partition during the production stage, the consumer instances maintain business continuity during retrieval. To maintain the message reading order, the consumer instances record an offset, a numerical pointer to the current position in the message queue. Based on this pointer, new messages can be continuously retrieved, avoiding duplicate consumption. After retrieving the target message, the message's order is checked based on the event sequence number attached to the message. The event sequence number is a globally unique and ordered identifier, thus serving as a crucial basis for determining whether messages are out of order or lost. The order check process includes checking the difference between adjacent sequence numbers; if a discontinuity occurs, it indicates out-of-order or lost messages. The anomaly is recorded in the log and compensation logic is triggered. Simultaneously, to ensure orderly subsequent processing, the target messages are sorted, and the sorting results are stored in correspondence with the message body, forming an ordered set called the message set to be parsed. The message set to be parsed acts as a buffer pool built on the consumer side, ensuring that the subsequent parsing process unfolds in the correct chronological order. For example, if the message with event sequence number 1002 arrives at the consumer before the message with sequence number 1001, the sorting logic will adjust the order in the message set to be parsed, placing 1001 before 1002. After the message set to be parsed is constructed, deserialization is performed on the target messages within it. Deserialization is the process of converting the message body from a storage or transmission format into an operable object. The message body can be encoded using data formats such as JSON, Avro, or Protobuf, which employ compact or structured representations for cross-platform transmission. The deserialization logic restores the original binary or text content into a data structure based on the field definitions in the message body. Each message generates parsed data after deserialization; this unit consists of field names and field values, which can be directly accessed and modified in memory. Subsequently, the parsed data is bound to the corresponding event sequence number to ensure that the original message can be traced through the sequence number in subsequent stages and to maintain processing consistency. This mapping is stored in a key-value mapping table, with the event sequence number as the key and the parsed data as the value. The parsed data is written to a memory buffer and, during the writing process, is converted into a first data object with a standard format according to a unified field definition. The first data object is the input unit for subsequent cleaning, enrichment, and deduplication stages. Its unified field format means that regardless of the source of the original message, a structured and consistent data representation can be obtained after the parsing stage.For example, messages from mobile devices and messages from web pages may differ in field naming or format, but after generating the first data object, they will be standardized to the same structure, thus facilitating unified operation in subsequent processing stages.

[0064] Step 140: Perform triple data processing on the first data object to generate the second data object.

[0065] Specifically, in the process of performing triple data processing on the first data object to generate the second data object, three types of technical operations are executed sequentially: cleaning, enrichment, and deduplication. Cleaning includes completing missing fields, converting fields with inconsistent formats, and correcting outliers to ensure data integrity and consistency. Enrichment relies on external data sources, such as user profile databases or dimension tables, to obtain user-related attributes based on user identifiers and add them to the first data object to form richer contextual information. Deduplication is achieved by constructing idempotent keys (based on a combination of user identifier, event identifier, event time, and event sequence number) and performing existence checks in a distributed index to exclude duplicate data.

[0066] In one optional implementation of this embodiment, the step of performing triple data processing on the first data object to generate the second data object includes: performing field completion and anomaly correction on the first data object to generate a cleaned data object; obtaining user profile attributes from an external data source based on the user identifier of the cleaned data object, merging the user profile attributes with the cleaned data object to generate an enriched data object; generating idempotent keys based on the user identifier, event identifier, and time information in the enriched data object, and obtaining the second data object by performing repeatability verification on the idempotent keys.

[0067] In this embodiment, when the first data object enters the subsequent processing stage, its fields are completed and anomaly corrected. Field completion means that when a data object is found to be missing necessary information, it is filled in through cross-source methods. For example, if the message sequence number exists but the timestamp is missing, the original data entry can be relocated based on the mapping relationship between the sequence number and storage offset established in the message topic, and the missing timestamp can be filled back into the current object. If the missing information is device information, another complete data object uploaded by the same user at a similar time can be retrieved from the short-term cache or log storage using the user identifier, and its device field can be added to the current object. The anomaly correction process includes adjusting fields that do not conform to the predefined range or the standard format. For example, when the event duration field has a negative value, the value is recalculated based on the event occurrence time and end time and overwrites the original field. Or, when the time field is not uniformly in the standard UTC format, a standard timestamp tag is generated by the time conversion module and written into the data object. Through the completion and correction operations, the first data object is transformed into a clean data object with a complete structure and field values ​​conforming to the defined range. After the cleaned data object is generated, user profile attributes are retrieved from external data sources based on the user identifiers. User profile attributes refer to static or dynamic characteristics of a user accumulated over time in a user profile library or dimension table, such as age, gender, region, device preference, interest tags, or spending power level. The retrieval process can be completed through real-time API calls or pre-loading a cache, querying the user identifier and extracting relevant fields. The retrieved user profile attributes need to be merged with the cleaned data object. The merging method uses a field mapping and key-value association strategy to align the profile attributes with the original fields in the data object and expand them into a new composite structure. For example, the cleaned data object only contains user identifiers and event information, while the merged enriched data object additionally includes the user's region, interest categories, and spending level information, thus giving the data object a higher semantic dimension. After the enriched data object is constructed, idempotent keys are generated based on the user identifiers, event identifiers, and time information. Idempotent keys are generated based on multiple fields in the enriched data object that are already defined and have business-distinguishing capabilities. The user identifier defines the subject to which the event belongs, the event identifier distinguishes different business behavior types, the event occurrence time uniquely positions the event over time, and the original event sequence number supplements the order and uniqueness information from the data collection phase. By combining these fields in a predetermined order, a combined identifier that uniquely represents a single business event is formed, and this combined identifier serves as the idempotent key. Since the generation of the idempotent key depends only on the business fields themselves and not on the processing node or processing time, the same business event always corresponds to the same idempotent key in scenarios of repeated delivery or parallel processing, thus supporting duplicate verification and uniqueness determination.The generated idempotent keys are sent to a distributed idempotent index for verification. This index is a high-speed query structure built on a key-value database or distributed cache, capable of determining whether an idempotent key already exists in a very short time. If the idempotent key already exists in the index, it means that the corresponding enriched data object has already been processed. In this case, the object is marked as duplicate data and written to the duplicate record table. If the idempotent key does not exist in the index, it is inserted into the index, and the enriched data object is recognized as a valid output object. After this process, the final second data object is a high-quality data unit that ensures uniqueness based on cleaning and enrichment. For example, if user U567 reports a click event twice at the same timestamp due to network jitter, the two enriched data objects will receive the same identifier after generating the idempotent key. The first time it enters the index, it is written; the second time it enters, it is recognized as a duplicate, thus outputting only one valid second data object. This processing method ensures the accuracy of statistics and analysis and avoids data duplication interfering with the overall calculation.

[0068] In one optional implementation, the step of performing field completion and anomaly correction on the first data object to generate a cleaned data object includes: performing field integrity checks on the first data object based on the event sequence number and timestamp; when missing fields are detected, retrieving compensation information from the message topic data based on the event sequence number, merging the compensation information with the first data object, and generating a compensated third data object; converting the time field of the third data object to a time format to generate a time window label, writing the time window label into the third data object to obtain intermediate data; encoding and replacing the device field of the intermediate data to generate a standardized device field, merging the standardized device field into the intermediate data to obtain extended data; correcting out-of-range numerical fields in the extended data according to predefined range rules to generate corrected numerical fields, and updating the corrected numerical fields to the extended data to obtain a cleaned data object.

[0069] In this embodiment, when performing field integrity checks on the first data object, the required fields should be checked item by item based on the event sequence number and timestamp carried, including user identifier, event type, time field, device information, and key business fields. During the verification process, a sequence number-based indexing and positioning mechanism is used to search for related entries in the message topic data to determine the compensation range. When a missing field is detected, a bounded search is initiated in the message topic data based on the event sequence number. The search range can be defined by a time window or sequence number offset. The optimal compensation record is selected from the retrieved candidate entries according to time proximity, field integrity, and consistency score. Then, the selected compensation record is merged with the original object at the field level according to field priority and conflict resolution rules to generate the compensated third data object. Subsequently, a unified format conversion is performed on the time field of the third data object, normalizing the original time value to the configured time zone and time unit. The timestamp is then rounded down according to the configured pane granularity (the length of time corresponding to each interval when a continuous time axis is divided into several adjacent time intervals according to a pre-set time unit, i.e., how long a time window covers) to generate a time window label. This label is then written into the third data object to form intermediate data. The time window label is used for subsequent fragmentation aggregation and routing determination. Next, the device fields in the intermediate data undergo encoding replacement. First, the original device strings are preprocessed, including noise reduction, case normalization, and delimiter splitting. Then, the preprocessed results are mapped to standard device codes, and the generated standardized device fields are incorporated into the intermediate data to obtain extended data. Finally, the numerical fields in the extended data are validated according to predefined range rules. Fields exceeding the upper and lower limits are recalculated, truncated, or set to empty according to priority. When the event duration is negative and has both start and end times, the difference is recalculated to overwrite the original value; otherwise, it is truncated or marked as empty according to the minimum acceptable value. The corrected numerical fields are then written back to the extended data to produce cleaned data objects. All compensation, conversion, and correction operations are recorded with audit information and exception entries for traceability and governance.

[0070] In one optional implementation, when an enriched data object enters the uniqueness determination stage, an idempotent key is first constructed based on the user identifier, event identifier, event occurrence time, and original event sequence number. An idempotent key is a composite identifier used to identify a single business event. Its generation process is not a simple concatenation, but rather a deterministic string formed by normalizing the aforementioned fields in a fixed order. The user identifier distinguishes different subjects, the event identifier distinguishes behavior types, the event occurrence time limits the time dimension, and the original event sequence number eliminates potential conflicts within the same time granularity. During the generation process, the time field is uniformly converted to a millisecond-level timestamp and encoded with other fields using a fixed-length encoding, ensuring that enriched data objects from different sources generate completely consistent idempotent keys under the same business semantics. For example, in a user click scenario on an e-commerce platform, if the same user triggers a click on the same product at the same time, even if it is reported multiple times due to network jitter, its enriched data object will still obtain the same idempotent key after the above combination and encoding. After the idempotent key is generated, it is sent to a distributed idempotent index for existence checking. A distributed idempotent index is a key-value index structure shared across nodes. Its core capability lies in supporting fast queries and condition writes under high concurrency conditions, ensuring consistency in the judgment of the same key value when multiple processing instances access it simultaneously. During the checking process, the idempotent key is submitted as a query condition to the index node, which then returns whether the key has already been recorded. To avoid judgment bias caused by concurrent competition, the index internally employs an atomicity verification mechanism (meaning that in a concurrent environment, the actions of "judging whether a condition is true" and "performing the corresponding operation based on the judgment" are bound into an indivisible whole process, ensuring that it will not be interrupted or modified by other concurrent operations during execution, thus guaranteeing the uniqueness and consistency of the result). This makes "judging existence" and "marking existence" logically inseparable operational units. For example, when an enriched data object generates an idempotent key, two actions need to be completed in the distributed idempotent index: first, to determine whether the idempotent key already exists; and second, to write the idempotent key to the index if it does not exist. If these two actions are split into independent operations, in high-concurrency scenarios, multiple processing instances may simultaneously determine that "the idempotent key does not exist" and write to the index at the same time, leading to duplicate data being mistakenly identified as valid data. Atomicity checking mechanisms are designed to avoid such concurrency conflicts. When the distributed idempotent index returns that the idempotent key already exists, the current enriched data object is determined to be a duplicate message. The duplicate message does not continue into the main data stream but is written to a dedicated duplicate record table. The duplicate record table stores key information about duplicate events, including user identifier, event identifier, event occurrence time, and the corresponding idempotent key. It also records the time point of the duplicate occurrence and the source topic to support statistical analysis and troubleshooting.Continuing with the e-commerce click scenario, if a user triggers multiple clicks during page lag, all enriched data objects except the initial click will be identified as duplicate messages and added to the duplicate record table, thus preventing click counts from being accumulated multiple times. When the distributed idempotent index returns that the idempotent key does not exist, the idempotent key is written into the distributed idempotent index for registration, and the corresponding enriched data object is confirmed and output as the second data object. The writing process and existence check are completed in the same logical flow, ensuring that multiple instances do not simultaneously confirm the same idempotent key in a concurrent environment. The second data object retains all the information of the enriched data object in terms of content, and has passed the uniqueness check in terms of semantics, representing a real and valid business event. Taking the order payment scenario as an example, even if the payment result callback is pushed multiple times, as long as its business identifier and time information are consistent, only one payment event will ultimately pass the idempotency check and become the second data object to enter the downstream processing chain. Through the above scheme, duplicate events can still be accurately identified and filtered in high-concurrency and unstable network environments, effectively improving the accuracy and consistency of the overall data processing results.

[0071] Step 150: Send the second data object to the downstream device according to the preset routing rules.

[0072] Specifically, during the process of sending the second data object to the downstream device according to preset routing rules, the configured routing rules are applied to the second data object. These routing rules combine event type, user-related attributes, and time information to determine the specific target downstream device. The routing rules can be statically pre-configured or a dynamic matching mechanism to adapt to different business needs. By judging the routing conditions attached to the second data object, a transmission request is generated and sent to the corresponding downstream device interface, such as a real-time computing engine, offline data warehouse, or audit storage system.

[0073] In one optional implementation of this embodiment, the step of sending the second data object to the downstream device according to a preset routing rule includes: obtaining a user profile tag corresponding to the user identifier in the second data object, and using the user profile tag as a first routing condition; generating a combined tag based on the event type and event occurrence time of the second data object, and using the combined tag as a second routing condition; generating a load weight value based on the current load status of the downstream device, and using the load weight value as a third routing condition; and determining the target downstream device of the second data object by performing a weighted matching of the first routing condition, the second routing condition, and the third routing condition.

[0074] In this embodiment, the tag set corresponding to the user identifier carried by the second data object is retrieved from the image tag table or cache, and this tag set is used as the first routing condition. Then, a combined tag is formed by concatenating the event type and the time window tag as the second routing condition. The time window tag adopts the aforementioned time pane granularity to identify the time slice of the event. Then, real-time load indicators (such as CPU utilization, memory usage, current concurrent connections, and queue length) of the target downstream devices are pulled from the routing configuration center or monitoring acquisition terminal, and these indicators are normalized and weighted to produce a load weight value representing the receiving capacity. This value is used as the third routing condition. Next, weighted matching is performed on the three types of routing conditions. The matching engine uses a scoring function to combine the tag matching degree, combined tag compatibility, and load weight according to configurable weight coefficients to form a total score. In the tag matching stage, an inverted index or bitmap is used to accelerate the matching. The inverted index maps the tag to the target downstream list containing the tag, and the bitmap uses binary bits to represent the existence relationship of the tag on the target set to support bit operation acceleration. To ensure the stability and interpretability of the decision-making process, the matching module applies priority rules and threshold controls after evaluation: when the total score is higher than the configured threshold, the target is included in the candidate set; otherwise, a backup routing strategy is adopted. For targets with similar scores in the candidate set, a deterministic selection strategy, such as consistent hashing or weighted round-robin, is applied to complete the final determination. Consistent hashing is a hashing scheme that maps keys and nodes to a logical ring, thereby reducing the amount of remapping when nodes change. To cope with transient fluctuations in downstream load, a dynamic feedback loop is introduced to make the load weight adaptive. Specifically, the acceptance rate, rejection rate, and processing latency are written back to the weight calculation module as real-time signals, triggering weight recalculation and cache refresh, so that subsequent routing decisions can reflect the latest carrying capacity. To improve matching efficiency and scalability, the label-target matrix can be pre-calculated in the routing engine and stored in a sparse representation. At the same time, hot labels are cached locally to reduce cross-node communication latency, and circuit breaking and degradation strategies are added to the scoring calculation to avoid repeatedly switching targets under extreme loads. For example, when the tag set includes "high-paying users" and the combined tag is "payment - peak", the billing real-time engine node will be matched first under the configuration of weight coefficient biased towards low latency and priority. If the load weight of the node exceeds the tolerance threshold, the scoring mechanism will automatically tilt to the backup node with a lighter load. The final routing decision and its scoring basis will be recorded in the audit log for backtracking and policy optimization.

[0075] According to the data processing method based on message middleware provided in this application, a data collection object with an event sequence number is obtained based on the data collection request from the business party; the data collection object is written into the corresponding message topic of the message middleware according to the event sequence number and event type to obtain business-isolated message queue data; target messages are obtained from the message queue data through several preset consumer instances, and the target messages are parsed according to the event sequence number to generate a first data object; the first data object is subjected to three-fold data processing to generate a second data object; and the second data object is sent to downstream devices according to preset routing rules. This application constructs an end-to-end data processing link for data collection, message middleware writing, distributed consumption, three-fold data processing, and routing rule-based distribution, effectively improving data processing efficiency and business decoupling capabilities.

[0076] Figure 2 This application provides a message middleware-based data processing apparatus, which can be used to implement the message middleware-based data processing method described in the foregoing embodiments. For example... Figure 2 As shown, the data processing device based on message middleware mainly includes:

[0077] The acquisition module 10 is used to acquire data objects with event sequence numbers based on the data acquisition request from the business party.

[0078] The writing module 20 is used to write the collected data object into the corresponding message topic of the message middleware according to the event sequence number and event type, so as to obtain the business-isolated message queue data.

[0079] The parsing module 30 is used to obtain the target message from the message queue data through a number of preset consumer instances, and parse the target message according to the event sequence number to generate the first data object;

[0080] Processing module 40 is used to perform triple data processing on the first data object to generate the second data object;

[0081] The sending module 50 is used to send the second data object to the downstream device according to the preset routing rules.

[0082] In one optional implementation of this embodiment, the writing module is specifically used to: classify and identify the collected data objects according to the event type field of the collected data objects, and generate a topic identifier corresponding to the event type; determine the message key value for writing to the message middleware by combining the event sequence number and the topic identifier information, and determine the target partition corresponding to the collected data object based on the message key value; write the collected data object to the corresponding message topic in the message middleware based on the topic identifier, and write the message key value as the partitioning basis to the target partition, thereby generating business-isolated message queue data.

[0083] In one optional implementation of this embodiment, the parsing module is specifically used to: obtain target messages from message queue data through a preset number of consumer instances; perform sequential verification on the target messages according to the event sequence number corresponding to the target messages, and store the sorting result of the sequential verification with the target messages to generate a set of messages to be parsed; perform deserialization operation on the target messages in the set of messages to be parsed, and generate parsed data according to the field definitions in the message body; establish a correspondence between the parsed data and the event sequence number, and generate a first data object with a unified field format in the memory buffer.

[0084] In one optional implementation of this embodiment, the processing module is specifically used to: perform field completion and anomaly correction on the first data object to generate a cleaned data object; obtain user profile attributes from an external data source based on the user identifier of the cleaned data object, merge the user profile attributes with the cleaned data object to generate an enriched data object; generate idempotent keys based on the user identifier, event identifier, and time information in the enriched data object, and obtain a second data object by performing repeatability verification on the idempotent keys.

[0085] In an optional implementation of this embodiment, the processing module is further configured to: perform field integrity detection on the first data object based on the event sequence number and timestamp of the first data object; when a missing field is detected, retrieve compensation information in the message topic data based on the event sequence number, and merge the compensation information with the first data object to generate a compensated third data object; perform time format conversion on the time field of the third data object to generate a time window label, write the time window label into the third data object to obtain intermediate data; perform encoding replacement on the device field of the intermediate data to generate a standardized device field, and merge the standardized device field into the intermediate data to obtain extended data; correct the numerical fields in the extended data that are out of range according to predefined range rules to generate corrected numerical fields, and update the corrected numerical fields to the extended data to obtain a cleaned data object.

[0086] In an optional implementation of this embodiment, the processing module is further configured to: generate an idempotent key based on the user identifier, event identifier, event occurrence time, and original event sequence number in the enriched data object; determine the existence of the idempotent key in the distributed idempotent index by performing a distributed check on the idempotent key; if the idempotent key already exists, record the enriched data object as a duplicate message and write it to the duplicate record table; if the idempotent key does not exist, insert the idempotent key into the distributed idempotent index and determine the enriched data object as the second data object.

[0087] In an optional implementation of this embodiment, the sending module is specifically used to: obtain a user profile tag corresponding to the user identifier in the second data object, and use the user profile tag as a first routing condition; generate a combined tag based on the event type and event occurrence time of the second data object, and use the combined tag as a second routing condition; generate a load weight value based on the current load status of the downstream device, and use the load weight value as a third routing condition; and determine the target downstream device of the second data object by performing weighted matching on the first routing condition, the second routing condition, and the third routing condition.

[0088] According to the data processing device based on message middleware provided in this application, the following steps are taken: First, a data object with an event sequence number is obtained based on the data collection request from the business party. Then, the data object is written into the corresponding message topic of the message middleware according to the event sequence number and event type, resulting in business-isolated message queue data. Next, a set of preset consumer instances are used to retrieve target messages from the message queue data, and the target messages are parsed according to the event sequence number to generate a first data object. Finally, the first data object undergoes triple data processing to generate a second data object. Finally, the second data object is sent to downstream devices according to preset routing rules. This application constructs an end-to-end data processing link, encompassing data collection, message middleware writing, distributed consumption, triple data processing, and routing rule-based distribution, effectively improving data processing efficiency and business decoupling capabilities.

[0089] According to the scheme provided in this application Figure 3 An electronic device is provided as an embodiment of this application. This electronic device can be used to implement the message middleware-based data processing method described in the foregoing embodiments, and mainly includes:

[0090] The system includes a memory 301, a processor 302, and a computer program 303 stored on the memory 301 and executable on the processor 302. The memory 301 and the processor 302 are connected via communication. When the processor 302 executes the computer program 303, it implements the message middleware-based data processing method described in the foregoing embodiments. The number of processors can be one or more.

[0091] The memory 301 can be a high-speed random access memory (RAM) or a non-volatile memory, such as a disk storage device. The memory 301 is used to store executable program code, and the processor 302 is coupled to the memory 301.

[0092] Furthermore, embodiments of this application also provide a computer-readable storage medium, which may be disposed in the electronic device described in the above embodiments, and the computer-readable storage medium may be as described above. Figure 3 The memory in the illustrated embodiment.

[0093] The computer-readable storage medium stores a computer program that, when executed by a processor, implements the message middleware-based data processing method described in the foregoing embodiments. Furthermore, the computer-readable storage medium can also be a USB flash drive, external hard drive, read-only memory (ROM), RAM, magnetic disk, or optical disk, or any other medium capable of storing program code.

[0094] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0095] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0096] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A data processing method based on message middleware, characterized in that, include: Obtain the collected data object with the event sequence number according to the data collection request from the business party; The collected data object is written into the corresponding message topic of the message middleware according to the event sequence number and event type to obtain business-isolated message queue data; The target message is obtained from the message queue data by a number of preset consumer instances, and the target message is parsed according to the event sequence number to generate a first data object; The first data object undergoes triple data processing to generate the second data object; The second data object is sent to the downstream device according to the preset routing rules; The step of writing the collected data object into the corresponding message topic of the message middleware according to the event sequence number and event type to obtain the business-isolated message queue data includes: Based on the event type field of the collected data object, the collected data object is classified and identified, and a topic identifier corresponding to the event type is generated. By combining the event sequence number with the topic identification information, a message key value for writing to the message middleware is determined, and the target partition corresponding to the collected data object is determined based on the message key value; The collected data object is written into the corresponding message topic in the message middleware based on the topic identifier, and the message key value is written into the target partition as the partitioning basis to generate business-isolated message queue data; wherein, the message topic includes several target partitions.

2. The data processing method based on message middleware according to claim 1, characterized in that, The step of obtaining the target message from the message queue data through a preset number of consumer instances, parsing the target message according to the event sequence number, and generating a first data object includes: The target message is obtained from the message queue data by using a number of preset consumer instances; The order of the target message is checked according to the event sequence number corresponding to the target message, and the sorting result of the order check is stored in correspondence with the target message to generate a message set to be parsed; The target message in the message set to be parsed is deserialized, and parsed data is generated according to the field definitions in the message body; The parsed data is associated with the event sequence number, and a first data object with a unified field format is generated in the memory buffer.

3. The data processing method based on message middleware according to claim 1, characterized in that, The step of performing triple data processing on the first data object to generate the second data object includes: The first data object is filled in with fields and anomalies are corrected to generate a cleaned data object; Based on the user identifier of the cleaned data object, user profile attributes are obtained from an external data source, and the user profile attributes are merged with the cleaned data object to generate an enriched data object. An idempotent key is generated based on the user identifier, event identifier, and time information in the enriched data object, and a second data object is obtained by performing a repeatability check on the idempotent key.

4. The data processing method based on message middleware according to claim 3, characterized in that, The step of performing field completion and anomaly correction on the first data object to generate a cleaned data object includes: Perform field integrity checks on the first data object based on the event sequence number and timestamp of the first data object; When a missing field is detected, compensation information is retrieved from the message topic data based on the event sequence number, and the compensation information is merged with the first data object to generate a compensated third data object. The time field of the third data object is converted to a time format to generate a time window label, and the time window label is written into the third data object to obtain intermediate data; The device field of the intermediate data is encoded and replaced to generate a standardized device field, and the standardized device field is merged into the intermediate data to obtain extended data; According to predefined range rules, the numerical fields in the extended data that are out of range are corrected to generate corrected numerical fields, and the corrected numerical fields are updated to the extended data to obtain a cleaned data object.

5. The data processing method based on message middleware according to claim 3, characterized in that, The step of generating an idempotent key based on the user identifier, event identifier, and time information in the enriched data object, and obtaining a second data object by performing a repeatability check on the idempotent key, includes: Based on the user identifier, event identifier, event occurrence time, and original event sequence number in the enriched data object, an idempotent key is generated; The existence of the idempotent key is determined in the distributed idempotent index by performing a distributed check on the idempotent key. If the idempotent key already exists, the enriched data object is recorded as a duplicate message and written to the duplicate record table. If the idempotent key does not exist, the idempotent key is inserted into the distributed idempotent key to determine the enriched data object as the second data object.

6. The data processing method based on message middleware according to claim 1, characterized in that, The step of sending the second data object to the downstream device according to the preset routing rules includes: Based on the user identifier in the second data object, obtain the user profile tag corresponding to the user identifier, and use the user profile tag as the first routing condition; A combined label is generated based on the event type and event occurrence time of the second data object, and the combined label is used as the second routing condition; A load weight value is generated based on the current load status of the downstream devices, and the load weight value is used as a third routing condition; The target downstream device of the second data object is determined by weighted matching of the first routing condition, the second routing condition, and the third routing condition.

7. A data processing device based on message middleware, characterized in that, The message middleware-based data processing device is used to implement the message middleware-based data processing method of claim 1, wherein the message middleware-based data processing device comprises: The acquisition module is used to obtain the collected data object with the event sequence number according to the data collection request from the business party; The writing module is used to write the collected data object into the corresponding message topic of the message middleware according to the event sequence number and event type, so as to obtain the business-isolated message queue data. The parsing module is used to obtain the target message from the message queue data through a number of preset consumer instances, and parse the target message according to the event sequence number to generate a first data object; The processing module is used to perform triple data processing on the first data object to generate the second data object; The sending module is used to send the second data object to the downstream device according to the preset routing rules; The writing module is further configured to classify and identify the collected data object according to the event type field of the collected data object, and generate a topic identifier corresponding to the event type; By combining the event sequence number with the topic identification information, a message key value for writing to the message middleware is determined, and the target partition corresponding to the collected data object is determined based on the message key value; The collected data object is written into the corresponding message topic in the message middleware based on the topic identifier, and the message key value is written into the target partition as the partitioning basis to generate business-isolated message queue data; wherein, the message topic includes several target partitions.

8. An electronic device, characterized in that, Includes memory and processor, of which: The processor is used to execute computer programs stored in the memory; When the processor executes the computer program, it implements the steps in the data processing method based on message middleware as described in any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the data processing method based on message middleware as described in any one of claims 1 to 6.

Images