A message data processing method and device, a storage medium and an electronic device

Abstract

The application provides a message data processing method and device, a storage medium and electronic equipment, and applies to the technical field of data processing. In the application, a certain number of target service message data are acquired in a specified message queue, and the data contain operation types, message identifiers, message time stamps, grouping attribute information and deduplication unique identifiers and the like. According to the operation types, the data are classified into a deduplication type and a merging type. The deduplication type message is divided into groups according to the grouping attribute, and is deduplicated according to the deduplication unique identifier, and the message with the earliest time stamp is retained. The merging type message is hedged and deduplicated according to the operation type. The result message set is sorted according to the time stamp and is pushed to a service queue. The application adopts the deduplication algorithm idea and the time stamp order preservation processing, realizes the deduplication and merging of the message data, and simultaneously ensures that the service execution and the dependency relationship remain unchanged, and further helps to improve the downstream service processing efficiency.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a message data processing method, apparatus, storage medium, and electronic device. Background Technology

[0002] With the rapid development of internet services, various demands are constantly emerging, especially in scenarios involving numerous changes to subscription services. For example, synchronizing video content from third-party platforms to a proprietary platform, or monitoring changes in a specific service to handle related dependent services. In these scenarios, upstream services typically proactively send messages via message queues (MQ) or monitor the binlog (MySQL's binary log) of database tables to capture changes. Downstream services then adjust their logic based on these change messages. However, these change messages may be duplicated extensively within a certain timeframe, causing downstream services to process the same data multiple times, impacting efficiency.

[0003] Previous solutions mainly consisted of two approaches: one was to improve efficiency by optimizing downstream business code to reduce the processing time of a single message, but this approach faced performance optimization bottlenecks; the other was to increase the number of instances and threads of downstream business services to improve processing performance, but this approach would increase service costs, and repeated processing was not only inefficient, but could also prolong message processing time due to locking operations on business IDs.

[0004] Therefore, effectively handling duplicate message data is key to improving the efficiency of downstream business processing. Summary of the Invention

[0005] In view of the above problems, the present invention provides a message data processing method, apparatus, storage medium, and electronic device that overcomes or at least partially solves the above problems. The technical solution is as follows:

[0006] A message data processing method, comprising:

[0007] A preset number of target business message data are retrieved from a designated message queue at preset time intervals. The target business message data includes message data attribute information, which includes the operation type, message identifier, message timestamp, grouping attribute information, and deduplication unique identifier of the target business message data.

[0008] Based on the operation type of the target business message data, the message attribute category of the target business message data is determined, and each target business message data with the message attribute category of deduplication is assigned to a deduplication message group, and each target business message data with the message attribute category of merging is assigned to a merging message group.

[0009] In the deduplication message group, the target business message data with the same grouping attribute information are divided into the same deduplication group. Each target business message data with the same deduplication unique identifier in each deduplication group is deduplicated. The target business message data with the earliest message timestamp is retained to obtain the message deduplication result.

[0010] In the merged message group, target business message data with the same grouping attribute information are divided into the same merge group. According to the positive and negative offsetting relationship of the operation type, each target business message data in each merge group is divided into the corresponding merge type group. Adjacent target business message data with the same deduplication unique identifier in each merge type group are deduplicated. After deduplication, offsetting processing is performed according to the positive and negative relationship of the operation type of each target business message data in the merge type group to obtain the message merging result.

[0011] The message deduplication result and the message merging result are combined into a target message set;

[0012] Each target business message in the target message set is sorted in ascending order according to the message timestamp, and then pushed to the target business processing queue in the sorted order.

[0013] Optionally, the operation types include adding, modifying, approving, going online, going offline, deleting, and restoring.

[0014] Optionally, determining the message attribute category of the target business message data based on the operation type of the target business message data includes:

[0015] When the operation type of the target business message data is a first preset operation, the message attribute category of the target business message data is determined to be a deduplication category, wherein the first preset operation includes adding, modifying, and approving.

[0016] When the operation type of the target business message data is a second preset operation, the message attribute category of the target business message data is determined to be a merge type, wherein the second preset operation includes going online, going offline, deleting, and restoring.

[0017] Optionally, the step of dividing each target business message data in each of the merging groups into corresponding merging type groups according to the positive and negative hedging relationship of the operation type includes:

[0018] The target business message data whose operation type is online or offline in the same merge group are divided into the first merge type group;

[0019] The target business message data whose operation type is deletion or restoration in the same merge group are divided into the second merge type group.

[0020] Optionally, when the operation type of the target business message data is online or restore, a positive identifier is added to the target business message data; when the operation type of the target business message data is offline or delete, a negative identifier is added to the target business message data. The deduplication process is then offset based on the positive or negative nature of the operation type of each target business message data in the merging type group, including:

[0021] Calculate the offsetting value of the merged type group by using the positive or negative identifier carried by each target business message data in the deduplicated merged type group.

[0022] When the hedging value is 1, the target business message data retained by the merge type group is of the positive operation type;

[0023] When the hedging value is -1, the target business message data retained by the merge type group is of the reverse operation type;

[0024] When the hedging value is 0, the merge type group is empty.

[0025] Optionally, the message data attribute information further includes the service type and service ID of the target service message data. Before retrieving a preset number of target service message data from the designated message queue at preset time intervals, the method further includes:

[0026] Listen to the original business message data pushed by the upstream business producer at the current moment;

[0027] Parse the original business message data to obtain the business type, business ID, operation type, and message identifier;

[0028] Assign the current time to the message timestamp of the original business message data;

[0029] Using the business ID and operation type of the original business message data, a unique identifier for deduplication of the original business message data is generated;

[0030] Using the business type and business ID of the original business message data, a cached unique identifier for the original business message data is generated;

[0031] If the cached unique identifier exists in the specified memory database, and if it does, the cached unique identifier is set as the grouping attribute information of the original business message data to obtain the target business message data. If not, the cached unique identifier is stored in the specified memory database, and the step of setting the cached unique identifier as the grouping attribute information of the original business message data to obtain the target business message data is executed.

[0032] Optionally, generating a unique identifier for the original business message data by utilizing the business ID and operation type of the original business message data includes:

[0033] Concatenate the business ID and the attribute name and attribute value of the operation type of the target business message data to obtain the business attribute string;

[0034] Calculate the hash value of the business attribute string, and confirm the hash value as the unique identifier for deduplication of the target business message data.

[0035] A message data processing device includes: a target business message data acquisition unit, a message attribute classification unit, a message deduplication result acquisition unit, a message merging result acquisition unit, a target message set acquisition unit, and a message push unit.

[0036] The target business message data acquisition unit is used to acquire a preset number of target business message data from a specified message queue at preset time intervals. The target business message data includes message data attribute information, which includes the operation type, message identifier, message timestamp, grouping attribute information, and deduplication unique identifier of the target business message data.

[0037] The message attribute category division unit is used to determine the message attribute category of the target business message data based on the operation type of the target business message data, divide each target business message data with the message attribute category of deduplication into a deduplication message group, and divide each target business message data with the message attribute category of merging into a merging message group.

[0038] The message deduplication result obtaining unit is used to divide the target business message data with the same grouping attribute information into the same deduplication group in the deduplication class message group, and to deduplicate each of the target business message data with the same deduplication unique identifier in each of the deduplication groups, retaining the target business message data with the earliest message timestamp, and obtaining the message deduplication result.

[0039] The message merging result obtaining unit is used to divide the target business message data with the same grouping attribute information into the same merging group in the merged message group, divide each target business message data in each merging group into the corresponding merging type group according to the positive and negative offset relationship of the operation type, deduplicate adjacent target business message data with the same deduplication unique identifier in each merging type group, and perform offset processing according to the positive and negative operation type of each target business message data in the merging type group after deduplication to obtain the message merging result;

[0040] The target message set obtaining unit is used to combine the message deduplication result and the message merging result into a target message set;

[0041] The message push unit is used to sort each target business message data in the target message set in ascending order according to the message timestamp, and push each target business message data in the target message set to the target business processing queue in the sorted order.

[0042] A computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the message data processing method described in any of the preceding claims.

[0043] An electronic device includes at least one processor, at least one memory connected to the processor, and a bus; wherein the processor and the memory communicate with each other via the bus; the processor is used to call program instructions in the memory to execute the message data processing method described above.

[0044] By employing the above technical solution, the present invention provides a message data processing method, apparatus, storage medium, and electronic device. The method retrieves a preset number of target business message data from a designated message queue at preset time intervals. The target business message data includes message data attribute information, which includes the operation type, message identifier, message timestamp, grouping attribute information, and a unique deduplication identifier. Based on the operation type of the target business message data, the message attribute category of the target business message data is determined. Target business message data with the message attribute category of deduplication are grouped into deduplication message groups, and target business message data with the message attribute category of merging are grouped into merging message groups. Within the deduplication message groups, target business message data with the same grouping attribute information are grouped into the same deduplication subgroup. The method then processes each target business message with the same unique deduplication identifier within each deduplication subgroup. The message data is deduplicated, retaining the target business message data with the earliest timestamp, to obtain the message deduplication result. In merged message groups, target business message data with the same grouping attribute information are grouped into the same merge group. Based on the positive and negative offsetting relationships of operation types, each target business message data in each merge group is further divided into its corresponding merge type group. Adjacent target business message data with the same unique deduplication identifier within each merge type group are deduplicated. After deduplication, offsetting processing is performed based on the positive and negative relationships of the operation types of each target business message data in the merge type group, to obtain the message merging result. The message deduplication result and the message merging result are combined into a target message set. Each target business message data in the target message set is sorted in ascending order by message timestamp, and then pushed to the target business processing queue in the sorted order. This invention uses a digest algorithm for deduplication, utilizes a delayed task queue, and uses timestamp-preserving processing to achieve message data deduplication and merging, thereby filtering out valid message content while ensuring that business execution and dependencies remain unchanged, thus helping to improve the efficiency of downstream business processing.

[0045] The above description is merely an overview of the technical solution of the present invention. In order to better understand the technical means of the present invention and to implement it in accordance with the contents of the specification, and in order to make the above and other objects, features and advantages of the present invention more apparent and understandable, specific embodiments of the present invention are described below. Attached Figure Description

[0046] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0047] Figure 1 This diagram illustrates an architecture diagram for message data deduplication and merging processing provided by an embodiment of the present invention.

[0048] Figure 2 A flowchart illustrating one embodiment of the message data processing method provided by this invention is shown.

[0049] Figure 3 An example diagram of message data attribute information provided by an embodiment of the present invention is shown;

[0050] Figure 4 An example diagram of message data deduplication and merging provided in an embodiment of the present invention is shown;

[0051] Figure 5 A flowchart illustrating another embodiment of the message data processing method provided in this invention is shown.

[0052] Figure 6 A schematic diagram of the structure of the message data processing apparatus provided in an embodiment of the present invention is shown;

[0053] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present invention is shown. Detailed Implementation

[0054] Exemplary embodiments of the invention will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the invention are shown in the drawings, it should be understood that the invention may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this invention will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art.

[0055] With the rapid development of internet services, various demands are constantly emerging, especially in scenarios involving large-scale subscription changes, such as synchronizing video content or monitoring business changes. Upstream businesses typically use message queues (MQ) or listen to database binlogs (MySQL's binary log) to capture changes, and downstream businesses adjust their logic based on these messages. However, these messages may be repeated in large quantities within a short period of time, causing downstream businesses to repeatedly process the same data, impacting efficiency. Traditional solutions include optimizing downstream business code to reduce processing time, but this faces performance bottlenecks; or increasing the number of service instances and threads, but this increases costs and extends processing time due to business ID locking operations.

[0056] Therefore, a seamless intermediate processing layer needs to be introduced between message producers and consumers to ensure that the processed message result is consistent with the original result and does not affect business processes. Furthermore, the dependencies and ordering of messages must be addressed. This message data deduplication and merging process differs fundamentally from traditional deduplication of raw data sets, which does not require consideration of complex dependencies between message objects and data. Solving these problems is crucial for improving downstream business processing efficiency and reducing costs.

[0057] like Figure 1 The diagram illustrates an architecture for message data deduplication and merging processing according to an embodiment of the present invention. To address the message processing efficiency issue between upstream and downstream businesses, this invention introduces an intermediate processing module—a modified message consumer—to acquire and encapsulate message content, push it into a delayed queue, group, deduplicate, and merge messages, and sort and push the merged messages to downstream businesses. This module is transparent to both upstream and downstream businesses and does not involve modifying their code. To achieve efficient message processing, a digest algorithm is used for deduplication, utilizing a Redis delayed task queue and a timestamp-preserving method to ensure that only valid message content is transmitted without affecting business execution and dependencies. Processed messages are pushed sequentially to downstream businesses, significantly improving synchronous processing efficiency. This method differs fundamentally from traditional methods that increase the number of downstream business instances and consumer threads, or optimize code logic. Traditional methods are prone to processing delays when distributed locks are required and message backlogs are likely to occur during peak periods. The solution provided by this patent fundamentally solves these problems and improves overall processing efficiency.

[0058] like Figure 2 The diagram shows a flowchart of one embodiment of the message data processing method provided by this invention. The method includes:

[0059] S100. Obtain a preset number of target business message data from a designated message queue at preset time intervals. The target business message data includes message data attribute information, which includes the operation type, message identifier, message timestamp, grouping attribute information, and deduplication unique identifier of the target business message data.

[0060] In this context, a message queue refers to a specific queue configured in a message middleware environment for temporary storage and transmission of message data. In this embodiment of the invention, the specified message queue can be an RQueue queue, which refers to a message queue pattern implemented using Redis, utilizing Redis's list structure to implement the queue.

[0061] Target business message data refers to message data pushed by upstream producers, which typically contains business-related information such as operation instructions and status updates. Target business message data is generally a string in JSON or XML format.

[0062] Among them, message data attribute information is additional information or metadata related to the data content carried by the target business message data itself. For example... Figure 3 The diagram shown is an example of message data attribute information provided in an embodiment of the present invention, which may include the business type, business ID, operation type, message identifier, message timestamp, grouping attribute information, and deduplication unique identifier of the target business message data.

[0063] The operation type specifies the specific operation involved in the target business message data.

[0064] Optionally, operation types can include adding, modifying, approving, going live, going offline, deleting, and restoring. Taking videos as an example, going live and going offline are similar to how online users publish and cancel publishing videos.

[0065] The message identifier is used to uniquely identify a target business message data. It can be a unique string or number. Unlike the business ID, the message identifier focuses on the uniqueness of the target business message data itself during the transmission process.

[0066] The message timestamp, also known as the message reception timestamp, is the system timestamp for receiving the target business message data, used to maintain the time sequence or perform time series analysis.

[0067] Among them, the grouping attribute information (group ID) is used to group multiple messages together, typically for batch processing or merging operations. For example, all target business message data with the same group ID can be grouped into the same group. The grouping attribute information carried by the target business message data is related to its business type and business ID.

[0068] The deduplication key is a unique identifier used for message deduplication. The deduplication key carried by the target business message data is related to its business ID and operation type.

[0069] Specifically, in this embodiment of the invention, threads can be started from a thread pool to pull a preset number of target business message data from the RQueue queue at preset time intervals. The preset time interval and preset number can be set according to specific business needs and traffic conditions to avoid message data processing delays or poor deduplication and merging effects.

[0070] S110. Based on the operation type of the target business message data, determine the message attribute category of the target business message data, and divide each target business message data with the message attribute category of deduplication into the deduplication message group, and divide each target business message data with the message attribute type of merging into the merging message group.

[0071] Message attribute categories refer to the classification of message attributes, which help the processing system understand how to process or route messages. Message attribute categories include deduplication categories and merging categories.

[0072] Specifically, in this embodiment of the invention, deduplication message groups and merging message groups can be pre-created. Based on the operation type of the acquired target business message data, target business message data with operation types that do not have a positive-negative offset relationship are assigned to the deduplication message group, and target business message data with operation types that have a positive-negative offset relationship are assigned to the merging message group.

[0073] S120. In the deduplication message group, target business message data with the same grouping attribute information are divided into the same deduplication group. Each target business message data with the same unique deduplication identifier in each deduplication group is deduplicated, and the target business message data with the earliest message timestamp is retained to obtain the message deduplication result.

[0074] Each target business message in the deduplication message group is a deduplicatable message. The grouping attribute information includes a group ID generated based on the business type and business ID of the target business message data. If the group IDs are the same, it means that the target business message data is operating on the same business object, and these target business message data can be divided into the same deduplication group.

[0075] For target business message data within the same deduplication group, although these target business message data operate on the same business object, the specific operation types involved may differ. For example, in deduplication group A, there are target business message data a and target business message data b. Target business message data a performs a new operation on business B, while target business message data b performs a modification operation on business B. Therefore, this embodiment of the invention can use the unique deduplication identifier related to the business ID and operation type of the target business message data as the deduplication basis. If the unique deduplication identifiers are the same, it indicates that these target business message data are deduplicatable messages with repeated operations. In the same deduplication group, multiple target business message data with the same unique deduplication identifier are deduplicated, retaining the target business message data with the earliest message timestamp and discarding the other target business message data. After each deduplication group has completed the aforementioned deduplication operation, the deduplication results of each deduplication group are integrated to obtain the message deduplication result.

[0076] It is important to note that the deduplication basis here is the unique identifier related to the business ID and operation type of the target business message data. This is because each target business message data has a unique message identifier. If the overall message body content of the target business message data is used as the deduplication basis, the program will identify each target business message data as a different message body. After deduplication, each target business message data will be retained, and the deduplication effect cannot be achieved.

[0077] S130. In the merged message group, target business message data with the same grouping attribute information are divided into the same merge group. According to the positive and negative offsetting relationship of the operation type, each target business message data in each merge group is divided into the corresponding merge type group. Adjacent target business message data with the same deduplication unique identifier in each merge type group are deduplicated. After deduplication, offsetting processing is performed according to the positive and negative operation type of each target business message data in the merge type group to obtain the message merging result.

[0078] In this context, positive and negative offsetting relationships refer to offsetting or merging certain operations based on their relative relationships when processing business messages, in order to simplify the result or achieve the desired final state. Positive and negative offsetting relationships appear in operations that need to be merged or offset each other, such as "add-remove", "join-remove", and "enable-disable".

[0079] Each target business message in the merged message group is a mergeable message. The grouping attribute information includes a group ID generated based on the business type and business ID of the target business message data. If the group IDs are the same, it means that the target business message data is operating on the same business object, and these target business message data can be divided into the same merge group.

[0080] For target business message data within the same merging group, although these target business message data operate on the same business objects, the positive and negative hedging relationships of the specific operation types involved may be different. For example, in merging group C, there are target business message data c, target business message data d, target business message data e, and target business message data h. Target business message data c performs an online operation on business D, target business message data d performs an offline operation on business D, target business message data e performs a deletion operation on business D, and target business message data h performs a restore operation on business D. Here, the operation types of target business message data c and target business message data d have a positive and negative hedging relationship, and the operation types of target business message data e and target business message data h also have a positive and negative hedging relationship.

[0081] This invention allows for the creation of a merge type group within the same merge group based on an online-offline offsetting relationship, and another merge type group based on a delete-restore offsetting relationship. The target business message data within the same merge group are then assigned to the corresponding merge type groups. For the same merge type group, since the original business may require idempotency support, two target business message data points with adjacent timestamps within the merge type group may contain duplicate forward or reverse messages. Therefore, deduplication is necessary between adjacent target business message data. Deduplication is still based on a unique deduplication identifier. If at least two target business message data points with adjacent timestamps within the same merge type group have the same unique deduplication identifier, then these target business message data points are deduplicated, retaining the target business message data with the earliest timestamp. After deduplication, the merge (offset) is performed based on the forward / reverse nature of the operation type of each target business message data point within the merge type group. For example, an online operation offsets an offline operation, and a delete operation offsets a restore operation. The remaining business operation messages after merging are considered the merge result for that merge type group. After each merging group completes the aforementioned deduplication and merging operations, the merging results of each merging type group are integrated to obtain the message merging result.

[0082] S140. Combine the message deduplication result and the message merging result into a target message set.

[0083] like Figure 4 The diagram shown is an example of message data deduplication and merging provided in an embodiment of the present invention. By deduplicating and merging message data, the amount of data processing in downstream services can be effectively reduced, thereby improving the processing efficiency of downstream services.

[0084] S150. Sort each target business message data in the target message set in ascending order according to the message timestamp, and push each target business message data in the target message set to the target business processing queue in the sorted order.

[0085] The target business processing queue is the message queue to be processed by downstream consumers. After receiving the target business message data from the target business processing queue, the downstream consumer parses the target business message data to extract the business type, business ID, and operation type. Then, based on the business ID, it calls a third-party API interface to obtain complete business information, which is then written to the downstream business database or processed by other business logic as needed.

[0086] This invention provides a message data processing method, comprising: retrieving a preset number of target business message data from a designated message queue at preset time intervals, wherein the target business message data includes message data attribute information, including operation type, message identifier, message timestamp, grouping attribute information, and deduplication unique identifier; determining the message attribute category of the target business message data based on the operation type, dividing each target business message data with the message attribute category of deduplication into a deduplication message group, and dividing each target business message data with the message attribute category of merging into a merging message group; within the deduplication message group, dividing target business message data with the same grouping attribute information into the same deduplication subgroup, and performing deduplication on each target business message data with the same deduplication unique identifier in each deduplication subgroup. The invention employs a digest algorithm for deduplication, utilizes a delayed task queue, and uses timestamp-preserving processing to achieve message deduplication and merging, thereby filtering out valid message content while ensuring that business execution and dependencies remain unchanged, thus improving the efficiency of downstream business processing. Within merged message groups, target business message data with the same grouping attribute information are grouped into the same merge group. Based on the positive and negative offsetting relationships of operation types, each target business message in each merge group is further divided into corresponding merge type groups. Adjacent target business message data with the same unique deduplication identifier within each merge type group are deduplicated. After deduplication, offsetting processing is performed based on the positive and negative relationships of the operation types of each target business message in the merge type group to obtain the message merging result. The message deduplication result and the message merging result are combined into a target message set. Each target business message in the target message set is sorted in ascending order according to its message timestamp, and then pushed to the target business processing queue in the sorted order.

[0087] Optionally, in the above Figure 2 Based on one or more corresponding embodiments, the message attribute category of the target business message data is determined according to the operation type of the target business message data, which may specifically include:

[0088] When the operation type of the target business message data is the first preset operation, the message attribute category of the target business message data is determined to be the deduplication category. The first preset operation includes adding, modifying, and approving. When the operation type of the target business message data is the second preset operation, the message attribute category of the target business message data is determined to be the merge category. The second preset operation includes going online, going offline, deleting, and restoring.

[0089] Specifically, in this embodiment of the invention, the message attribute category of target business message data with operation types of adding, modifying, and approving can be determined as a deduplication category. This means that when processing these operations, special attention needs to be paid to repetition to ensure that the same operation is not repeatedly executed or the same data is repeatedly inserted in downstream processing.

[0090] Specifically, in this embodiment of the invention, the message attribute category of target business message data with operation types including online, offline, deletion, and restoration can be determined as a merging category. This indicates that when processing these operations, it may be necessary to merge multiple related messages or operations to ensure the consistency of the final state. For example, rapid online and offline operations may need to be processed in a merged manner to prevent unnecessary fluctuations in the system.

[0091] Through clear classification and processing strategies, the embodiments of the present invention enable the system to better handle concurrent operations, reduce potential problems caused by repetitive or conflicting operations, improve overall reliability and stability, and allow users to experience faster and more consistent system responses. In particular, the system can provide a smoother experience when operations involving approval and status changes are involved.

[0092] Optionally, in the above Figure 2 Based on one or more corresponding embodiments, according to the positive and negative hedging relationship of the operation type, the target business message data in each merging group are divided into the corresponding merging type group, including:

[0093] Target business message data with operation type of "online" or "offline" in the same merge group are divided into the first merge type group.

[0094] Within the merge team, all operations involving going online or offline are grouped into the first merge type group. These operations typically involve state changes to resources or services and may require merge processing to ensure that, under multiple similar requests, there is ultimately only one unified state.

[0095] Target business message data with operation type of deletion or restoration in the same merge group are divided into the second merge type group.

[0096] Within the merge group, all operations involving deletion or restoration are categorized into the second merge type group. These operations involve the existence state of resources and therefore require special handling to avoid issues such as accidental deletion or duplicate restoration.

[0097] This invention effectively reduces conflicts between different operations by grouping them into different operation types. For example, the operations of going online and offline focus on state switching, while the operations of deleting and restoring focus on resource existence. Processing them separately reduces the complexity caused by different operation types. At the same time, through reasonable grouping and merging strategies, data consistency can be better maintained, ensuring that the final operation results accurately reflect business needs.

[0098] Optionally, in the above Figure 2 Based on one or more corresponding embodiments, when the operation type of the target business message data is online or restore, a positive identifier is added to the target business message data; when the operation type of the target business message data is offline or delete, a negative identifier is added to the target business message data. After deduplication, offsetting processing is performed according to the positive and negative nature of the operation type of each target business message data in the merging type group, which may specifically include:

[0099] Calculate the offset value of the merge type group by using the positive or negative identifier carried by each target business message data in the deduplicated merge type group; when the offset value is 1, the target business message data retained by the merge type group is of the positive operation type; when the offset value is -1, the target business message data retained by the merge type group is of the reverse operation type; when the offset value is 0, the merge type group is empty.

[0100] This invention simplifies complex decision-making processes by transforming message operations into hedging value calculations, enabling rapid determination of the type of operation to be executed. Simultaneously, through precise positive / negative labeling and hedging calculations, the positive / inverse nature of the operation is reflected, ensuring consistency between system state and business logic.

[0101] Optionally, the message data attribute information provided in this embodiment of the invention also includes the service type and service ID of the target service message data, based on... Figure 2 The method shown is as follows: Figure 5 As shown in the flowchart of another embodiment of the message data processing method provided by the present invention, before step S100, the method may further include:

[0102] S200: Monitor the raw business message data pushed by the upstream business producer at the current moment.

[0103] Specifically, in this embodiment of the invention, a thread can be started at the message consumption end as an intermediate intervention consumer to listen to the business message data pushed by the upstream business producer, so as to ensure that the business message data can be captured and processed in the first time.

[0104] S210. Parse out the business type, business ID, operation type, and message identifier of the original business message data.

[0105] The business type is an identifier for the business category to which the target business message data belongs, used to distinguish different types of businesses. The specific business type of the target business message data depends on the business of each business system itself. For example, in a media asset management system, it refers to: video, compilation, playlist, celebrity, etc.

[0106] The Business Identifier is an identifier used to uniquely identify a specific business entity; that is, a unique identifier ID corresponding to each business type. For example, when the business type is video, the Business Identifier represents the video ID. When the business type is a collection, the Business Identifier represents the collection ID, and so on.

[0107] S220. Assign the current time to the message timestamp of the original business message data.

[0108] In this embodiment of the invention, a message timestamp can be added to business message data to record its reception time, so as to perform time-related processing or sorting.

[0109] S230. Using the business ID and operation type of the original business message data, generate a unique identifier for deduplicating the original business message data.

[0110] In this embodiment of the invention, a unique identifier for deduplication is generated to ensure that duplicate messages of the same operation type are not processed repeatedly.

[0111] S240. Using the business type and business ID of the original business message data, generate a cached unique identifier for the original business message data.

[0112] The cache unique identifier is an identifier used to uniquely identify a data record in the caching system, i.e., a group ID. In this embodiment of the invention, the cache unique identifier of the original business message data can be obtained by concatenating the business type and business ID of the original business message data.

[0113] S250. Query the specified memory database to see if a cached unique identifier exists. If yes, proceed to step S260; otherwise, proceed to step S270.

[0114] S260. Set the cached unique identifier to the grouping attribute information of the original business message data to obtain the target business message data.

[0115] In this embodiment of the invention, a cached unique identifier is used as a grouping attribute for business message data to facilitate subsequent grouping operations.

[0116] S270. Store the cached unique identifier in the specified memory database and proceed to step S260.

[0117] The embodiments of the present invention set message timestamps, grouping attribute information and unique deduplication identifiers for business message data, which helps to quickly group, deduplicatize and merge message data in subsequent message data processing, thereby improving the efficiency of message data processing and downstream business processing.

[0118] Optionally, in the above Figure 5 Based on one or more corresponding embodiments, a unique identifier for deduplicating the original business message data is generated using the business ID and operation type of the original business message data. Specifically, this may include:

[0119] Concatenate the business ID and operation type attribute names and attribute values ​​of the target business message data to obtain the business attribute string; calculate the hash value of the business attribute string, and confirm the hash value as the unique identifier for deduplication of the target business message data.

[0120] This invention can sort the attribute names and values ​​of the target business message data (business ID and operation type) and concatenate them according to the sorting order to form an ordered string containing the attribute names and values ​​of the business ID and operation type. For example: based on Figure 2 The message data attribute information shown has a corresponding business attribute string of "businessld150779920optType1". Next, the business attribute string is hashed using the MD5 message digest algorithm to generate a 32-bit fixed-length hash value. Finally, the generated hash value is set as the unique identifier for deduplicating this business message data.

[0121] This invention generates a unique hash value as a deduplication identifier, which can quickly determine the duplication between different messages. In the processing of large amounts of data, it can quickly identify and eliminate duplicate business message data, improve the processing efficiency of the system, and at the same time avoid using the entire message body content of the business message data as the basis for deduplication, which would cause the problem of being unable to deduplicate.

[0122] Although the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous.

[0123] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0124] Corresponding to the above method embodiments, this invention also provides a message data processing device, the structure of which is as follows: Figure 6 As shown, it may include: a target business message data acquisition unit 10, a message attribute category classification unit 20, a message deduplication result acquisition unit 30, a message merging result acquisition unit 40, a target message set acquisition unit 50, and a message push unit 60.

[0125] The target business message data acquisition unit 10 is used to acquire a preset number of target business message data from a specified message queue at preset time intervals. The target business message data includes message data attribute information, which includes the operation type, message identifier, message timestamp, grouping attribute information, and deduplication unique identifier of the target business message data.

[0126] The message attribute category division unit 20 is used to determine the message attribute category of the target business message data based on the operation type of the target business message data, and to divide each target business message data with the message attribute category of deduplication into the deduplication message group, and to divide each target business message data with the message attribute type of merging into the merging message group.

[0127] The message deduplication result acquisition unit 30 is used to divide the target business message data with the same grouping attribute information into the same deduplication group in the deduplication class message group, and to deduplicate each target business message data with the same deduplication unique identifier in each deduplication group, retaining the target business message data with the earliest message timestamp, and obtaining the message deduplication result.

[0128] The message merging result acquisition unit 40 is used to divide target business message data with the same grouping attribute information into the same merging group in the merged message group, divide each target business message data in each merging group into the corresponding merging type group according to the positive and negative offsetting relationship of the operation type, deduplicate adjacent target business message data with the same deduplication unique identifier in each merging type group, and perform offsetting processing according to the positive and negative operation type of each target business message data in the merging type group to obtain the message merging result.

[0129] The target message set acquisition unit 50 is used to combine the message deduplication result and the message merging result into a target message set.

[0130] The message push unit 60 is used to sort the target business message data in the target message set in ascending order according to the message timestamp, and push the target business message data in the target message set to the target business processing queue in the sorted order.

[0131] Optional operation types include adding, modifying, approving, launching, delisting, deleting, and restoring.

[0132] Optionally, the message attribute classification unit 20 can be specifically used to determine the message attribute category of the target business message data as a deduplication category when the operation type of the target business message data is a first preset operation, wherein the first preset operation includes adding, modifying, and approving; and to determine the message attribute category of the target business message data as a merging category when the operation type of the target business message data is a second preset operation, wherein the second preset operation includes going online, going offline, deleting, and restoring.

[0133] Optionally, the message merging result obtaining unit 40 can be specifically used to divide target business message data with operation type of online or offline in the same merging group into the first merging type group; and to divide target business message data with operation type of deletion or restoration in the same merging group into the second merging type group.

[0134] Optionally, the message data processing apparatus may also include an identifier adding unit.

[0135] The identifier addition unit is used to add a positive identifier to the target business message data when the operation type of the target business message data is online or restore; and to add a negative identifier to the target business message data when the operation type of the target business message data is offline or delete.

[0136] The message merging result acquisition unit 40 can be used to calculate the offset value of the merging type group by utilizing the positive or negative identifier carried by each target business message data in the deduplicated merging type group; when the offset value is 1, the target business message data retained by the merging type group is of the positive operation type; when the offset value is -1, the target business message data retained by the merging type group is of the reverse operation type; when the offset value is 0, the merging type group is empty.

[0137] Optionally, the message data attribute information provided in this embodiment of the invention may also include the business type and business ID of the target business message data. The message data processing device may further include a business message data monitoring unit, a business message data parsing unit, a message timestamp assignment unit, a deduplication unique identifier generation unit, a cache unique identifier generation unit, a cache unique identifier query unit, a group attribute information setting unit, and a cache unique identifier storage unit.

[0138] The business message data monitoring unit is used to monitor the original business message data pushed by the upstream business producer at the current time before the target business message data acquisition unit 10 acquires a preset number of target business message data from the specified message queue at a preset time interval.

[0139] The business message data parsing unit is used to parse out the business type, business ID, operation type, and message identifier of the original business message data.

[0140] The message timestamp assignment unit is used to assign the current time to the message timestamp of the original business message data.

[0141] The deduplication unique identifier generation unit is used to generate a deduplication unique identifier for the original business message data using the business ID and operation type of the original business message data.

[0142] The cache unique identifier generation unit is used to generate a cache unique identifier for the original business message data using the business type and business ID of the original business message data.

[0143] The cached unique identifier query unit is used to query whether a cached unique identifier exists in the specified in-memory database. If it does, the group attribute information setting unit is triggered; otherwise, the cached unique identifier storage unit is triggered.

[0144] The grouping attribute information setting unit is used to set the cached unique identifier as the grouping attribute information of the original business message data to obtain the target business message data.

[0145] The cached unique identifier storage unit is used to store the cached unique identifier in a specified memory database and trigger the group attribute information setting unit.

[0146] Optionally, the deduplication unique identifier generation unit is specifically used to concatenate the business ID and operation type attribute name and attribute value of the target business message data to obtain the business attribute string; calculate the hash value of the business attribute string, and confirm the hash value as the deduplication unique identifier of the target business message data.

[0147] This invention provides a message data processing apparatus, which is used to: retrieve a preset number of target business message data from a designated message queue at preset time intervals, wherein the target business message data includes message data attribute information, including the operation type, message identifier, message timestamp, grouping attribute information, and deduplication unique identifier of the target business message data; determine the message attribute category of the target business message data according to the operation type of the target business message data, divide each target business message data with the message attribute category of deduplication into a deduplication message group, and divide each target business message data with the message attribute category of merging into a merging message group; in the deduplication message group, divide the target business message data with the same grouping attribute information into the same deduplication subgroup, and perform deduplication on each target business message data with the same deduplication unique identifier in each deduplication subgroup. The invention employs a digest algorithm for deduplication, utilizes a delayed task queue, and uses timestamp-preserving processing to achieve message deduplication and merging, thereby filtering out valid message content while ensuring that business execution and dependencies remain unchanged, thus improving the efficiency of downstream business processing. Within merged message groups, target business message data with the same grouping attribute information are grouped into the same merge group. Based on the positive and negative offsetting relationships of operation types, each target business message in each merge group is further divided into corresponding merge type groups. Adjacent target business message data with the same unique deduplication identifier within each merge type group are deduplicated. After deduplication, offsetting processing is performed based on the positive and negative relationships of the operation types of each target business message in the merge type group to obtain the message merging result. The message deduplication result and the message merging result are combined into a target message set. Each target business message in the target message set is sorted in ascending order according to its message timestamp, and then pushed to the target business processing queue in the sorted order.

[0148] Regarding the apparatus in the above embodiments, the specific manner in which each unit performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0149] The message data processing device includes a processor and a memory. The target business message data acquisition unit 10, message attribute classification unit 20, message deduplication result acquisition unit 30, message merging result acquisition unit 40, target message set acquisition unit 50, and message push unit 60 are all stored as program units in the memory. The processor executes the program units stored in the memory to realize the corresponding functions.

[0150] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured. By adjusting kernel parameters, a digest algorithm can be used for deduplication, and timestamp-based ordering can be employed to achieve message data deduplication and merging, while ensuring that business execution and dependencies remain unchanged, thereby improving the efficiency of downstream business processing.

[0151] This invention provides a computer-readable storage medium storing a program thereon, which, when executed by a processor, implements the message data processing method.

[0152] This invention provides a processor for running a program, wherein the program executes the message data processing method during runtime.

[0153] like Figure 7 As shown, this embodiment of the invention provides an electronic device 1000, which includes at least one processor 1001, at least one memory 1002 connected to the processor 1001, and a bus 1003. The processor 1001 and the memory 1002 communicate with each other via the bus 1003. The processor 1001 is used to call program instructions in the memory 1002 to execute the aforementioned message data processing method. The electronic device in this document can be a server, PC, PAD, mobile phone, etc.

[0154] The present invention also provides a computer program product that, when executed on an electronic device, is suitable for executing a program with initialization message data processing method steps.

[0155] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatuses, electronic devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0156] In a typical configuration, an electronic device includes one or more processors (CPUs), memory, and a bus. The electronic device may also include input / output interfaces, network interfaces, etc.

[0157] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, like read-only memory (ROM) or flash RAM, and memory includes at least one memory chip. Memory is an example of computer-readable media.

[0158] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0159] In the description of this invention, it should be understood that if the terms "upper", "lower", "front", "rear", "left" and "right" are used to indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the position or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this invention.

[0160] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0161] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0162] The above are merely embodiments of the present invention and are not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. A message data processing method, characterized in that, include: A preset number of target business message data are retrieved from a designated message queue at preset time intervals. The target business message data includes message data attribute information, which includes the operation type, message identifier, message timestamp, grouping attribute information, and deduplication unique identifier of the target business message data. Based on the operation type of the target business message data, the message attribute category of the target business message data is determined, and each target business message data with the message attribute category of deduplication is assigned to a deduplication message group, and each target business message data with the message attribute category of merging is assigned to a merging message group. In the deduplication message group, the target business message data with the same grouping attribute information are divided into the same deduplication group. Each target business message data with the same deduplication unique identifier in each deduplication group is deduplicated. The target business message data with the earliest message timestamp is retained to obtain the message deduplication result. In the merged message group, target business message data with the same grouping attribute information are divided into the same merge group. According to the positive and negative offsetting relationship of the operation type, each target business message data in each merge group is divided into the corresponding merge type group. Adjacent target business message data with the same deduplication unique identifier in each merge type group are deduplicated. After deduplication, offsetting processing is performed according to the positive and negative relationship of the operation type of each target business message data in the merge type group to obtain the message merging result. The message deduplication result and the message merging result are combined into a target message set; Each target business message in the target message set is sorted in ascending order according to the message timestamp, and then pushed to the target business processing queue in the sorted order.

2. The method according to claim 1, characterized in that, The operation types include adding, modifying, approving, going online, going offline, deleting, and restoring.

3. The method according to claim 2, characterized in that, Determining the message attribute category of the target business message data based on the operation type of the target business message data includes: When the operation type of the target business message data is a first preset operation, the message attribute category of the target business message data is determined to be a deduplication category, wherein the first preset operation includes adding, modifying, and approving. When the operation type of the target business message data is a second preset operation, the message attribute category of the target business message data is determined to be a merge type, wherein the second preset operation includes going online, going offline, deleting, and restoring.

4. The method according to claim 2, characterized in that, The step of dividing the target business message data in each of the merging groups into corresponding merging type groups according to the positive and negative hedging relationship of the operation type includes: The target business message data whose operation type is online or offline in the same merge group are divided into the first merge type group; The target business message data whose operation type is deletion or restoration in the same merge group are divided into the second merge type group.

5. The method according to claim 2, characterized in that, When the operation type of the target business message data is "online" or "restore", a positive identifier is added to the target business message data; when the operation type of the target business message data is "offline" or "delete", a negative identifier is added to the target business message data. After deduplication, offsetting processing is performed based on the positive or negative nature of the operation type of each target business message data in the merging type group, including: Calculate the offsetting value of the merged type group by using the positive or negative identifier carried by each target business message data in the deduplicated merged type group. When the hedging value is 1, the target business message data retained by the merge type group is of the positive operation type; When the hedging value is -1, the target business message data retained by the merge type group is of the reverse operation type; When the hedging value is 0, the merge type group is empty.

6. The method according to any one of claims 1 to 5, characterized in that, The message data attribute information also includes the service type and service ID of the target service message data. Before retrieving a preset number of target service message data from a designated message queue at preset time intervals, the method further includes: Listen to the original business message data pushed by the upstream business producer at the current moment; Parse the original business message data to obtain the business type, business ID, operation type, and message identifier; Assign the current time to the message timestamp of the original business message data; Using the business ID and operation type of the original business message data, a unique identifier for deduplication of the original business message data is generated; Using the business type and business ID of the original business message data, a cached unique identifier for the original business message data is generated; If the cached unique identifier exists in the specified memory database, and if it does, the cached unique identifier is set as the grouping attribute information of the original business message data to obtain the target business message data. If not, the cached unique identifier is stored in the specified memory database, and the step of setting the cached unique identifier as the grouping attribute information of the original business message data to obtain the target business message data is executed.

7. The method according to claim 6, characterized in that, The step of generating a unique identifier for the original business message data by utilizing the business ID and operation type of the original business message data includes: Concatenate the business ID and the attribute name and attribute value of the operation type of the target business message data to obtain the business attribute string; Calculate the hash value of the business attribute string, and confirm the hash value as the unique identifier for deduplication of the target business message data.

8. A message data processing device, characterized in that, include: The system includes a target business message data acquisition unit, a message attribute category classification unit, a merged message classification unit, a message deduplication result acquisition unit, a message merging result acquisition unit, a target message set acquisition unit, and a message push unit. The target business message data acquisition unit is used to acquire a preset number of target business message data from a specified message queue at preset time intervals. The target business message data includes message data attribute information, which includes the operation type, message identifier, message timestamp, grouping attribute information, and deduplication unique identifier of the target business message data. The message attribute category division unit is used to determine the message attribute category of the target business message data based on the operation type of the target business message data, divide each target business message data with the message attribute category of deduplication into a deduplication message group, and divide each target business message data with the message attribute category of merging into a merging message group. The message deduplication result obtaining unit is used to divide the target business message data with the same grouping attribute information into the same deduplication group in the deduplication class message group, and to deduplicate each of the target business message data with the same deduplication unique identifier in each of the deduplication groups, retaining the target business message data with the earliest message timestamp, and obtaining the message deduplication result. The message merging result obtaining unit is used to divide the target business message data with the same grouping attribute information into the same merging group in the merged message group, divide each target business message data in each merging group into the corresponding merging type group according to the positive and negative offset relationship of the operation type, deduplicate adjacent target business message data with the same deduplication unique identifier in each merging type group, and perform offset processing according to the positive and negative operation type of each target business message data in the merging type group after deduplication to obtain the message merging result; The target message set obtaining unit is used to combine the message deduplication result and the message merging result into a target message set; The message push unit is used to sort each target business message data in the target message set in ascending order according to the message timestamp, and push each target business message data in the target message set to the target business processing queue in the sorted order.

9. A computer-readable storage medium having a program stored thereon, characterized in that, When the program is executed by the processor, it implements the message data processing method as described in any one of claims 1 to 7.

10. An electronic device, the electronic device comprising at least one processor, and at least one memory and a bus connected to the processor; wherein, The processor and the memory communicate with each other via the bus; The processor is used to invoke program instructions in the memory to execute the message data processing method as described in any one of claims 1 to 7.

Images