Message data processing method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a message data processing method, comprising: in response to obtaining first message data in a first message queue, contending for a first distributed lock based on a primary key and state data of the first message data; determining whether the first message data has been consumed based on the primary key and the state data of the first message data; in a case where it is determined that the first message data has not been consumed, processing data in a target data wide table corresponding to the primary key of the first message data in a second database using the first message data; adding second message data generated based on the primary key of the first message data to a second message queue; and releasing the first distributed lock. The present disclosure also provides a message data processing apparatus, an electronic device, a readable storage medium and a computer program product.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and more particularly, to a message data processing method, a message data processing device, an electronic device, a readable storage medium, and a computer program product. Background Art

[0002] With the widespread application of distributed systems in system development, message queues are widely used in various distributed system projects to achieve decoupling and peak load reduction. However, message queues also introduce new challenges for distributed systems. For example, in data type products, data from multiple message queues must be assembled into a wide table for user queries based on various conditions.

[0003] In the process of implementing the concept of the present disclosure, the inventors found that there are at least the following problems in the related art: in the process of using data in multiple message queues to assemble a data wide table, the data in multiple message queues will compete for the same lock, thereby reducing data processing efficiency. Summary of the Invention

[0004] In view of this, the present disclosure provides a message data processing method, a message data processing device, an electronic device, a readable storage medium, and a computer program product.

[0005] One aspect of the present disclosure provides a message data processing method, including: in response to obtaining first message data in a first message queue, competing for a first distributed lock based on a primary key and status data of the first message data, wherein the first distributed lock is implemented based on a first database; based on the primary key and status data of the first message data, determining whether the first message data has been consumed; if it is determined that the first message data has not been consumed, using the first message data to process data in a target data wide table in a second database corresponding to the primary key of the first message data; adding second message data generated based on the primary key of the first message data to a second message queue; and releasing the first distributed lock.

[0006] According to an embodiment of the present disclosure, the above-mentioned determining whether the above-mentioned first message data has been consumed based on the primary key and status data of the above-mentioned first message data includes: obtaining multiple historical message data from the historical message set of the above-mentioned first database; based on the primary key and status data, comparing the above-mentioned first message data with the multiple historical message data in turn to obtain a first comparison result; if the above-mentioned first comparison result shows that the multiple historical message data do not match the above-mentioned first message data, determining that the above-mentioned first message data has not been consumed; if the above-mentioned first comparison result shows that there is message data matching the above-mentioned first message data among the multiple historical message data, determining that the above-mentioned first message data has been consumed.

[0007] According to an embodiment of the present disclosure, the above method also includes: when it is determined that the above first message data has not been consumed, storing the above first message data as historical message data in the historical message set of the above first database; and when it is determined that the above first message data has been consumed, releasing the above first distributed lock.

[0008] According to an embodiment of the present disclosure, the above-mentioned target data wide table has multiple columns, and each column has corresponding status data; wherein, the above-mentioned use of the above-mentioned first message data to process the data in the target data wide table corresponding to the primary key of the above-mentioned first message data in the second database includes: determining the target column corresponding to the status data of the above-mentioned first message data in the above-mentioned target data wide table; when there is existing data in the above-mentioned target column, using the status data of the above-mentioned first message data to replace the existing data in the above-mentioned target column; when there is no existing data in the above-mentioned column, storing the status data of the above-mentioned first message data in the above-mentioned target column.

[0009] According to an embodiment of the present disclosure, the above-mentioned method also includes: in response to obtaining the above-mentioned second message data in the above-mentioned second message queue, acquiring a second distributed lock based on the primary key competition of the above-mentioned second message data, wherein the above-mentioned second distributed lock is implemented based on the above-mentioned first database; calculating the current encoding value of the target data wide table in the above-mentioned second database; based on the above-mentioned current encoding value, determining whether the above-mentioned second message data has been consumed; in the case of determining that the above-mentioned second message data has not been consumed, setting the current version number of the above-mentioned target data wide table based on the current timestamp; storing the third message data generated based on the above-mentioned target data wide table in the third message queue; and releasing the second distributed lock of the above-mentioned second message data.

[0010] According to an embodiment of the present disclosure, the above-mentioned determination of whether the above-mentioned second message data has been consumed is based on the above-mentioned current coding value, and the determination of whether the above-mentioned second message data has been consumed includes: obtaining the historical coding value of the above-mentioned target data wide table from the above-mentioned first database; comparing the above-mentioned historical coding value with the above-mentioned current coding value to obtain a second comparison result; when the above-mentioned second comparison result shows that the above-mentioned historical coding value is different from the above-mentioned current coding value, determining that the above-mentioned second message data has not been consumed; and when the above-mentioned second comparison result shows that the above-mentioned historical coding value is the same as the above-mentioned current coding value, determining that the above-mentioned second message data has been consumed.

[0011] According to an embodiment of the present disclosure, the above method also includes: when it is determined that the above second message data has not been consumed, storing the above current coding value in the above first database to replace the above historical coding value with the above current coding value; and when it is determined that the above second message data has been consumed, releasing the above second distributed lock.

[0012] According to an embodiment of the present disclosure, the above-mentioned method also includes: in response to obtaining the above-mentioned third message data in the above-mentioned third message queue, acquiring a third distributed lock based on the primary key competition of the target data wide table in the above-mentioned third message data, wherein the above-mentioned third distributed lock is implemented based on the above-mentioned first database; based on the current version number of the target data wide table in the above-mentioned third message data, determining whether the above-mentioned third message data has been consumed; if it is determined that the above-mentioned third message data has not been consumed, storing the above-mentioned target data wide table in the third database; and releasing the above-mentioned third distributed lock.

[0013] According to an embodiment of the present disclosure, the above-mentioned determination of whether the above-mentioned third message data has been consumed based on the above-mentioned current version number includes: obtaining the historical version number of the above-mentioned target data wide table from the above-mentioned first database; comparing the above-mentioned historical version number with the above-mentioned current version number to obtain a third comparison result; in case the above-mentioned third comparison result shows that the above-mentioned current version number is greater than the above-mentioned historical version number, determining that the above-mentioned third message data has not been consumed; and in case the above-mentioned third comparison result shows that the above-mentioned current version number is less than or equal to the above-mentioned historical version number, determining that the above-mentioned third message data has been consumed.

[0014] According to an embodiment of the present disclosure, the method further includes: releasing the third distributed lock when it is determined that the third message data has been consumed.

[0015] Another aspect of the present disclosure provides a message data processing device, including a first locking module, a first judgment module, a first processing module, a second processing module and a first unlocking module. The first locking module is used to, in response to obtaining the first message data in the first message queue, compete for the first distributed lock based on the primary key and status data of the first message data, wherein the first distributed lock is implemented based on the first database; the first judgment module is used to determine whether the first message data has been consumed based on the primary key and status data of the first message data; the first processing module is used to use the first message data to process the data in the target data wide table corresponding to the primary key of the first message data in the second database when it is determined that the first message data has not been consumed; the second processing module is used to add the second message data generated based on the primary key of the first message data to the second message queue; and the first unlocking module is used to release the first distributed lock.

[0016] Another aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more instructions, wherein when the one or more instructions are executed by the one or more processors, the one or more processors implement the method described above.

[0017] Another aspect of the present disclosure provides a computer-readable storage medium storing computer-executable instructions, which are used to implement the method described above when executed.

[0018] Another aspect of the present disclosure provides a computer program product comprising computer executable instructions, which are used to implement the method described above when the instructions are executed.

[0019] According to an embodiment of the present disclosure, before acquiring first message data, a first distributed lock is competed for based on the primary key and status data of the first message data. After successfully acquiring the first distributed lock, if the first message data has not been consumed, the first message data is stored in a second database to generate a data wide table. Second message data is then generated based on the primary key of the first message data, and the first distributed lock is released. By utilizing a combination of primary keys and status data to acquire distributed locks, the technical problem of reduced data processing efficiency caused by data in multiple message queues competing for the same lock is at least partially overcome, thereby effectively improving the efficiency of message data processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0021] Figure 1An exemplary system architecture to which the message data processing method according to an embodiment of the present disclosure can be applied is schematically shown.

[0022] Figure 2 The flowchart of the message data processing method according to the embodiment of the present disclosure is schematically shown.

[0023] Figure 3 The figure schematically shows a message data processing method according to another embodiment of the present disclosure.

[0024] Figure 4A The figure is a flow chart of a method for producing a wide data table according to another embodiment of the present disclosure.

[0025] Figure 4B The figure is a flowchart of a data wide table version management method according to another embodiment of the present disclosure.

[0026] Figure 4C The figure is a flowchart of a method for storing data in a wide table according to another embodiment of the present disclosure.

[0027] Figure 5 The block diagram schematically shows a message data processing device according to an embodiment of the present disclosure.

[0028] Figure 6 The block diagram schematically shows an electronic device suitable for implementing the message data processing method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present disclosure. In the detailed description below, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0030] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise," "include," etc. used herein indicate the presence of the features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0031] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0032] When expressions such as “at least one of A, B, and C, etc.” are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (e.g., “a system having at least one of A, B, and C” should include but is not limited to systems having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, and C, etc.). When expressions such as “at least one of A, B, or C, etc.” are used, they should generally be interpreted in accordance with the meaning commonly understood by those skilled in the art (e.g., “a system having at least one of A, B, or C” should include but is not limited to systems having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, and C, etc.).

[0033] In related technologies, Redis is mainly used as an intermediary to assemble data in multiple message queues into a data wide table, and each consumer in the message queue can obtain and set data from the database.

[0034] However, when using Redis as an intermediary to generate wide data tables, each time a new message queue is added, it is necessary to consider whether the message queue will affect other message queues, and the logic of system development is relatively complex.

[0035] In view of this, the embodiments of the present disclosure improve the processing method of the message queue composition data wide table to reduce development complexity and improve program operation efficiency.

[0036] Specifically, an embodiment of the present disclosure provides a message data processing method, a message data processing device, an electronic device, a readable storage medium, and a computer program product. The method includes: in response to obtaining first message data in a message queue, competing to obtain a first distributed lock based on the primary key and status data of the first message data, wherein the first distributed lock is implemented based on a first database; based on the primary key and status data of the first message data, determining whether the first message data has been consumed; if it is determined that the first message data has not been consumed, using the first message data to process data in a target data wide table corresponding to the primary key of the first message data in a second database; adding second message data generated based on the primary key of the first message data to the message queue; and releasing the first distributed lock.

[0037] Figure 1 The following schematically illustrates an exemplary system architecture to which the message data processing method according to an embodiment of the present disclosure can be applied. Figure 1The examples shown are merely examples of system architectures to which the embodiments of the present disclosure may be applied, to help those skilled in the art understand the technical content of the present disclosure, but do not mean that the embodiments of the present disclosure may not be used in other devices, systems, environments or scenarios.

[0038] like Figure 1 As shown, the system architecture according to this embodiment may include terminal devices 101 , 102 , 103 , a network 104 and a server 105 .

[0039] The terminal devices 101 , 102 , and 103 may be electronic devices having a display screen and supporting message data reading and writing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0040] Various client applications may be installed on the terminal devices 101 , 102 , and 103 , such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software.

[0041] The server 105 can be a server that provides various services and can provide service support for various client applications installed on the terminal devices 101 , 102 , and 103 .

[0042] Different types of databases may be installed in the server 105 , including but not limited to MySQL, Redis, HBASE, etc.

[0043] The network 104 is a medium for providing a communication link between the terminal devices 101, 102, and 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc. Users can use the terminal devices 101, 102, and 103 to interact with the server 105 via the network 104 to receive or send messages, etc.

[0044] It should be noted that the message data processing method provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the message data processing device provided in the embodiment of the present disclosure can generally be set in the server 105. The message data processing method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the message data processing device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Alternatively, the message data processing method provided in the embodiment of the present disclosure can also be executed by the terminal devices 101, 102, or 103, or can also be executed by other terminal devices different from the terminal devices 101, 102, or 103. Accordingly, the message data processing device provided in the embodiment of the present disclosure may also be set in the terminal device 101 , 102 , or 103 , or in other terminal devices different from the terminal device 101 , 102 , or 103 .

[0045] For example, when a user uses a client application on a terminal device (for example, terminal device 101) to perform business operations, message data is generated; then, the terminal device 101 can locally execute the message data processing method provided by the embodiment of the present disclosure, or send the message data to other terminal devices, servers, or server clusters, and the other terminal devices, servers, or server clusters that receive the message data execute the message data processing method provided by the embodiment of the present disclosure.

[0046] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0047] Figure 2 The flowchart of the message data processing method according to the embodiment of the present disclosure is schematically shown.

[0048] like Figure 2 As shown, the method includes operations S201 to S205.

[0049] It should be noted that, unless it is explicitly stated that there is a sequence of execution between different operations shown in the flowchart in the embodiments of the present disclosure, or there is a sequence of execution between different operations in technical implementation, otherwise, the execution order between multiple operations may not be prioritized, and multiple operations may also be executed simultaneously.

[0050] In operation S201 , in response to obtaining first message data in a first message queue, a first distributed lock is acquired based on a primary key and state data of the first message data.

[0051] In operation S202 , it is determined whether the first message data has been consumed based on the primary key and state data of the first message data.

[0052] In operation S203 , if it is determined that the first message data has not been consumed, data in a target data wide table corresponding to a primary key of the first message data in the second database is processed using the first message data.

[0053] In operation S204 , the second message data generated based on the primary key of the first message data is added to the second message queue.

[0054] In operation S205 , the first distributed lock is released.

[0055] According to embodiments of the present disclosure, there can be multiple first message queues, and message data with the same primary key but different status data can be distributed across different first message queues. For example, an order system can have three message queues, where the primary key of the message data in each of these three message queues is the order number, and the status data are order status, order confirmation status, and order information, respectively.

[0056] According to an embodiment of the present disclosure, the first distributed lock may be implemented based on the first database.

[0057] According to an embodiment of the present disclosure, the type of the first database is not limited. For example, it can be Redis in a key-value database.

[0058] According to embodiments of the present disclosure, different first distributed locks can be used for different state data of the same primary key. According to embodiments of the present disclosure, due to factors such as network delays and system error restarts, duplicate first message data may exist in the message queue. Therefore, duplicate message data can be deduplicated by determining whether the first message data has been consumed, thereby reducing the occurrence of data redundancy in the generated data wide table and improving data processing efficiency.

[0059] According to the embodiments of the present disclosure, any method can be used to determine whether the first message data has been consumed. For example, when consuming a message, the consumed message data can be stored in a cache area of ​​the first database. When reading new first message data, the characteristics of the first message data (e.g., primary key, status data, etc.) are compared with the characteristics of the message data in the cache area, and based on the comparison result, it is determined whether the first message data has been consumed.

[0060] According to the embodiment of the present disclosure, the type of the second database is not limited, for example, it can be HBASE in a key-value database.

[0061] According to an embodiment of the present disclosure, there may be multiple pre-created data wide tables in the second database, and each data wide table may have multiple columns, wherein the first column of the data wide table stores a primary key to indicate the message data that the data wide table can store; the other columns of the data wide table can store status data in the message data.

[0062] According to an embodiment of the present disclosure, when the first message data is not consumed, it indicates that the data in the data wide table of the second database will change. At this time, the second message data can be generated according to the primary key and stored in the second message queue. The second message data can be consumed by other modules in the system.

[0063] According to an embodiment of the present disclosure, the second message queue may be different from the first message queue.

[0064] According to an embodiment of the present disclosure, before acquiring first message data, a first distributed lock is competed for based on the primary key and status data of the first message data. After successfully acquiring the first distributed lock, if the first message data has not been consumed, the first message data is stored in a second database to generate a data wide table. Second message data is then generated based on the primary key of the first message data, and the first distributed lock is released. By utilizing a combination of primary keys and status data to acquire distributed locks, the technical problem of reduced data processing efficiency caused by data in multiple message queues competing for the same lock is at least partially overcome, thereby effectively improving the efficiency of message data processing.

[0065] Figure 3 The figure schematically shows a message data processing method according to another embodiment of the present disclosure.

[0066] like Figure 3 As shown, the message data processing method may include a data wide table generating method, a data wide table version management method, and a data wide table storage method.

[0067] According to an embodiment of the present disclosure, in the data wide table generation method, after deduplication, the message data in the first message queue 301 will be stored in the first database 302 and the second database 303 respectively, and the second message data will be sent to the second message queue 304.

[0068] According to an embodiment of the present disclosure, in the data wide table version management method, after receiving the second message data in the second message queue 304, the data wide table 305 corresponding to the primary key of the second message data can be retrieved from the second database 303, and based on the historical coding value of the data wide table 305 originally stored in the first database 302, it is determined whether the version of the data wide table 305 needs to be updated; if the version of the data wide table 305 needs to be updated, the current coding value of the data wide table 305 is stored in the first database 302, and the third message data is sent to the third message queue 306.

[0069] According to an embodiment of the present disclosure, in the wide data table storage method, the version number of the wide data table 305 can be verified based on the first database 302. If the verification result indicates that the wide data table 305 needs to be saved, the wide data table 305 is stored in the third database 307. According to an embodiment of the present disclosure, the third database can be any database, for example, MySQL, a relational database. The third database has better compatibility than the first and second databases.

[0070] According to an embodiment of the present disclosure, the three sub-methods in the message data processing method can be implemented by three independent software / hardware modules, and the three modules interact with each other through message queues.

[0071] Reference below Figures 4A to 4C , combined with specific embodiments Figure 2 and Figure 3 The method shown is further explained.

[0072] Figure 4A The figure is a flow chart of a method for producing a wide data table according to another embodiment of the present disclosure.

[0073] like Figure 4A As shown, the method includes operations S4101 to S4108.

[0074] In operation S4101, first message data in a first message queue is acquired.

[0075] In operation S4102 , a first distributed lock based on a primary key and state data of the first message data is acquired through competition.

[0076] In operation S4103, it is determined whether the first distributed lock is acquired successfully. If it is determined that the first distributed lock acquisition fails, operation S4104 is performed; if it is determined that the first distributed lock is acquired successfully, operation S4105 is performed.

[0077] In operation S4104 , a retry procedure is enabled to compete for acquiring the first distributed lock again.

[0078] In operation S4105, it is determined whether the first message data has been consumed. If it is determined that the first message data has not been consumed, operation S4106 is performed; if it is determined that the first message data has been consumed, operation S4109 is performed.

[0079] In operation S4106, the first message data is stored in a target data wide table of the second database.

[0080] In operation S4107 , second message data generated based on the primary key of the first message data is sent to the second message queue.

[0081] In operation S4108, the first message data is stored in the first database.

[0082] In operation S4109, the first distributed lock is released.

[0083] According to an embodiment of the present disclosure, any method may be used to determine whether the first distributed lock is successfully acquired.

[0084] For example, since the first distributed lock is implemented based on the first database, the identification value of the first distributed lock can be obtained from the first database; if the identification value of the first distributed lock is determined to be a first preset value indicating that it can be obtained, the first distributed lock is determined to have been successfully obtained; if the identification value of the first distributed lock is determined to be a second preset value indicating that it cannot be obtained, the first distributed lock is determined to have failed to be obtained. When acquiring and releasing the first distributed lock, the identification value of the first distributed lock in the first database can be modified accordingly to prevent multiple message data from competing for the same distributed lock at the same time.

[0085] According to an embodiment of the present disclosure, the first preset value and the second preset value may be any two different values ​​and may be set by a system developer when creating a distributed lock.

[0086] According to an embodiment of the present disclosure, the restart program may continue to attempt to compete for the first distributed lock after a period of time, and the specific time interval may be set arbitrarily.

[0087] According to an embodiment of the present disclosure, the consumed first message data may be stored as historical message data in a historical message set of the first database.

[0088] According to an embodiment of the present disclosure, whether the first message data has been consumed can be determined by the following method: obtain multiple historical message data stored in the historical message set in the first database; compare the primary key and status data of the first message data with the primary keys and status data of multiple historical message data in turn to obtain a first comparison result; if the first comparison result shows that the multiple historical message data do not match the first message data, determine that the first message data has not been consumed; if the first comparison result shows that there is message data matching the first message data among the multiple historical message data, determine that the first message data has been consumed.

[0089] According to an embodiment of the present disclosure, the historical message data in the historical message set in the first database can be cleaned up according to a program operation cycle, an enterprise data processing cycle, or any other preset method to ensure the operation efficiency of the system.

[0090] According to an embodiment of the present disclosure, a data wide table may uniquely correspond to a primary key, may have multiple columns, and each column may have corresponding status data. The data wide table may be as shown in Table 1.

[0091] Table 1

[0092] Rowkey Column 1 Column 2 Column 3 Column 4 Primary Key Status data 1 Status data 2 Status data 3 Status data 4

[0093] According to an embodiment of the present disclosure, the data in the target data wide table can be processed by the following method: determining the target column corresponding to the status data of the first message data in the target data wide table; when there is existing data in the target column, using the status data of the first message data to replace the existing data in the target column; when there is no existing data in the column, storing the status data of the first message data in the target column.

[0094] Taking orders as an example, the information originally stored in the wide target data table in the second database may be as shown in Table 2. The state data of the first message data may correspond to "YnState," and its state data value is "0." In this case, the data in the wide target data table, namely, in the "YnState" column, is processed using the first message data, replacing the original "1" with "0."

[0095] Table 2

[0096] Rowkey OrderState YnState info Order ID 9 1 {"orderId": 1234, "storeId": 131229, "amt": 160.8}

[0097] Figure 4B The figure is a flowchart of a data wide table version management method according to another embodiment of the present disclosure.

[0098] like Figure 4B As shown, the method includes operations S4201 to S4211.

[0099] In operation S4201, second message data in the second message queue is acquired.

[0100] In operation S4202 , a second distributed lock based on a primary key of the second message data is acquired through competition.

[0101] In operation S4203, it is determined whether the second distributed lock is acquired successfully. If it is determined that the second distributed lock acquisition fails, operation S4204 is performed; if it is determined that the second distributed lock is acquired successfully, operation S4205 is performed.

[0102] In operation S4204 , a retry procedure is enabled to compete for acquiring the second distributed lock again.

[0103] In operation S4205, the historical code value stored in the first database is obtained.

[0104] In operation S4206, a current encoding value of the target data wide table in the second database is calculated.

[0105] In operation S4207, it is determined whether the historical code value and the current code value are the same. If it is determined that the historical code value and the current code value are different, operation S4208 is performed; if it is determined that the historical code value and the current code value are the same, operation S4211 is performed.

[0106] In operation S4208, a current version number of the target data wide table is set based on the current timestamp.

[0107] In operation S4209 , the third message data generated based on the target data wide table is sent to the third message queue.

[0108] In operation S4210, the current code value is used to replace the historical code value in the first database.

[0109] In operation S4211, the second distributed lock is released.

[0110] According to an embodiment of the present disclosure, the second distributed lock may be generated based on the primary key of the message data.

[0111] According to an embodiment of the present disclosure, determining whether the second distributed lock is successfully acquired may adopt the same technical solution as determining whether the first distributed lock is successfully acquired, which will not be repeated here.

[0112] According to an embodiment of the present disclosure, the historical code value may correspond to the primary key of the message data.

[0113] According to an embodiment of the present disclosure, in operation S4205, there may be no historical coding value in the first database, indicating that the current process is the process of the first registration version number of the data wide table corresponding to the primary key of the second message data; in this case, the historical coding value can be set as the current coding value.

[0114] According to an embodiment of the present disclosure, the encoding value of the data wide table may be generated by any encoding algorithm or encryption algorithm. For example, MD5 encoding may be used, and the MD5 value of the target data wide table may be used as the current encoding value.

[0115] According to an embodiment of the present disclosure, the current timestamp can be obtained by collecting system time. The magnitude of the current timestamp can be set arbitrarily, for example, the current timestamp can be a millisecond-level timestamp.

[0116] According to an embodiment of the present disclosure, the current timestamp can be directly used as the current version number of the target data wide table; or, a limited number of function operations can be performed on the pre-point timestamp, and the operation results can be used as the current version number of the target data wide table.

[0117] Figure 4C The figure is a flowchart of a method for storing data in a wide table according to another embodiment of the present disclosure.

[0118] like Figure 4C As shown, the method includes operations S4301 to S4309.

[0119] In operation S4301, third message data in a third message queue is acquired.

[0120] In operation S4302 , a third distributed lock based on a primary key of a target data wide table in the third message data is acquired through competition.

[0121] In operation S4303, it is determined whether the third distributed lock is acquired successfully. If it is determined that the third distributed lock acquisition fails, operation S4304 is performed; if it is determined that the third distributed lock is acquired successfully, operation S4305 is performed.

[0122] In operation S4304, a retry procedure is enabled to compete for acquiring the third distributed lock again.

[0123] In operation S4305 , a historical version number of the target data wide table stored in the first database is obtained.

[0124] In operation S4306, it is determined whether the current version number of the target data wide table in the third message data is greater than the historical version number. If the current version number is greater than the historical version number, operation S4307 is executed; if the current version number is less than or equal to the historical version number, operation S4308 is executed.

[0125] In operation S4307, the target data wide table is stored in the third database.

[0126] In operation S4308, the current version number is stored in the first database.

[0127] In operation S4309, the third distributed lock is released.

[0128] According to an embodiment of the present disclosure, the third distributed lock may be generated based on the primary key of the message data.

[0129] According to an embodiment of the present disclosure, determining whether the third distributed lock is successfully acquired may adopt the same technical solution as determining whether the first distributed lock is successfully acquired, which will not be repeated here.

[0130] According to an embodiment of the present disclosure, the historical version number may correspond to the primary key of the data wide table.

[0131] According to an embodiment of the present disclosure, in operation S4305, the historical version number may not exist in the first database, indicating that the current process is the process of storing the data wide table of the third message data type into the third database for the first time; in this case, the historical version number can be set to any value smaller than the current version number.

[0132] Figure 5 The block diagram schematically shows a message data processing device according to an embodiment of the present disclosure.

[0133] like Figure 5 As shown, the message data processing device includes a first locking module 510 , a first judging module 520 , a first processing module 530 , a second processing module 540 and a first unlocking module 550 .

[0134] The first locking module 510 is configured to, in response to obtaining first message data in the first message queue, compete for a first distributed lock based on a primary key and status data of the first message data, wherein the first distributed lock is implemented based on a first database.

[0135] The first judgment module 520 is configured to determine whether the first message data has been consumed based on the primary key and status data of the first message data.

[0136] The first processing module 530 is configured to, when it is determined that the first message data has not been consumed, use the first message data to process data in a target data wide table corresponding to a primary key of the first message data in the second database.

[0137] The second processing module 540 is configured to add the second message data generated based on the primary key of the first message data into the second message queue.

[0138] The first unlocking module 550 is configured to release the first distributed lock.

[0139] According to an embodiment of the present disclosure, before acquiring first message data, a first distributed lock is competed for based on the primary key and status data of the first message data. After successfully acquiring the first distributed lock, if the first message data has not been consumed, the first message data is stored in a second database to generate a data wide table. Second message data is then generated based on the primary key of the first message data, and the first distributed lock is released. By utilizing a combination of primary keys and status data to acquire distributed locks, the technical problem of reduced data processing efficiency caused by data in multiple message queues competing for the same lock is at least partially overcome, thereby effectively improving the efficiency of message data processing.

[0140] According to an embodiment of the present disclosure, the first judgment module 520 includes a first judgment unit, a second judgment unit, a third judgment unit, and a fourth judgment unit.

[0141] The first judgment unit is configured to obtain a plurality of historical message data from a historical message set in a first database.

[0142] The second judgment unit is configured to compare the first message data with the plurality of historical message data in sequence based on the primary key and the status data to obtain a first comparison result.

[0143] The third judgment unit is configured to determine that the first message data has not been consumed if the first comparison result indicates that the plurality of historical message data do not match the first message data.

[0144] The fourth judgment unit is configured to determine that the first message data has been consumed if the first comparison result indicates that message data matching the first message data exists in the plurality of historical message data.

[0145] According to an embodiment of the present disclosure, the third judgment unit is further configured to store the first message data as historical message data in a historical message set of the first database when it is determined that the first message data has not been consumed.

[0146] According to an embodiment of the present disclosure, the fourth judgment unit is further configured to release the first distributed lock when it is determined that the first message data has been consumed.

[0147] According to an embodiment of the present disclosure, the target data wide table has multiple columns, and each column has corresponding status data.

[0148] According to an embodiment of the present disclosure, the first processing module 530 includes a first processing unit, a second processing unit, and a third processing unit.

[0149] The first processing unit is configured to determine a target column corresponding to the status data of the first message data in the target data wide table.

[0150] The second processing unit is configured to replace the existing data in the target column with the state data of the first message data when there is existing data in the target column.

[0151] The third processing unit is configured to store the state data of the first message data into a target column if no existing data exists in the column.

[0152] According to an embodiment of the present disclosure, the device further includes a second locking module, a coding module, a second judgment module, a setting module, a third processing module and a second unlocking module.

[0153] The second locking module is configured to acquire a second distributed lock based on a primary key of the second message data in response to obtaining the second message data in the second message queue, wherein the second distributed lock is implemented based on the first database.

[0154] The encoding module is used to calculate and obtain the current encoding value of the target data wide table in the second database.

[0155] The second judgment module is used to determine whether the second message data has been consumed based on the current encoding value.

[0156] The setting module is configured to set a current version number of the target data wide table based on a current timestamp when it is determined that the second message data has not been consumed.

[0157] The third processing module is configured to store the third message data generated based on the target data wide table into the third message queue.

[0158] The second unlocking module is used to release the second distributed lock.

[0159] According to an embodiment of the present disclosure, the second judgment module includes a fifth judgment unit, a sixth judgment unit, a seventh judgment unit, and an eighth judgment unit.

[0160] The fifth judgment unit is used to obtain the historical coding value of the target data wide table from the first database.

[0161] The sixth judgment unit is used to compare the historical code value with the current code value to obtain a second comparison result.

[0162] The seventh judgment unit is configured to determine that the second message data has not been consumed if the second comparison result indicates that the historical coding value is different from the current coding value.

[0163] An eighth judgment unit is configured to determine that the second message data has been consumed if the second comparison result indicates that the historical coding value is the same as the current coding value.

[0164] According to an embodiment of the present disclosure, the seventh judgment unit is further configured to store the current coding value in the first database when it is determined that the second message data has not been consumed, so as to replace the historical coding value with the current coding value.

[0165] According to an embodiment of the present disclosure, the eighth judgment unit is further configured to release the second distributed lock when it is determined that the second message data has been consumed.

[0166] According to an embodiment of the present disclosure, the device further includes a third locking module, a third judgment module, a storage module and a third unlocking module.

[0167] The third locking module is used to obtain a third distributed lock based on the primary key of the target data wide table in the third message data in response to obtaining the third message data in the third message queue, wherein the third distributed lock is implemented based on the first database.

[0168] The third judgment module is configured to determine whether the third message data has been consumed based on the current version number of the target data wide table in the third message data.

[0169] The storage module is configured to store the target data wide table in a third database when it is determined that the third message data has not been consumed.

[0170] The third unlocking module is used to release the third distributed lock of the third message data.

[0171] According to an embodiment of the present disclosure, the third judgment module includes a ninth judgment unit, a tenth judgment unit, an eleventh judgment unit, and a twelfth judgment unit.

[0172] The ninth judgment unit is configured to obtain a historical version number of the target data wide table from the first database.

[0173] The tenth judgment unit is used to compare the historical version number with the current version number to obtain a third comparison result.

[0174] The eleventh judgment unit is configured to determine that the third message data has not been consumed if the third comparison result indicates that the current version number is greater than the historical version number.

[0175] The twelfth judgment unit is configured to determine that the third message data has been consumed if the third comparison result indicates that the current version number is less than or equal to the historical version number.

[0176] According to an embodiment of the present disclosure, the twelfth judgment unit is further configured to release the third distributed lock when it is determined that the third message data has been consumed.

[0177] According to the modules, submodules, units, and subunits of the embodiments of the present invention, any multiple or at least part of the functions of any multiple thereof can be implemented in one module. According to the modules, submodules, units, and subunits of the embodiments of the present invention, any one or more thereof can be split into multiple modules for implementation. According to the modules, submodules, units, and subunits of the embodiments of the present invention, any one or more thereof can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware of any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation modes of software, hardware, and firmware or in an appropriate combination of any of them. Alternatively, according to the modules, submodules, units, and subunits of the embodiments of the present invention, one or more thereof can be at least partially implemented as a computer program module, which can perform the corresponding function when the computer program module is run.

[0178] For example, any multiple of the first locking module 510, the first determination module 520, the first processing module 530, the second processing module 540, and the first unlocking module 550 may be combined into one module / unit / sub-unit for implementation, or any one of these modules / units / sub-units may be split into multiple modules / units / sub-units. Alternatively, at least part of the functionality of one or more of these modules / units / sub-units may be combined with at least part of the functionality of other modules / units / sub-units and implemented in one module / unit / sub-unit. According to an embodiment of the present disclosure, at least one of the first locking module 510, the first judgment module 520, the first processing module 530, the second processing module 540, and the first unlocking module 550 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware through any other reasonable means of integrating or packaging circuits, or can be implemented in any one of the three implementation methods of software, hardware, and firmware, or in any appropriate combination of any of them. Alternatively, at least one of the first locking module 510, the first judgment module 520, the first processing module 530, the second processing module 540, and the first unlocking module 550 can be at least partially implemented as a computer program module, which can perform the corresponding function when executed.

[0179] It should be noted that the message data processing device part in the embodiment of the present disclosure corresponds to the message data processing method part in the embodiment of the present disclosure. The description of the message data processing device part specifically refers to the message data processing method part and will not be repeated here.

[0180] Figure 6 The block diagram schematically shows an electronic device suitable for implementing the message data processing method according to an embodiment of the present disclosure. Figure 6 The electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0181] like Figure 6 As shown, the computer electronic device 600 according to an embodiment of the present disclosure includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (such as a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (such as an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include an onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for executing different actions of the method flow according to an embodiment of the present disclosure.

[0182] Various programs and data required for the operation of the electronic device 600 are stored in the RAM 603. The processor 601, ROM 602, and RAM 603 are connected to each other via a bus 604. The processor 601 executes the various operations of the method flow according to the embodiment of the present disclosure by executing the programs in the ROM 602 and / or RAM 603. It should be noted that the programs may also be stored in one or more memories other than the ROM 602 and RAM 603. The processor 601 may also execute the various operations of the method flow according to the embodiment of the present disclosure by executing the programs stored in the one or more memories.

[0183] According to an embodiment of the present disclosure, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to the bus 604. The electronic device 600 may further include one or more of the following components connected to the I / O interface 605: an input portion 606 including a keyboard, a mouse, etc.; an output portion 607 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage portion 608 including a hard disk; and a communication portion 609 including a network interface card such as a LAN card or a modem. The communication portion 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed in the drive 610 as needed, so that a computer program read therefrom can be installed into the storage portion 608 as needed.

[0184] According to an embodiment of the present disclosure, the method flow according to an embodiment of the present disclosure can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, the above-mentioned functions defined in the system of the embodiment of the present disclosure are executed. According to an embodiment of the present disclosure, the system, equipment, device, module, unit, etc. described above can be implemented by a computer program module.

[0185] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not be incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, and when executed, implements the method according to the embodiments of the present disclosure.

[0186] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium. For example, it may include, but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0187] For example, according to an embodiment of the present disclosure, the computer-readable storage medium may include the ROM 602 and / or the RAM 603 described above and / or one or more memories other than the ROM 602 and the RAM 603 .

[0188] An embodiment of the present disclosure also includes a computer program product, which includes a computer program, which contains program code for executing the method provided by the embodiment of the present disclosure. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the message data processing method provided by the embodiment of the present disclosure.

[0189] When the computer program is executed by the processor 601, the above functions defined in the system / device of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0190] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 609, and / or installed from a removable medium 611. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0191] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0192] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0193] Those skilled in the art will appreciate that various combinations and / or combinations of features described in the various embodiments and / or claims of this disclosure may be made, even if such combinations or combinations are not explicitly described in this disclosure. In particular, various combinations and / or combinations of features described in the various embodiments and / or claims of this disclosure may be made, without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0194] The embodiments of the present disclosure are described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be used in combination to advantage. The scope of the present disclosure is defined by the appended claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A message data processing method, comprising: In response to obtaining first message data in a first message queue, competing to obtain a first distributed lock based on a primary key and state data of the first message data, wherein the first distributed lock is implemented based on a first database; Determining whether the first message data has been consumed based on the primary key and status data of the first message data includes: comparing the primary key and status data of the first message data with the primary key and status data of consumed message data in a cache area of ​​the first database, and determining whether the first message data has been consumed based on the comparison result; When it is determined that the first message data has not been consumed, using the first message data to process data in a target data wide table corresponding to a primary key of the first message data in a second database; Adding second message data generated based on the primary key of the first message data to a second message queue; and Release the first distributed lock.

2. The method according to claim 1, wherein The determining, based on the primary key and status data of the first message data, whether the first message data has been consumed includes: Acquire a plurality of historical message data from the historical message set of the first database; Based on the primary key and the state data, the first message data is sequentially compared with the plurality of historical message data to obtain a first comparison result; If the first comparison result indicates that none of the plurality of historical message data matches the first message data, determining that the first message data has not been consumed; and If the first comparison result indicates that message data matching the first message data exists among the plurality of historical message data, it is determined that the first message data has been consumed.

3. The method according to claim 2, further comprising: If it is determined that the first message data has not been consumed, storing the first message data as historical message data in a historical message set of the first database; as well as When it is determined that the first message data has been consumed, the first distributed lock is released.

4. The method according to claim 1, wherein The target data wide table has multiple columns, each column has corresponding status data; The step of using the first message data to process data in a target data wide table in the second database corresponding to a primary key of the first message data includes: Determine a target column corresponding to the status data of the first message data in the target data wide table; In the case where there is existing data in the target column, replacing the existing data in the target column with the state data of the first message data; and When no existing data exists in the column, the state data of the first message data is stored in the target column.

5. The method according to claim 1, further comprising: In response to obtaining the second message data in the second message queue, acquiring a second distributed lock based on the primary key competition of the second message data, wherein the second distributed lock is implemented based on the first database; Calculate and obtain the current encoding value of the target data wide table in the second database; determining, based on the current encoding value, whether the second message data has been consumed; If it is determined that the second message data has not been consumed, setting a current version number of the target data wide table based on a current timestamp; storing the third message data generated based on the target data wide table into a third message queue; and Release the second distributed lock.

6. The method according to claim 5, wherein: The determining, based on the current encoding value, whether the second message data has been consumed, wherein determining whether the second message data has been consumed includes: Acquire historical coding values ​​of the target data wide table from the first database; Comparing the historical code value with the current code value to obtain a second comparison result; If the second comparison result indicates that the historical encoding value is different from the current encoding value, determining that the second message data has not been consumed; and If the second comparison result indicates that the historical encoding value is identical to the current encoding value, it is determined that the second message data has been consumed.

7. The method according to claim 6, further comprising: If it is determined that the second message data has not been consumed, storing the current encoding value in the first database to replace the historical encoding value with the current encoding value; as well as When it is determined that the second message data has been consumed, the second distributed lock is released.

8. The method according to claim 5, further comprising: In response to obtaining the third message data in the third message queue, acquiring a third distributed lock based on the primary key of the target data wide table in the third message data through competition, wherein the third distributed lock is implemented based on the first database; determining, based on a current version number of a target data wide table in the third message data, whether the third message data has been consumed; If it is determined that the third message data has not been consumed, storing the target data wide table in a third database; and Release the third distributed lock.

9. The method according to claim 8, wherein The determining, based on the current version number, whether the third message data has been consumed includes: Obtaining a historical version number of the target data wide table from the first database; Comparing the historical version number with the current version number to obtain a third comparison result; If the third comparison result indicates that the current version number is greater than the historical version number, determining that the third message data has not been consumed; and If the third comparison result indicates that the current version number is less than or equal to the historical version number, it is determined that the third message data has been consumed.

10. The method according to claim 9, further comprising: When it is determined that the third message data has been consumed, the third distributed lock is released.

11. A message data processing device, comprising: a first locking module, configured to, in response to obtaining first message data in a first message queue, compete for a first distributed lock based on a primary key and state data of the first message data, wherein the first distributed lock is implemented based on a first database; a first judgment module, configured to determine whether the first message data has been consumed based on the primary key and status data of the first message data, comprising: comparing the primary key and status data of the first message data with the primary key and status data of consumed message data in the cache area of ​​the first database, and determining whether the first message data has been consumed based on the comparison result; a first processing module, configured to, when determining that the first message data has not been consumed, use the first message data to process data in a target data wide table corresponding to a primary key of the first message data in a second database; A second processing module, configured to add second message data generated based on the primary key of the first message data to a second message queue; and A first unlocking module is configured to release the first distributed lock.

12. An electronic device comprising: one or more processors; a memory for storing one or more instructions, Wherein, when the one or more instructions are executed by the one or more processors, the one or more processors are enabled to implement the method according to any one of claims 1 to 10.

13. A computer-readable storage medium having executable instructions stored thereon, wherein when the executable instructions are executed by a processor, the processor is enabled to implement the method according to any one of claims 1 to 10.

14. A computer program product comprising computer executable instructions for implementing the method of any one of claims 1 to 10 when the computer executable instructions are executed.

Images