Data processing method and device, computer equipment and computer readable storage medium

Abstract

The application discloses a data processing method and device, computer equipment and a computer readable storage medium, which can be applied to cloud technology, artificial intelligence, intelligent transportation, auxiliary driving and various scenes, and the method comprises the steps of obtaining a plurality of messages from a message queue; storing the plurality of messages and the update time of each message in a first database synchronously, and adding the plurality of messages to a plurality of memory queues; in the process that a target data source consumes messages from the plurality of memory queues, storing the first head time and the first tail time corresponding to each memory queue in the plurality of memory queues in a second database, the first head time comprising the update time of a head message, and the first tail time comprising the update time of a tail message, the first database and the second database being used for recovering the messages in the plurality of memory queues to the target data source when a restart event occurs. Through the application, the data synchronization efficiency can be improved, and data loss can be prevented, so that the safety of data is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to data processing methods, data processing apparatus, computer equipment, and computer-readable storage media. Background Technology

[0002] Message Queues (MQ) are distributed system middleware designed based on a first-in, first-out (FIFO) data structure, primarily addressing issues such as application coupling, asynchronous messaging, and traffic shaping. Currently, data synchronization solutions typically involve placing data into message queues and using persistence technology to store the message queues as files on the local disk, then using the message queues to synchronize data with different data sources. However, on the one hand, disk read / write speeds are slow, resulting in low data synchronization efficiency; on the other hand, when using virtualization container technology for data synchronization, more and more services are not dependent on specific machines. If a machine rebuild occurs, the host machine may be changed, leading to the loss of backup disk files.

[0003] Therefore, improving data synchronization efficiency and data security are problems that need to be solved. Summary of the Invention

[0004] This application provides a data processing method, apparatus, computer equipment, and computer-readable storage medium, which can improve data synchronization efficiency and prevent data loss, thereby improving data security.

[0005] On one hand, embodiments of this application provide a data processing method, the method comprising:

[0006] Retrieve multiple messages from a message queue, which contains messages read from a database log file;

[0007] Multiple messages and their update times are synchronously stored in the first database, and the messages are added to multiple memory queues.

[0008] During the process of the target data source consuming messages from multiple memory queues, the first head time and the first tail time corresponding to each memory queue are stored in the second database. The first head time includes the update time of the head message and the first tail time includes the update time of the tail message. The first database and the second database are used to restore the messages in the multiple memory queues to the target data source when a restart event occurs.

[0009] On one hand, embodiments of this application provide a data processing apparatus, the apparatus comprising:

[0010] The retrieval unit is used to retrieve multiple messages from a message queue, which includes messages read from a database log file;

[0011] The processing unit is used to synchronously store multiple messages and the update time of each message in the first database, and add multiple messages to multiple memory queues;

[0012] The processing unit is also used to store the first head time and the first tail time corresponding to each memory queue in the multiple memory queues into the second database during the process of the target data source consuming messages from multiple memory queues. The first head time includes the update time of the head message and the first tail time includes the update time of the tail message. The first database and the second database are used to restore the messages in the multiple memory queues to the target data source when a restart event occurs.

[0013] On one hand, embodiments of this application provide a computer device, which includes a memory and a processor. The memory stores a computer program, and when the computer program is executed by the processor, the processor performs the data processing method described above.

[0014] On one hand, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when read and executed by a processor of a computer device, causes the computer device to perform the aforementioned data processing method.

[0015] On one hand, embodiments of this application provide a computer program product, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the aforementioned data processing method.

[0016] In this embodiment, multiple messages are first retrieved from a message queue. Then, the multiple messages and their update times are synchronously stored in a first database, and the messages are added to multiple memory queues. During the process of the target data source consuming messages from the multiple memory queues, the first head time and first tail time corresponding to each memory queue are stored in a second database. Here, the first head time includes the update time of the head message, and the first tail time includes the update time of the tail message. Furthermore, the first and second databases are used to restore messages from the multiple memory queues to the target data source in the event of a restart. By using the first database to back up the multiple messages and their update times, and using the second database to store the first head time and first tail time corresponding to each memory queue, the messages in the multiple memory queues can be restored using the data stored in the first and second databases in the event of a restart. This improves data synchronization efficiency, prevents data loss, and thus enhances data security. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the structure of a data processing system provided in an embodiment of this application;

[0019] Figure 2 This is a flowchart illustrating a data processing method provided in an embodiment of this application;

[0020] Figure 3 This is a flowchart illustrating another data processing method provided in an embodiment of this application;

[0021] Figure 4 This is a flowchart illustrating another data processing method provided in an embodiment of this application;

[0022] Figure 5 This is a flowchart illustrating another data processing method provided in an embodiment of this application;

[0023] Figure 6 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0024] Figure 7 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0025] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0026] It should be noted that the terms "first," "second," etc., used in the embodiments of this application are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a technical feature specified with "first" or "second" may explicitly or implicitly include at least one of those features.

[0027] This application involves cloud technology, which refers to a hosting technology that unifies hardware, software, and network resources within a wide area network (WAN) or local area network (LAN) to achieve data computation, storage, processing, and sharing. Cloud technology is a general term encompassing network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form resource pools, providing flexible and convenient on-demand access. Cloud computing technology will become a crucial support. Backend services of technical network systems require substantial computing and storage resources, such as video websites, image websites, and many portal websites. With the rapid development and application of the internet industry, every item may have its own identification mark in the future, requiring transmission to a backend system for logical processing. Data at different levels will be processed separately, and various industry data will require robust system support, which can only be achieved through cloud computing.

[0028] Cloud storage is a new concept that extends and develops from cloud computing. A distributed cloud storage system (hereinafter referred to as a storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to aggregate a large number of various types of storage devices (also called storage nodes) in a network through application software or application interfaces to work together and provide data storage and business access functions. Currently, the storage method of a storage system is as follows: Logical volumes are created. When creating a logical volume, physical storage space is allocated to each logical volume. This physical storage space may consist of the disks of one or several storage devices. Clients store data on a logical volume, which means storing the data on the file system. The file system divides the data into many parts, each part being an object. Each object contains not only data but also additional information such as a data identifier (ID entity). The file system writes each object to the physical storage space of the logical volume and records the storage location information of each object. Therefore, when a client requests access to data, the file system can allow the client to access the data based on the storage location information of each object. The process by which a storage system allocates physical storage space to a logical volume is as follows: the physical storage space is pre-divided into strips according to the capacity estimate of the objects stored in the logical volume (this estimate often has a large margin relative to the actual capacity of the objects to be stored) and the grouping of Redundant Array of Independent Disks (RAID). A logical volume can be understood as a strip, thus allocating physical storage space to the logical volume.

[0029] A database, simply put, can be viewed as an electronic filing cabinet—a place to store electronic files, where users can perform operations such as adding, querying, updating, and deleting data. A "database" is a collection of data stored together in a certain way, shared by multiple users, with minimal redundancy, and independent of application programs. A Database Management System (DBMS) is a computer software system designed to manage databases, generally possessing basic functions such as storage, retrieval, security, and backup. DBMSs can be classified according to the database model they support, such as relational or Extensible Markup Language (XML); or according to the type of computer they support, such as server clusters or mobile phones; or according to the query language used, such as Structured Query Language (SQL); or according to performance priorities, such as maximum scale or maximum operating speed; or other classification methods. Regardless of the classification method used, some DBMSs can cross categories, for example, simultaneously supporting multiple query languages. In this application, the embodiments can be applied to key-value (kv) databases. A kv database is a database that stores data in key-value pairs, where each key corresponds to a unique value. For example, Redis, short for Remote Dictionary Server, is an open-source, in-memory, high-performance, persistent key-value storage system that can be used as a database, cache, and message broker.

[0030] Big data refers to data sets that cannot be captured, managed, and processed within a certain timeframe using conventional software tools. It represents massive, rapidly growing, and diverse information assets that require new processing models to achieve stronger decision-making, insightful discovery, and process optimization capabilities. With the advent of the cloud era, big data has attracted increasing attention. Big data requires specialized technologies to effectively process large amounts of data within a tolerable timeframe. Technologies suitable for big data include massively parallel processing databases, data mining, distributed file systems, distributed databases, cloud computing platforms, the internet, and scalable storage systems.

[0031] Based on the aforementioned cloud technology, this application proposes a data processing method to ensure data security. Specifically, the general principle of this data processing method is as follows: First, multiple messages are retrieved from a message queue; then, the multiple messages and their update times are synchronously stored in a first database, and the multiple messages are added to multiple memory queues; and during the process of the target data source consuming messages from the multiple memory queues, the first head time and the first tail time corresponding to each memory queue are stored in a second database. Here, the first head time includes the update time of the head message, and the first tail time includes the update time of the tail message. Furthermore, the first and second databases are used to restore the messages in the multiple memory queues to the target data source in the event of a restart. Here, "consumption" refers to the target data source retrieving messages from the memory queues and executing corresponding data logic processing.

[0032] In specific implementations, the data processing methods mentioned above can be executed by computer equipment, which can be a terminal device or a server. The terminal device can be, for example, a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, vehicle terminal, aircraft, etc., but is not limited to these. The server can be, for example, a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The embodiments of this application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0033] Specifically, the data processing methods mentioned above can be executed by the server. For example, see... Figure 1 As shown: Server 101 retrieves multiple messages from a message queue, then synchronously stores the multiple messages and their update times in a first database, and adds the multiple messages to multiple memory queues; and during the process of the synchronization data terminal 102 consuming messages from the multiple memory queues, it stores the first head time and first tail time corresponding to each memory queue in a second database, etc. Here, the synchronization data terminal 102 can be considered the target data source, which can refer to the aforementioned server or a terminal device, specifically corresponding to various business platforms or different functional departments within an enterprise, capable of consuming messages from server 101 to achieve data synchronization.

[0034] In this embodiment, a first database is used to back up multiple messages and the update time of each message, and a second database is used to store the first head time and the first tail time corresponding to each memory queue. When a restart event occurs, the data stored in the first and second databases can be used to restore the messages in the multiple memory queues, which can improve data synchronization efficiency and prevent data loss, thereby improving data security.

[0035] It is understood that the system architecture diagrams described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0036] Based on the above explanation, the following will be combined with Figure 2 The flowchart shown further illustrates the data processing method proposed in the embodiments of this application. In this embodiment, the data processing method is mainly described using the aforementioned computer device as an example. Please refer to... Figure 2 The data processing method may specifically include steps S201 to S203:

[0037] S201. Retrieve multiple messages from the message queue.

[0038] In this embodiment, the messages in the message queue are read from a database log file. A message queue (MQ) is a distributed system middleware designed based on a first-in, first-out (FIFO) data structure, primarily addressing issues such as application coupling, asynchronous messaging, and traffic shaping. The computer device reads multiple messages from the database log file and adds them to the message queue, allowing subsequent retrieval of messages directly from the queue. The database log file includes multiple messages and the update time of each message, used to trigger changes in other data sources. To ensure order, the message queue uses a single-partition approach, making it applicable to various services such as message distribution services and collaborative data recording services.

[0039] S202. Synchronously store multiple messages and the update time of each message in the first database, and add multiple messages to multiple memory queues.

[0040] In this embodiment, the memory queue can be viewed as a container for storing messages, employing a first-in, first-out (FIFO) data structure. Messages are retrieved from the container when needed. Each message has an update time. The computer device can perform collaborative data recording services through the message queue. This collaborative data recording service refers to synchronously storing multiple messages and their update times in a first database, thus backing up multiple messages and their update times. It can also perform message distribution services through the message queue. This message distribution service refers to adding multiple messages to multiple memory queues, allowing a target data source to consume messages from these queues. Specifically, the computer device can perform modulo or hash calculations on the primary key identifier of each message, adding messages with the same primary key identifier to the same memory queue. It should be understood that the computer device performs collaborative data recording and message distribution services synchronously. That is, while performing message distribution, the first database stores each distributed message and its update time. This method improves data synchronization efficiency and prevents data loss, thereby enhancing data security.

[0041] It should be noted that the first database is a temporary message storage database. Utilizing its transactional characteristics, it guarantees that each message is synchronized at least once. When the first database stores a large amount of data, expired messages can be deleted periodically. For example, at midnight each day, the computer will delete messages that have been confirmed to be synchronized to the target data source from the first database; or the computer will delete the earliest 100 stored messages from the first database.

[0042] In one possible implementation, the computer device adds multiple messages to multiple memory queues. Specifically, for any one of the messages, a target memory queue is determined from the multiple memory queues based on the primary key identifier of that message; then, that message is added to the target memory queue. It should be noted that after retrieving multiple messages from the message queues, the computer device performs a modulo or hash calculation on the primary key identifier and adds each message to its corresponding memory queue. This method ensures that messages with the same primary key identifier are in the same memory queue and guarantees the message order.

[0043] For example, the messages in the message queue are, from first to last, message A, message B, and message C. Message A has a primary key of 1, message B has a primary key of 2, message C has a primary key of 1, and message D has a primary key of 2. Assume the message with primary key 1 is added to memory queue a, and the message with primary key 2 is added to memory queue b. Based on the primary key of each message, message A is added to memory queue a, message B to memory queue b, message C to memory queue a, and message D to memory queue b. Therefore, the messages in memory queue a, from first to last, are message A and message C, and the messages in memory queue b, from first to last, are message B and message D.

[0044] S203. During the process of the target data source consuming messages from multiple memory queues, the first head time and the first tail time corresponding to each memory queue are stored in the second database.

[0045] In this embodiment, the first head time includes the update time of the head message, and the first tail time includes the update time of the tail message. A first database and a second database are used to restore messages in the multiple memory queues to the target data source in the event of a restart. Specifically, the first database stores multiple messages and the update time of each message, while the second database stores the first head time and the first tail time corresponding to each memory queue. The first database can be a temporary database, backing up multiple messages and their update times to ensure data security; the second database only needs to store the first head time and the first tail time corresponding to each memory queue, reducing the storage pressure on the second database. The target data source can consume messages from the multiple memory queues, and each memory queue has a separate thread that synchronizes to the target data source. For example, if a restart event occurs (such as a power outage restart or machine rebuild), the first head time and first tail time corresponding to each memory queue can be obtained from the second database. By using the first head time and first tail time corresponding to each memory queue, combined with multiple messages stored in the first database and the update time of each message, the computer device can obtain the message corresponding to each memory queue from the first database, realize the recovery of messages in multiple memory queues, and thus ensure the security of data.

[0046] In one possible implementation, the computer device stores the first head time and the first tail time corresponding to each of the multiple memory queues into a second database. Specifically, the implementation involves: obtaining the head message and the tail message of each memory queue; determining the first head time corresponding to each memory queue based on the update time of the head message, and determining the first tail time corresponding to each memory queue based on the update time of the tail message; and storing the first head time and the first tail time corresponding to each memory queue into the second database based on the queue identifier, head identifier, and tail identifier of each memory queue.

[0047] It should be noted that the second database can be a Redis database, which is a database that stores data in key-value pairs. The computer device can use the queue identifier and head identifier of each memory queue as the key field corresponding to the first head time, and the update time of the head message as the value field corresponding to the first head time; similarly, it can use the queue identifier and tail identifier of each memory queue as the key field corresponding to the first tail time, and the update time of the tail message as the value field corresponding to the first tail time.

[0048] In one possible implementation, the computer device stores the first head time and the first tail time corresponding to each memory queue in a second database based on the queue identifier, head identifier, and tail identifier of each memory queue. Specifically, the implementation is as follows: based on the queue identifier and head identifier of each memory queue, the second head time corresponding to each memory queue stored in the second database is obtained; if the first head time and the second head time corresponding to each memory queue are different, the first head time corresponding to each memory queue is stored in the second database.

[0049] It should be noted that before storing the first header time corresponding to each memory queue into the second database, the computer device needs to first retrieve the second header time corresponding to each memory queue already stored in the second database. Only if the first header time and the second header time are different is the first header time for each memory queue stored in the second database; otherwise, it is not stored. In other words, when the first header times of the memory queues are the same, they only need to be recorded once. When restoring messages, all messages within the update time interval stored in the second database must be restored, ensuring the at-least-once characteristic of messages. When comparing the first header time and the second header time, the smallest time unit can be seconds, minutes, or milliseconds; no specific limitation is made here. This method reduces the number of read / write operations on the computer device and improves data processing efficiency.

[0050] For example, using minutes as the smallest unit of time, the first header time is 9:10:20 and the second header time is 9:10:45. Comparing the first header time and the second header time, both correspond to 9:10. Since the first header time and the second header time are the same, there is no need to store the first header time in the second database.

[0051] For example, using minutes as the smallest unit of time, the first header time is 9:10:20 and the second header time is 9:15:45. Comparing the first header time and the second header time, the first header time corresponds to 9:10 and the second header time corresponds to 9:15. The first header time and the second header time are different, so the first header time needs to be stored in the second database.

[0052] In one possible implementation, the method further includes: obtaining the second tail time corresponding to each memory queue stored in the second database based on the queue identifier and tail identifier of each memory queue; if the first tail time and the second tail time corresponding to each memory queue are different, then the first tail time corresponding to each memory queue is stored in the second database. Similarly, before storing the first tail time corresponding to each memory queue in the second database, the computer device needs to first obtain the second tail time corresponding to each memory queue already stored in the second database. If the first tail time and the second tail time are different, then the first tail time corresponding to each memory queue is stored in the second database; if the first tail time and the second tail time are the same, then the first tail time corresponding to each memory queue is not stored in the second database. That is to say, when the first tail times of the memory queues are the same, only one recording is needed. When restoring messages, all messages within the update time interval stored in the second database need to be restored, ensuring the at least-once characteristic of messages. When comparing the first tail time and the second tail time, the smallest time unit can be seconds, minutes, or milliseconds; no limitation is made here. This method reduces the number of read / write operations on computer devices and improves data processing efficiency.

[0053] In summary, in this embodiment, multiple messages are first retrieved from a message queue; then, the multiple messages and their update times are synchronously stored in a first database, and the messages are added to multiple memory queues; and during the process of the target data source consuming messages from the multiple memory queues, the first head time and first tail time corresponding to each memory queue are stored in a second database. Here, the first head time includes the update time of the head message, and the first tail time includes the update time of the tail message. Furthermore, the first and second databases are used to restore messages from the multiple memory queues to the target data source in the event of a restart. By using the first database to back up the multiple messages and their update times, and using the second database to store the first head time and first tail time corresponding to each memory queue, the messages in the multiple memory queues can be restored using the data stored in the first and second databases in the event of a restart. This improves data synchronization efficiency, prevents data loss, and thus enhances data security.

[0054] The following is combined with Figure 3 The flowchart shown further illustrates the data processing method proposed in the embodiments of this application. In this embodiment, the data processing method is mainly described using the aforementioned computer device as an example. Please refer to... Figure 3 The data processing method may specifically include steps S301 to S306.

[0055] in:

[0056] S301. Retrieve multiple messages from the message queue.

[0057] S302. Synchronously store multiple messages and the update time of each message in the first database, and add multiple messages to multiple memory queues.

[0058] S303. During the process of the target data source consuming messages from multiple memory queues, the first head time and the first tail time corresponding to each memory queue in the multiple memory queues are stored in the second database.

[0059] The specific implementation methods of steps S301 to S303 can be referred to the specific implementation methods of steps S201 to S203 above, and will not be repeated here.

[0060] S304. If a restart event occurs, retrieve the first head time and the first tail time corresponding to each memory queue from the second database.

[0061] In this embodiment, when a computer device experiences a restart event (such as a power outage restart or machine rebuild), messages in the memory queues are lost, affecting data synchronization. Therefore, it is necessary to restore the messages corresponding to each memory queue from the target data source. First, the computer device obtains the first head time and the first tail time corresponding to each memory queue from the second database. In other words, it obtains the update time interval corresponding to each memory queue, and subsequently, all messages within this update time interval need to be restored.

[0062] S305. Based on the first head time and the first tail time corresponding to each memory queue, retrieve the message corresponding to each memory queue from the first database.

[0063] In this embodiment of the application, after the computer device obtains the first head time and the first tail time corresponding to each memory queue from the second database, it uses the first head time and the first tail time corresponding to each memory queue, combined with the multiple messages stored in the first database and the update time of each message, to enable the computer device to obtain the message corresponding to each memory queue from the first database.

[0064] In one possible implementation, the computer device retrieves the messages corresponding to each memory queue from a first database based on the first head time and the first tail time corresponding to each memory queue. Specifically, this involves: determining a target time interval for each memory queue based on the first head time and the first tail time; identifying messages whose update times fall within the target time interval from the messages stored in the first database; and using these messages as the messages corresponding to each memory queue. It should be noted that the computer device uses a second database to determine the target time interval for each memory queue, and then searches the first database for messages whose update times fall within the target time interval. These messages are then the messages corresponding to each memory queue, thus enabling the retrieval of the messages corresponding to each memory queue.

[0065] For example, suppose the first head time of memory queue 'a' stored in the second database is 1:00 AM, and the first tail time of memory queue 'a' is 2:00 AM. Then the target time interval for memory queue 'a' is 1:00 AM to 2:00 AM. The update time for message A stored in the first database is 1:20 AM, the update time for message B is 1:40 AM, the update time for message C is 2:00 AM, and the update time for message D is 2:40 AM. Therefore, the messages whose update times fall within this target time interval are messages A and B, meaning messages A and B are the messages corresponding to memory queue 'a'.

[0066] S306. Restore the messages corresponding to each memory queue to the target data source.

[0067] In this embodiment, after the computer device determines the message corresponding to each memory queue, it restores the message corresponding to each memory queue to the target data source, thereby restoring the entire memory data. Subsequently, the computer device can continue to consume messages from the memory queue normally, thus avoiding the impact of the restart event and ensuring data security.

[0068] In one possible implementation, the computer device recovers the messages corresponding to each memory queue from the target data source. Specifically, this can be achieved by: sending the messages corresponding to each memory queue to the target data source so that the target data source can consume the messages from each memory queue; or by adding the messages corresponding to each memory queue to each memory queue so that the target data source can consume the messages from each memory queue. It should be noted that the computer device can directly send the messages corresponding to each memory queue determined from the first database to the target data source for consumption, or it can add the messages corresponding to each memory queue determined from the first database to each memory queue, and then the target data source consumes the messages from each memory queue; this is not limited here.

[0069] In summary, the data processing methods described above can be categorized into the following five parts: Figure 4As shown, the first part is the database log file (DB Binlog) 401, which includes multiple messages and the update time of each message, used to trigger changes in other data sources. The second part is the message queue 402, which adds the messages included in the database log file 401 to the message queue 402. The message queue 402 acts as a distributed system middleware, using a single partition to ensure order, and can provide message distribution services, collaborative data recording services, etc. The third part is the message distribution service (reading message queue data 403), which adds multiple messages to multiple memory queues. The downstream synchronized target data source 406 consumes messages from multiple memory queues, and using memory queues can improve consumption speed. The fourth part is the collaborative data recording service, which uses the first database 404 as a consumer of the message queue to synchronously store multiple messages and the update time of each message, realizing the backup of multiple messages. The fifth part is the storage in the second database 405, which can be a Redis database, used to store the first head time and the first tail time corresponding to each queue. The first head time here includes the update time of the head message, and the first tail time here includes the update time of the tail message. In the event of a restart, the first head time and first tail time corresponding to each queue can be obtained using the second database 405. Combined with the multiple messages stored in the first database 404 and the update time of each message, messages in multiple memory queues can be restored to the downstream synchronized target data source 406.

[0070] In one embodiment, the head time can be denoted as Headtime, and the tail time can be denoted as Backtime.

[0071] Specifically, this data processing method can be summarized into the following steps: Figure 5 As shown, step 1 involves writing multiple messages from the database log file (DB Binlog) into a message queue (MQ). Each piece of data written must include an update time. Ultimately, this update time will be converted into the first header time and the first tail time, which are used to identify the time range of the memory queue.

[0072] Step 2 is the consumption process (message distribution service), which retrieves messages from a single partition of the message queue. In other words, it adds multiple messages to multiple in-memory queues. Each message corresponds to an update time.

[0073] Step 3 involves the message distribution service. After reading messages from the message queue, each message is added to a different memory queue based on its primary key. This ensures that messages with the same primary key are placed in the same memory queue, guaranteeing the order of messages. Simultaneously, the queue identifier and tail identifier of each queue are used as the key field corresponding to the first tail time of each queue, and the update time of the tail message in each queue is used as the value field corresponding to the first tail time. This data is then stored in a second database. The saved second tail time and first tail time can be compared; messages with the same time do not need to be stored in the second database again, thus reducing the number of read / write operations on the computer and improving data processing efficiency.

[0074] Step 4 involves using the queue identifier and head identifier of each memory queue as the key field corresponding to the first head time of each queue, and the update time of the head message of each queue as the value field corresponding to the first head time of each queue. This data is then saved using a second database. The saved second head time and first head time can be compared; if they are the same, the second database storage is not called again, thus reducing the number of read / write operations on the computer and improving data processing efficiency.

[0075] Step 5 is the collaborative data recording service, which synchronizes and stores multiple messages and the update time of each message to the first database. The transaction characteristics of the database are used to ensure that each message is stored at least once. The first database is a temporary database for storing messages. When the data volume is large, expired data can be deleted periodically.

[0076] Step 6 is that when the message distribution service encounters an anomaly and a restart event occurs, it will first read the first head time and the first tail time corresponding to all memory queues in the second database, use these two times to determine the target time interval, and retrieve the messages whose update time is within the target time interval from the first database, that is, the messages corresponding to each memory queue.

[0077] Step 7 is to restore the messages corresponding to each memory queue to the target data source, thereby restoring the entire memory data. The computer device can then continue to consume messages from the memory queue normally, thus avoiding the impact of the restart event and ensuring data security.

[0078] In summary, in this embodiment, multiple messages are first retrieved from a message queue; then, the multiple messages and their update times are synchronously stored in a first database, and the messages are added to multiple memory queues; and during the process of the target data source consuming messages from the multiple memory queues, the first head time and first tail time corresponding to each memory queue are stored in a second database. Here, the first head time includes the update time of the head message, and the first tail time includes the update time of the tail message. If a restart event occurs, the first head time and first tail time corresponding to each memory queue are retrieved from the second database; then, based on the first head time and first tail time corresponding to each memory queue, the messages corresponding to each memory queue are retrieved from the first database, and the messages corresponding to each memory queue are restored to the target data source. By using the first database to back up multiple messages and their update times, and using the second database to store the first head time and first tail time corresponding to each memory queue, the messages in the multiple memory queues can be restored using the data stored in the first and second databases in the event of a restart event. This improves data synchronization efficiency, prevents data loss, and thus enhances data security.

[0079] Based on the above data processing method, this application provides a data processing apparatus. Please refer to... Figure 6 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. The data processing device 600 can operate the following units:

[0080] The acquisition unit 601 is used to acquire multiple messages from a message queue, the message queue including messages read from a database log file;

[0081] The processing unit 602 is used to synchronously store multiple messages and the update time of each message in a first database, and to add multiple messages to multiple memory queues;

[0082] The processing unit 602 is further configured to store the first head time and the first tail time corresponding to each memory queue in the multiple memory queues into a second database during the process of the target data source consuming messages from multiple memory queues. The first head time includes the update time of the head message and the first tail time includes the update time of the tail message. The first database and the second database are used to restore the messages in the multiple memory queues to the target data source when a restart event occurs.

[0083] In one embodiment, the processing unit 602 is further configured to: if a restart event occurs, obtain the first head time and the first tail time corresponding to each memory queue from the second database; obtain the message corresponding to each memory queue from the first database based on the first head time and the first tail time corresponding to each memory queue; and restore the message corresponding to each memory queue to the target data source.

[0084] In another embodiment, when the processing unit 602 retrieves the message corresponding to each memory queue from the first database based on the first head time and the first tail time corresponding to each memory queue, it may specifically be used to: determine the target time interval corresponding to each memory queue based on the first head time and the first tail time corresponding to each memory queue; determine the message whose update time is within the target time interval from the messages stored in the first database; and use the message whose update time is within the target time interval as the message corresponding to each memory queue.

[0085] In another embodiment, when adding multiple messages to multiple memory queues, the processing unit 602 may specifically be used to: determine the target memory queue from the multiple memory queues based on the primary key identifier of any message among the multiple messages; and add any message to the target memory queue.

[0086] In another embodiment, when storing the first head time and the first tail time corresponding to each memory queue in the multiple memory queues into the second database, the processing unit 602 may specifically be used to: obtain the head message and the tail message of each memory queue in the multiple memory queues; determine the first head time corresponding to each memory queue based on the update time of the head message, and determine the first tail time corresponding to each memory queue based on the update time of the tail message; and store the first head time and the first tail time corresponding to each memory queue into the second database based on the queue identifier, head identifier, and tail identifier of each memory queue.

[0087] In another embodiment, when the processing unit 602 stores the first head time and the first tail time corresponding to each memory queue into the second database based on the queue identifier, head identifier, and tail identifier of each memory queue, it can specifically be used to: obtain the second head time corresponding to each memory queue stored in the second database based on the queue identifier and head identifier of each memory queue; if the first head time and the second head time corresponding to each memory queue are different, then store the first head time corresponding to each memory queue into the second database.

[0088] In another implementation, when processing unit 602 recovers the message corresponding to each memory queue from the target data source, it may specifically be used to: send the message corresponding to each memory queue to the target data source so that the target data source consumes the message corresponding to each memory queue; or add the message corresponding to each memory queue to each memory queue so that the target data source consumes the message from each memory queue.

[0089] According to another embodiment of this application, Figure 6 The data processing apparatus shown can be constructed by combining each unit individually or entirely into one or more other units, or one or more of the units can be further divided into multiple functionally smaller units. This achieves the same operation without affecting the technical effects of the embodiments of this application. The above-mentioned units are based on logical function division. In practical applications, the function of one unit can be implemented by multiple units, or the function of multiple units can be implemented by one unit. In other embodiments of this application, the data processing apparatus may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented collaboratively by multiple units.

[0090] According to another embodiment of this application, the following can be achieved by running on a general-purpose computing device, such as a computer, which includes processing elements and storage elements such as a central processing unit (CPU), random access memory (RAM), and read-only memory (ROM), a device capable of performing operations such as... Figure 2 or Figure 3 The computer program (including program code) for each step involved in the corresponding method shown, to construct such... Figure 6 The data processing apparatus shown herein, and the data processing method for implementing the embodiments of this application, are described. The computer program may be recorded on, for example, a computer-readable recording medium, loaded onto the aforementioned computing device via the computer-readable recording medium, and run therein.

[0091] In this embodiment, multiple messages are first retrieved from a message queue. Then, the multiple messages and their update times are synchronously stored in a first database, and the messages are added to multiple memory queues. During the process of the target data source consuming messages from the multiple memory queues, the first head time and first tail time corresponding to each memory queue are stored in a second database. Here, the first head time includes the update time of the head message, and the first tail time includes the update time of the tail message. Furthermore, the first and second databases are used to restore messages from the multiple memory queues to the target data source in the event of a restart. By using the first database to back up the multiple messages and their update times, and using the second database to store the first head time and first tail time corresponding to each memory queue, the messages in the multiple memory queues can be restored using the data stored in the first and second databases in the event of a restart. This improves data synchronization efficiency, prevents data loss, and thus enhances data security.

[0092] Based on the description of the above method and apparatus embodiments, this application also provides a computer device. Please refer to... Figure 7 The computer device 700 includes at least a processor 701, a communication interface 702, and a computer storage medium 703. The processor 701, communication interface 702, and computer storage medium 703 can be connected via a bus or other means. The computer storage medium 703 can be stored in the memory 704 of the computer device 700. The computer storage medium 703 is used to store computer programs, which include program instructions. The processor 701 is used to execute the program instructions stored in the computer storage medium 703. The processor 701 (or CPU (Central Processing Unit)) is the computing and control core of the computer device, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions to achieve a corresponding method flow or corresponding function.

[0093] In one embodiment, the processor 701 described in this application can be used to perform a series of data processing operations, specifically including: retrieving multiple messages from a message queue, the messages in which the message queue are read from a database log file; synchronously storing the multiple messages and the update time of each message in a first database, and adding the multiple messages to multiple memory queues; during the process of the target data source consuming messages from the multiple memory queues, storing the first head time and the first tail time corresponding to each memory queue in the multiple memory queues in a second database, the first head time including the update time of the head message and the first tail time including the update time of the tail message; the first database and the second database are used to restore the messages in the multiple memory queues to the target data source when a restart event occurs, etc.

[0094] This application embodiment also provides a computer storage medium (memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer storage medium provides storage space that stores the operating system of the computer device. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by the processor 701. These instructions can be one or more computer programs (including program code). It should be noted that the computer storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device; optionally, it can also be at least one computer storage medium located remotely from the aforementioned processor.

[0095] In one embodiment, a processor may load and execute one or more instructions stored in a computer storage medium to achieve the aforementioned... Figure 2 or Figure 3 The corresponding steps of the method in the data processing method embodiment shown; in specific implementation, one or more instructions in the computer storage medium are loaded by the processor 701 and executed as follows:

[0096] Retrieve multiple messages from a message queue, which contains messages read from a database log file;

[0097] Multiple messages and their update times are synchronously stored in the first database, and the messages are added to multiple memory queues.

[0098] During the process of the target data source consuming messages from multiple memory queues, the first head time and the first tail time corresponding to each memory queue are stored in the second database. The first head time includes the update time of the head message and the first tail time includes the update time of the tail message. The first database and the second database are used to restore the messages in the multiple memory queues to the target data source when a restart event occurs.

[0099] In one implementation, the one or more instructions may be loaded and executed by the processor: if a restart event occurs, the first head time and the first tail time corresponding to each memory queue are obtained from the second database; based on the first head time and the first tail time corresponding to each memory queue, the message corresponding to each memory queue is obtained from the first database; and the message corresponding to each memory queue is restored to the target data source.

[0100] In another implementation, when retrieving messages corresponding to each memory queue from the first database based on the first head time and the first tail time corresponding to each memory queue, the one or more instructions can be loaded and specifically executed by the processor: determining the target time interval corresponding to each memory queue based on the first head time and the first tail time corresponding to each memory queue; determining the messages whose update time is within the target time interval from the messages stored in the first database; and using the messages whose update time is within the target time interval as the messages corresponding to each memory queue.

[0101] In another implementation, when adding multiple messages to multiple memory queues, the one or more instructions can be loaded and executed by the processor: for any one of the multiple messages, determine the target memory queue from the multiple memory queues based on the primary key identifier of any one message; add any one message to the target memory queue.

[0102] In another implementation, when storing the first head time and first tail time corresponding to each memory queue in the multiple memory queues into the second database, the one or more instructions can be loaded and specifically executed by the processor: obtaining the head message and tail message of each memory queue in the multiple memory queues; determining the first head time corresponding to each memory queue based on the update time of the head message, and determining the first tail time corresponding to each memory queue based on the update time of the tail message; and storing the first head time and first tail time corresponding to each memory queue into the second database based on the queue identifier, head identifier, and tail identifier of each memory queue.

[0103] In another implementation, when storing the first head time and the first tail time corresponding to each memory queue in the second database based on the queue identifier, head identifier, and tail identifier of each memory queue, the one or more instructions can be loaded and specifically executed by the processor: based on the queue identifier and head identifier of each memory queue, obtain the second head time corresponding to each memory queue stored in the second database; if the first head time and the second head time corresponding to each memory queue are different, then store the first head time corresponding to each memory queue in the second database.

[0104] In another implementation, when recovering messages corresponding to each memory queue to the target data source, the one or more instructions can be loaded and executed by the processor: sending messages corresponding to each memory queue to the target data source so that the target data source consumes messages corresponding to each memory queue; or adding messages corresponding to each memory queue to each memory queue so that the target data source consumes messages from each memory queue.

[0105] In this embodiment, multiple messages are first retrieved from a message queue. Then, the multiple messages and their update times are synchronously stored in a first database, and the messages are added to multiple memory queues. During the process of the target data source consuming messages from the multiple memory queues, the first head time and first tail time corresponding to each memory queue are stored in a second database. Here, the first head time includes the update time of the head message, and the first tail time includes the update time of the tail message. Furthermore, the first and second databases are used to restore messages from the multiple memory queues to the target data source in the event of a restart. By using the first database to back up the multiple messages and their update times, and using the second database to store the first head time and first tail time corresponding to each memory queue, the messages in the multiple memory queues can be restored using the data stored in the first and second databases in the event of a restart. This improves data synchronization efficiency, prevents data loss, and thus enhances data security.

[0106] It should be noted that, according to one aspect of this application, a computer program product or computer program is also provided, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the aforementioned... Figure 2 or Figure 3The data processing method embodiments shown are provided in various optional ways. Furthermore, it should be understood that the above-disclosed embodiments are merely preferred embodiments of this application and should not be construed as limiting the scope of this application. Therefore, equivalent variations made in accordance with the claims of this application are still within the scope of this application.

Claims

1. A data processing method, characterized in that, include: Retrieve multiple messages from a message queue, wherein the messages in the message queue are read from a database log file; The multiple messages and their update time are synchronously stored in the first database, and the multiple messages are added to multiple memory queues; During the process of the target data source consuming messages from the multiple memory queues, the first head time and the first tail time corresponding to each memory queue are stored in the second database. The first head time includes the update time of the head message and the first tail time includes the update time of the tail message. If a restart event occurs, the first head time and the first tail time corresponding to each memory queue are obtained from the second database, and the target time interval corresponding to each memory queue is determined based on the first head time and the first tail time corresponding to each memory queue. The message whose update time is within the target time interval is determined from the messages stored in the first database, and the message whose update time is within the target time interval is used as the message corresponding to each memory queue. Recover the messages corresponding to each memory queue from the target data source.

2. The method according to claim 1, characterized in that, Adding the multiple messages to multiple memory queues includes: For any one of the multiple messages, the target memory queue is determined from the multiple memory queues based on the primary key identifier of the message; Add any of the messages to the target memory queue.

3. The method according to claim 1, characterized in that, The step of storing the first head time and the first tail time corresponding to each of the plurality of memory queues into the second database includes: Obtain the head message and tail message of each of the multiple memory queues; The first head time corresponding to each memory queue is determined based on the update time of the head message, and the first tail time corresponding to each memory queue is determined based on the update time of the tail message. Based on the queue identifier, head identifier, and tail identifier of each memory queue, the first head time and the first tail time corresponding to each memory queue are stored in the second database.

4. The method according to claim 3, characterized in that, The step of storing the first head time and first tail time corresponding to each memory queue into the second database based on the queue identifier, head identifier, and tail identifier of each memory queue includes: Based on the queue identifier and head identifier of each memory queue, obtain the second head time corresponding to each memory queue stored in the second database; If the first head time and the second head time corresponding to each memory queue are different, then the first head time corresponding to each memory queue is stored in the second database.

5. The method according to claim 1, characterized in that, The step of restoring the messages corresponding to each memory queue to the target data source includes: Send the message corresponding to each memory queue to the target data source so that the target data source consumes the message corresponding to each memory queue; Alternatively, the messages corresponding to each memory queue can be added to each memory queue so that the target data source consumes messages from each memory queue.

6. A data processing apparatus, characterized in that, The device includes: The acquisition unit is used to acquire multiple messages from a message queue, wherein the messages in the message queue are read from a database log file; The processing unit is used to synchronously store the multiple messages and the update time of each message in a first database, and to add the multiple messages to multiple memory queues; The processing unit is further configured to store the first head time and the first tail time corresponding to each memory queue in the plurality of memory queues into the second database during the process of the target data source consuming messages from the plurality of memory queues. The first head time includes the update time of the head message and the first tail time includes the update time of the tail message. The processing unit is further configured to, if a restart event occurs, obtain the first head time and the first tail time corresponding to each memory queue from the second database, and determine the target time interval corresponding to each memory queue based on the first head time and the first tail time corresponding to each memory queue; determine the message whose update time is within the target time interval from the messages stored in the first database, and use the message whose update time is within the target time interval as the message corresponding to each memory queue; and restore the message corresponding to each memory queue to the target data source.

7. A computer device, characterized in that, The computer device includes a memory and a processor, the memory storing a computer program that, when executed by the processor, causes the processor to perform the data processing method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more computer programs, which are adapted to be loaded by a processor and executed by the data processing method as described in any one of claims 1 to 5.

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a computer processor, it implements the data processing method as described in any one of claims 1 to 5.

Images