Rocket MQ-based real-name system data pushing method and apparatus, and electronic device

Abstract

The invention discloses a Rocket MQ-based real-name system data pushing method and apparatus, and an electronic device. The method comprises the steps of storing real-name information in a relational database; receiving data in the relational database, identifying the data and then sending the data to the Rocket MQ; and monitoring the Rocket MQ queue, starting to push the message to another target system until accumulated data in the memory reaches a preset threshold value, confirming the message in the Rocket MQ through a message confirmation mechanism if the pushing is successful, and pushing the message to a retry queue of the Rocket MQ for retry if the pushing is failed. According to the invention, the accuracy and timeliness of real-name system data pushing in logistics information can be improved, a pushing system with high reliability and high flexibility can be realized through unified pushing design, development cost is reduced when a new place is docked, and working efficiency is improved.

Description

Technical Field

[0001] The present invention belongs to the field of logistics, and in particular relates to a method, a device and an electronic device for pushing real-name data based on RocketMQ. Background Art

[0002] With the rapid development of express delivery business, the state has implemented real-name management for express delivery business and e-commerce business, requiring users to provide real identity information when sending and receiving express delivery. This policy aims to ensure logistics safety and prevent illegal and criminal activities, but it also increases the complexity of information management for express delivery companies and e-commerce platforms. In order to meet the requirements of the real-name system, express delivery companies and e-commerce platforms need to upgrade their business systems to achieve fast and accurate information connection. Enterprises need a practical method and solution for real-name data push. Based on this, this solution is specially proposed. Summary of the invention

[0003] Purpose of the invention: In order to overcome the deficiencies in the prior art, the present invention provides a method, device and electronic device for real-name data push based on RocketMQ, which can complete the real-name data push of logistics information.

[0004] Technical solution: To achieve the above purpose, the technical solution of the present invention is as follows:

[0005] In a first aspect, a method for real-name data push based on RocketMQ comprises the following steps:

[0006] Step a. Save the real-name information into a relational database;

[0007] Step b. Receive data from the relational database, and send the data to RocketMQ after marking it;

[0008] Step c. Monitor the RocketMQ queue until the accumulated data in the memory reaches the preset threshold and then start pushing to another target system. If the push is successful, the message in RocketMQ is confirmed through the message confirmation mechanism. If the push fails, it is pushed to the RocketMQ retry queue for retry.

[0009] Furthermore, before step a, data collection and data cleaning are also included, collecting real-name information from various sources, and performing data verification and cleaning.

[0010] Furthermore, the source of the real-name information includes at least one of user registration, a third-party data interface, and an e-commerce platform.

[0011] Furthermore, the real-name information includes at least one or more of the user's mailing and receiving address, user name information, ID card information and telephone number.

[0012] Furthermore, in step b, receiving data from a relational database and sending the data to RocketMQ after marking the data includes:

[0013] Data extraction: extract the data to be pushed from the relational database quantitatively or in real time;

[0014] Data identification: before sending data to RocketMQ, add necessary identification to each piece of data to facilitate subsequent data tracking and processing;

[0015] Message sending: Use RocketMQ client API to encapsulate the identified data into a message and send it to the specified RocketMQ topic.

[0016] Furthermore, the identification includes adding any one or more of a timestamp, address information and a unique ID to each piece of data.

[0017] Furthermore, in step c, monitoring the RocketMQ queue includes:

[0018] Configure RocketMQ consumers to listen to messages in the topic. The consumers are configured in pull mode or push mode.

[0019] Furthermore, the preset threshold includes a preset quantity threshold or a time threshold.

[0020] In a second aspect, a device for real-name data push based on RocketMQ is provided, comprising:

[0021] A cleaning module, used to clean the acquired real-name information data;

[0022] The message producer module is used to produce messages and interact with the push module;

[0023] The push module receives messages from the message producer module, generates several message queues, and processes message distribution and stores message data according to client requests;

[0024] A message consumer module, used to interact with the push module and configured to consume at least one message queue through multiple consumer threads;

[0025] The data monitoring module is used to monitor the cleaning module and the pushing module.

[0026] According to a third aspect, an electronic device comprises a memory and a processor, wherein the memory stores program instructions, and when the processor runs the program instructions, the steps of any one of the methods described in the first aspect are executed.

[0027] Beneficial effects: The present invention can improve the accuracy and timeliness of real-name data push in logistics information. Through unified push design, a push system with high reliability and strong flexibility can be realized, which reduces development costs when new places are connected and improves work efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Attached Figure 1 It is a schematic diagram of the overall process of the invention. DETAILED DESCRIPTION

[0029] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0030] The terms used in the present disclosure are for the purpose of describing specific embodiments only and are not intended to limit the present disclosure. The singular forms "a", "said" and "the" used in the present disclosure and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise.

[0031] The specific implementation of the present invention is further described in detail below in conjunction with the drawings and examples.

[0032] As attached Figure 1 As shown, in the first aspect, a method for real-name data push based on RocketMQ includes the following steps:

[0033] Step a. Save the real-name information to a relational database; wherein the relational database is any one of Oracle Database, SQL Server, and Access databases. In the embodiment of this solution, Oracle Database is used. Oracle database system is a popular relational database management system and the most widely used database management system. As a general database system, it has complete data management functions; the system is portable, easy to use, and powerful, and is suitable for all kinds of large, medium, and small microcomputer environments. It is a highly efficient, reliable, and high-throughput database solution.

[0034] Step b. Receive data from the relational database, and send the data to RocketMQ after marking it;

[0035] Accept Oracle GoldenGate data and send it to RocketMQ after tagging. MQ (Message Queue) is a middleware that provides message queue service Q, also known as message middleware. It is a software system that provides APIs for the entire process of massive message production, storage, and consumption.

[0036] RocketMQ significantly improves the efficiency and throughput of message storage and reduces CPU and memory overhead by adopting sequential disk writing, zero-copy technology and batch processing mechanism.

[0037] High availability: RocketMQ ensures high availability through master-slave replication, failover, and load balancing. The master Broker is responsible for writing, and the slave Broker is responsible for reading. When the master Broker fails, the slave Broker can take over the service to ensure the continuity of the message service.

[0038] High reliability: RocketMQ ensures the reliability of messages through mechanisms such as message persistence, automatic retry mechanism, and dead letter queue. Even in the event of a system crash, messages will not be lost.

[0039] Flexible scalability: RocketMQ's architectural design supports horizontal expansion and dynamic configuration management. It can cope with the growing volume of messages by adding nodes, improving the system's flexibility and operation and maintenance efficiency.

[0040] Rich message model: RocketMQ supports publish / subscribe and point-to-point message models, supports both pull and push message modes, provides the ability to accumulate billions of messages, and supports multiple message protocols and distributed high-availability deployment architecture.

[0041] Step c. Monitor the RocketMQ queue until the accumulated data in the memory reaches the preset threshold and then start pushing to another target system. If the push is successful, the message in RocketMQ is confirmed through the message confirmation mechanism. If the push fails, it is pushed to the RocketMQ retry queue for retry.

[0042] Among them, the preset threshold is to extract the data to be pushed from the relational database quantitatively or in real time; RocketMQ can improve the efficiency and throughput of message storage, and can push messages according to the preset quantitative or timing, thereby greatly reducing the possibility of failure of a single message in the push structure, and the message confirmation mechanism can prevent the failure of message push. The present invention can improve the accuracy and timeliness of real-name data push in logistics information, and through a unified push design, a push system with high reliability and strong flexibility can be realized, which reduces development costs when new places are connected and improves work efficiency.

[0043] In some specific embodiments, before step a, data collection and data cleaning are also included, collecting real-name information from various sources, and performing data verification and cleaning. The source of the real-name information includes at least one of user registration, third-party data interface, and e-commerce platform.

[0044] In some specific embodiments, the real-name information includes at least one or more of the user's mailing address, user name information, ID card information, and telephone number. In addition, in this process, encryption is required when storing sensitive data.

[0045] In some specific embodiments, in step b, receiving data from a relational database and sending the data to RocketMQ after marking the data includes:

[0046] Data extraction: extract the data to be pushed from the relational database quantitatively or in real time;

[0047] Data identification: Before sending data to RocketMQ, add necessary identification to each piece of data to facilitate subsequent data tracking and processing; the identification includes adding any one or more of timestamp, address information and unique ID to each piece of data.

[0048] Message sending: Use RocketMQ client API to encapsulate the identified data into a message and send it to the specified RocketMQ topic.

[0049] In step c, monitoring the RocketMQ queue includes:

[0050] Configure RocketMQ consumers to listen to messages in the topic. The consumers are configured in pull mode or push mode. Pull information or actively push information according to user needs.

[0051] The preset threshold includes a preset quantity threshold or a time threshold. By quantitatively or timing the same push message, the probability of failure or leakage of a single message can be reduced, and security and accuracy can be increased.

[0052] In a second aspect, a device for real-name data push based on RocketMQ is provided, comprising:

[0053] A cleaning module, used to clean the acquired real-name information data;

[0054] The message producer module is used to produce messages and interact with the push module;

[0055] The push module receives messages from the message producer module, generates several message queues, and processes message distribution and stores message data according to client requests;

[0056] A message consumer module, used to interact with the push module and configured to consume at least one message queue through multiple consumer threads;

[0057] The data monitoring module is used to monitor the cleaning module and the pushing module.

[0058] According to a third aspect, an electronic device comprises a memory and a processor, wherein the memory stores program instructions, and when the processor runs the program instructions, the steps of any one of the methods described in the first aspect are executed.

[0059] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information.

[0060] In the description of the present invention, it is necessary to understand that the terms "upper", "lower", "front", "back", "left", "right", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship are based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0061] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for real-name data push based on RocketMQ, characterized in that: The following steps are involved: Step a. Save the real-name information into a relational database; Step b. Receive data from the relational database, and send the data to RocketMQ after marking it; Step c. Monitor the RocketMQ queue until the accumulated data in the memory reaches the preset threshold and then start pushing to another target system. If the push is successful, the message in RocketMQ is confirmed through the message confirmation mechanism. If the push fails, it is pushed to the RocketMQ retry queue for retry.

2. A method for pushing data based on RocketMQ real-name system according to claim 1, characterized in that: Before step a, data collection and data cleaning are also included, which involves collecting real-name information from various sources and performing data verification and cleaning.

3. A method for pushing data based on RocketMQ real-name system according to claim 1, characterized in that: The source of real-name information includes at least one of user registration, third-party data interface, and e-commerce platform.

4. A method for pushing data based on RocketMQ real-name system according to claim 1, characterized in that: The real-name information includes at least one or more of the user's delivery address, user name information, ID card information and telephone number.

5. The method for pushing data based on RocketMQ real-name system according to claim 1, characterized in that: In step b, receiving data from a relational database and sending the data to RocketMQ after marking the data includes: Data extraction: extract the data to be pushed from the relational database quantitatively or in real time; Data identification: before sending data to RocketMQ, add necessary identification to each piece of data to facilitate subsequent data tracking and processing; Message sending: Use RocketMQ client API to encapsulate the identified data into a message and send it to the specified RocketMQ topic.

6. A method for pushing data based on RocketMQ real-name system according to claim 5, characterized in that: The identification includes adding any one or more of a timestamp, address information, and a unique ID to each piece of data.

7. A method for pushing data based on RocketMQ real-name system according to claim 1, characterized in that: In step c, monitoring the RocketMQ queue includes: Configure RocketMQ consumers to listen to messages in the topic. The consumers are configured in pull mode or push mode.

8. The method for pushing data based on RocketMQ real-name system according to claim 1, characterized in that: The preset threshold includes a preset quantity threshold or a time threshold.

9. A device for real-name data push based on RocketMQ, characterized in that: include: A cleaning module, used to clean the acquired real-name information data; The message producer module is used to produce messages and interact with the push module; The push module receives messages from the message producer module, generates several message queues, and processes message distribution and stores message data according to client requests; A message consumer module, used to interact with the push module and configured to consume at least one message queue through multiple consumer threads; The data monitoring module is used to monitor the cleaning module and the pushing module.

10. An electronic device, comprising a memory and a processor, wherein the memory stores program instructions, and when the processor runs the program instructions, the steps of the method according to any one of claims 1 to 8 are executed.

Images