Customer list synchronization method, device, and processor

Abstract

The present application discloses a method, device, and processor for synchronizing a customer list. The method relates to the field of financial technology and includes: synchronizing the incremental list of customers in at least one source list database to a distributed message queue and parsing the list through a data replication component to obtain a first target list; writing the first target list to an online queue through an online queue write service; using a queue consumption process to determine whether the timestamp of the first target list is greater than the timestamp in the target list database; if the timestamp of the first target list is greater than the timestamp in the target list database, synchronizing the first target list in the online queue to the target list database. This application solves the problem of poor timeliness of customer list synchronization technology in related technologies.

Description

Technical Field

[0001] The present application relates to the field of financial technology, and specifically, to a method, device, and processor for synchronizing a customer list. Background Art

[0002] Existing customer risk list synchronization methods have a long cycle. The source list database periodically exports list files and then synchronizes them in batches to the risk list database. This prevents banks from accessing updated lists within the same day, reducing customer efficiency in bank transactions. Furthermore, the lists in the risk list database depend on the order in which they are loaded. If the source list database fails to export and synchronize the lists in chronological order, this will directly affect the status of the lists in the risk list database, causing the bank to read incorrect list information. Furthermore, queries based on the storage structure of the existing customer risk list database will result in multiple queries for the same customer traversing the customer risk list database multiple times, resulting in low query efficiency.

[0003] Currently, no effective solution has been proposed to address the problem of poor timeliness of customer list synchronization technology in related technologies. Summary of the Invention

[0004] The main purpose of this application is to provide a customer list synchronization method, device, and processor to solve the problem of poor timeliness of customer list synchronization technology in related technologies.

[0005] To achieve the above objectives, according to one aspect of the present application, a method for synchronizing a customer list is provided. The method comprises: synchronizing, through a data replication component, an incremental list of customers in at least one source list database to a distributed message queue and parsing the list to obtain a first target list; writing the first target list to an online queue through an online queue write service; utilizing a queue consumer process to determine whether the timestamp of the first target list is greater than the timestamp in the target list database; and synchronizing the first target list in the online queue to the target list database if the timestamp of the first target list is greater than the timestamp in the target list database.

[0006] Furthermore, before synchronizing the incremental list of customers in at least one source list database to a distributed message queue and parsing it through a data replication component to obtain a first target list, the method also includes: if the incremental list of customers in the source list database has been successfully updated, synchronizing the certificate information and list type of the incremental list to the decontrol cache, and setting the effective time of the incremental list in the decontrol cache.

[0007] Furthermore, before synchronizing the incremental list of customers in at least one source list database to the distributed message queue and parsing it through the data replication component to obtain the first target list, the method also includes: using a batch preloading method to process the list of customers already existing in at least one source list database to obtain a second target list; writing the second target list into the batch queue through a batch queue writing process; using a queue consumption process to determine whether the timestamp of the second target list is greater than the timestamp in the target list database; if the timestamp of the second target list is greater than the timestamp in the target list database, synchronizing the second target list to the target list database.

[0008] Furthermore, writing the first target list into the online queue through the online queue writing service includes: mapping the certificate information in the first target list into a hash value of fixed length; dividing the last digit of the hash value by the number of partitions of the online queue and taking the remainder to obtain a first partition number, wherein the first partition number is the partition number for writing the first target list into the online queue; and writing the first target list into the online queue according to the first partition number.

[0009] Furthermore, if the timestamp of the first target list is greater than the timestamp in the target list database, synchronizing the first target list in the online queue to the target list database includes: storing the first target list in the target list database using a key-value storage style, wherein the key-value storage style includes at least keywords and values, wherein the keyword is the customer's certificate information, and the value is various list information integrated into a JSON string.

[0010] Furthermore, if the incremental list of the customer in the source list database has been successfully updated, the certificate information and list type of the incremental list are synchronized to the decontrol cache library, and the effective time of the incremental list in the decontrol cache library is set. The method also includes: if a query instruction for the list to be queried is received, querying whether the list to be queried exists in the decontrol cache library; if the list to be queried exists in the decontrol cache library, and the query time is within the effective time of the list to be queried in the decontrol cache library, obtaining a first query result; if the list to be queried does not exist in the decontrol cache library, searching from the target list database to obtain a second query result.

[0011] Furthermore, writing the second target list into the batch queue through the batch queue writing process includes: mapping the certificate information in the second target list into a hash value of fixed length; dividing the last digit of the hash value by the number of partitions of the batch queue and taking the remainder to obtain a second partition number, wherein the second partition number is the partition number for writing the second target list into the batch queue; and writing the second target list into the batch queue according to the second partition number.

[0012] To achieve the above-mentioned objectives, according to another aspect of the present application, a device for synchronizing a customer list is provided. The device comprises: a first parsing unit, configured to synchronize an incremental customer list in at least one source list database to a distributed message queue and parse the list through a data replication component to obtain a first target list; a first writing unit, configured to write the first target list into an online queue through an online queue writing service; a first determining unit, configured to determine, using a queue consumption process, whether the timestamp of the first target list is greater than the timestamp in the target list database; and a first synchronizing unit, configured to synchronize the first target list in the online queue to the target list database if the timestamp of the first target list is greater than the timestamp in the target list database.

[0013] Furthermore, the device also includes: a setting unit, which is used to synchronize the incremental list of customers in at least one source list database to the distributed message queue and parse it through the data replication component, and before obtaining the first target list, if the incremental list of customers in the source list database has been successfully updated, then synchronize the certificate information and list type of the incremental list to the decontrol cache library, and set the effective time of the incremental list in the decontrol cache library.

[0014] Furthermore, the device also includes: a second parsing unit, which is used to synchronize the incremental list of customers in at least one source list database to the distributed message queue and parse it through the data replication component, and before obtaining the first target list, use a batch preloading method to process the list of customers already existing in at least one source list database to obtain a second target list; a second writing unit, which is used to write the second target list into the batch queue through a batch queue writing process; a second judgment unit, which is used to use a queue consumption process to judge whether the timestamp of the second target list is greater than the timestamp in the target list database; and a second synchronization unit, which is used to synchronize the second target list to the target list database if the timestamp of the second target list is greater than the timestamp in the target list database.

[0015] Furthermore, the first writing unit includes: a mapping subunit, used to map the certificate information in the first target list into a hash value of a fixed length; a calculation subunit, used to obtain a first partition number by dividing the last digit of the hash value by the number of partitions of the online queue and taking the remainder, wherein the first partition number is the partition number for writing the first target list into the online queue; and a writing subunit, used to write the first target list into the online queue according to the first partition number.

[0016] Furthermore, the first synchronization unit includes: a storage unit, used to store the first target list in the target list database using a key-value storage style, wherein the key-value storage style includes at least keywords and values, wherein the keywords are the customer's certificate information, and the values ​​are various list information integrated into a JSON string.

[0017] Furthermore, the setting unit also includes: a first query unit, which is used to synchronize the certificate information and list type of the incremental list to the de-control cache library if the incremental list of the customer in the source list database has been successfully updated, and set the effective time of the incremental list in the de-control cache library. If a query instruction for the list to be queried is received, the de-control cache library is queried whether the list to be queried exists; a second query unit, which is used to obtain a first query result if the list to be queried exists in the de-control cache library and the query time is within the effective time of the list to be queried in the de-control cache library; a third query unit, which is used to search from the target list database to obtain a second query result if the list to be queried does not exist in the de-control cache library.

[0018] Furthermore, the second writing unit includes: a mapping subunit, used to write the second target list into the batch queue through a batch queue writing process, by mapping the certificate information in the second target list into a hash value of a fixed length; a calculation subunit, used to divide the last digit of the hash value by the number of partitions of the batch queue and take the remainder to obtain a second partition number, wherein the second partition number is the partition number for writing the second target list into the batch queue; a writing subunit, used to write the second target list into the batch queue according to the second partition number.

[0019] Through the present application, the following steps are adopted: through a data replication component, the incremental list of customers in at least one source list database is synchronized to a distributed message queue and parsed to obtain a first target list; through an online queue write service, the first target list is written into an online queue; a queue consumption process is used to determine whether the timestamp of the first target list is greater than the timestamp in the target list database; if the timestamp of the first target list is greater than the timestamp in the target list database, the first target list in the online queue is synchronized to the target list database, thereby solving the problem of poor timeliness of customer list synchronization technology in related technologies, and thus achieving the technical effect of improving the timeliness of customer list synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings, which constitute part of this application, are intended to provide a further understanding of this application. The exemplary embodiments and descriptions of this application are intended to explain this application and do not constitute an improper limitation on this application. In the accompanying drawings:

[0021] Figure 1 is a flowchart of a method for synchronizing a customer list according to an embodiment of the present application;

[0022] Figure 2 This is a schematic diagram of an optional customer list synchronization method provided in an embodiment of the present application. Figure 1 ;

[0023] Figure 3 This is a schematic diagram of an optional customer list synchronization method provided in an embodiment of the present application. Figure 2 ;

[0024] Figure 4 This is a schematic diagram of an optional customer list synchronization method provided in an embodiment of the present application. Figure 3 ;

[0025] Figure 5 is a schematic diagram of a customer list synchronization device provided according to an embodiment of the present application;

[0026] Figure 6 is a schematic diagram of an electronic device provided according to an embodiment of the present application. DETAILED DESCRIPTION

[0027] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in this application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0028] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] For ease of description, some nouns or terms involved in the embodiments of the present application are explained below:

[0031] Distributed message queue: This system uses a pluggable model and integrates ActiveMQ and Kafka, two relatively mature open source message middleware products in the financial technology field. It also provides automatic installation and deployment, one-click operation and maintenance, and message service monitoring functions to achieve platform-based and automated message services.

[0032] Data replication component: This component implements near-real-time data replication from host DB2, platform Oracle, and MySQL to the open platform, enabling cross-application data sharing of the host and platform "dual-core" databases. Replication links include DB2 to Oracle replication, DB2 to Kafka replication, Oracle to Oracle replication, Oracle to Kafka replication, and MySQL to Kafka replication.

[0033] Distributed Cache: Distributed cache offers high performance, dynamic scalability, high availability, and ease of use, serving primarily as an effective complement to databases. Distributed cache alleviates database pressure, reduces system response time, and improves user experience. It also supports the transition from traditional monolithic IT architectures to distributed architectures, meeting the demands of high-volume, high-concurrency, and rapidly growing business access scenarios.

[0034] Table Storage Service: This integrates multiple HBase-related services, supporting read and write access to massive amounts of data by primary key. It also implements bidirectional data replication across two campuses through message-based middleware, achieving high availability across both campuses. By achieving resource isolation, it effectively improves cluster resource utilization, enhances data sharing between applications, and reduces development and maintenance challenges.

[0035] The present invention will be described below in conjunction with preferred implementation steps. Figure 1 Flowchart of the method for synchronizing a customer list according to an embodiment of the present application. Figure 1 As shown, the method includes the following steps:

[0036] Step S101: Synchronize the incremental list of customers in at least one source list database to the distributed message queue and parse it through the data replication component to obtain a first target list.

[0037] For example, when a bank teller initiates a list deregistration operation, an incremental list is received in the source list database, which is then synchronized to the distributed message queue and parsed. After the parsing is complete, the first target list is obtained.

[0038] In this embodiment of the present application, when a customer list is updated in one or more source list databases, a data replication component copies the incremental list from at least one source list database to a distributed message queue. Simultaneously, the incremental list is parsed to obtain a first target list. Processing incremental lists from multiple source list databases horizontally expands data sources, improves data processing efficiency, and further enhances data processing efficiency.

[0039] Step S102 , writing the first target list into the online queue via the online queue writing service.

[0040] The parsed incremental list is processed, including extracting the detailed information required by the risk detail database and incorporating it into the risk detail database. The incremental list is then written to the online queue by calculating the number of partitions for storing the incremental list. Lists for the same customer are placed in a fixed position in the online queue, facilitating the integration of all relevant list information for the same customer and storing them in adjacent positions within the online queue.

[0041] For example, the first target list contains three entries for Zhang San and four entries for Li Si. These seven entries are distributed across the first target list. Using the online queue write service, Zhang San's three entries are placed in positions 1 through 3 of the online queue, while Li Si's four entries are placed in positions 5 through 8. By consolidating the different entries for the same customer and storing them at the customer level, query efficiency is improved.

[0042] Step S103: Using the queue consumption process, determine whether the timestamp of the first target list is greater than the timestamp in the target list database.

[0043] In order to ensure that the order of list loading does not affect the list status in the target list database, a timestamp judgment mechanism is introduced to judge whether the timestamp of the first target list is greater than the timestamp in the target list database, so as to determine whether the first target list needs to be synchronized with the target list database.

[0044] Step S104: If the timestamp of the first target list is greater than the timestamp in the target list database, the first target list in the online queue is synchronized to the target list database.

[0045] For example, the timestamp of Zhang San's list in the online queue is 10:00 on January 1, and the timestamp of Zhang San's list in the target list database is 9:00 on January 1. Based on the timestamp, it can be determined that Zhang San's list at 10:00 on January 1 is updated data and needs to be written into the target list database; if the timestamp of Zhang San's list in the target list database is 11:00 on January 1, it can be determined based on the timestamp that January 10:00 is not the latest data of Zhang San's list. Therefore, there is no need to write Zhang San's list with a timestamp of 10:00 on January 1 into the target list database.

[0046] From the above content, it can be seen that if the timestamp of the first target list is greater than the timestamp in the target list database, the first target list will be synchronized to the target list database; if the timestamp of the first target list is not greater than the timestamp in the target list database, it means that the latest information of the customer is already in the target list database. At this time, the customer cannot use the first target list which is not the latest information to replace the latest information in the target list database. Therefore, there is no need to synchronize the first target list to the target list database.

[0047] In summary, the customer list synchronization method provided in the embodiment of the present application uses a data replication component to synchronize the incremental list of customers in at least one source list database to a distributed message queue and parse it to obtain a first target list; uses an online queue write service to write the first target list to an online queue; uses a first queue consumer process to parse the first target list in the online queue to obtain a second target list; if the timestamp of the second target list is greater than the timestamp in the target list database, the second target list is synchronized to the target list database, thereby solving the problem of poor timeliness of customer list synchronization technology in related technologies. By introducing a timestamp judgment mechanism and using an online queue to promptly process the incremental list, the incremental list can be queried shortly after the list is updated, thereby shortening the list synchronization cycle and improving the timeliness of customer list synchronization.

[0048] Optionally, in the customer list synchronization method provided in the embodiment of the present application, before synchronizing the incremental list of customers in at least one source list database to a distributed message queue and parsing it through a data replication component to obtain a first target list, the method also includes: if the incremental list of customers in the source list database has been successfully updated, synchronizing the certificate information and list type of the incremental list to the decontrol cache library, and setting the effective time of the incremental list in the decontrol cache library.

[0049] The deregistration cache is used to store updated incremental rosters from the source roster database. A validity period is set for each incremental roster in the deregistration cache. Within the validity period, the deregistration cache stores the incremental roster. If the validity period is exceeded, the incremental roster is removed from the deregistration cache.

[0050] In this application, a hash structure is designed to store the list in the deregulation cache. This storage structure uses the ID number as the primary key, associates the list type number, and sets the validity period. The specific structure is as follows:

[0051] key: document type#document number;

[0052] field: list type number;

[0053] value:Effective flag#Generate timestamp.

[0054] For example, when a bank counter initiates a list decontrol operation, and an incremental list is received in the source list database and successfully synchronized to the target list database, the risk list desynchronization service will be called to synchronize the document information and list type in the incremental list directly to the decontrol cache library, store it in the above-mentioned Hash structure, and set the validity period.

[0055] This solution synchronizes updated incremental lists from the source list database to the deregulation cache, ensuring the timeliness and accuracy of data queries. Furthermore, if a query request is received for a customer's list while the incremental list is being synchronized to the target list database, the deregulation cache can be used to retrieve the updated list status for that customer. Furthermore, storing incremental lists in the deregulation cache as a hash structure improves the efficiency of querying list information.

[0056] Optionally, in the customer list synchronization method provided in the embodiment of the present application, before synchronizing the incremental list of customers in at least one source list database to the distributed message queue and parsing it through the data replication component to obtain the first target list, the method also includes: using a batch preloading method to process the list of customers already existing in at least one source list database to obtain a second target list; writing the second target list into the batch queue through a batch queue writing process; using a queue consumption process to determine whether the timestamp of the second target list is greater than the timestamp in the target list database; if the timestamp of the second target list is greater than the timestamp in the target list database, synchronizing the second target list to the target list database.

[0057] In this embodiment of the present application, for existing lists in the source list database, a batch pre-load method is used at a certain periodic interval to export the existing data. The data replication component then copies the existing data to the data lake and parses it to obtain the second target list. Furthermore, the batch queue write process writes the second target list to the batch queue.

[0058] In this application, if the batch queue stores the first target list and the online queue stores the second target list, the queue consumption process will prioritize the second target list in the online queue for consumption.

[0059] Through the above solution, the existing lists in the source list database can be synchronized to the target list database, and the incremental lists in the source list database can be synchronized to the target list database in a timely manner, ensuring the integrity and real-time nature of the customer list data.

[0060] Optionally, in the customer list synchronization method provided in the embodiment of the present application, writing the first target list into the online queue through the online queue writing service includes: mapping the certificate information in the first target list into a hash value of a fixed length; dividing the last digit of the hash value by the number of partitions of the online queue and taking the remainder to obtain the first partition number, wherein the first partition number is the partition number for writing the first target list into the online queue; and writing the first target list into the online queue according to the first partition number.

[0061] When an incremental list is written to the online queue, the number of partitions to which the incremental list is written is calculated. First, the ID information in the incremental list is mapped to a fixed-length hash value. The remainder of the hash value is divided by the total number of partitions in the online queue. This remainder is the number of partitions in the current incremental list.

[0062] In this application, since the online queue is a multi-partition queue, if there is no restriction on the storage location of the list information, different list information of the same customer will be randomly placed at any location in the online queue, and then when synchronized to the target list database, it may cause different list information of the same customer to be synchronized to the target list database for a long time. When there is a query instruction, it will result in the inability to obtain all updated information of the same customer.

[0063] For example, Zhang San has two incremental lists, A and B, that need to be synchronized to the target list database. However, these two incremental lists are located in different locations in the online queue. The queue consumer process first synchronizes list A to the target list database. Then, the queue consumer process continues processing the remaining incremental lists in the online queue. After a 10-minute interval, the queue consumer process synchronizes list B to the target list database. If a query request for Zhang San's list information is received within the 10-minute interval, only list A will be found in the target list database; Zhang San's list B will not be found.

[0064] Through the above solution, the incremental list information of the same customer is integrated together, and the list is stored at the customer granularity. After the customer's incremental list is synchronized to the target list database, the integrity of the customer information is guaranteed, thereby improving the efficiency of the query and ensuring the timeliness and accuracy of the customer information.

[0065] Optionally, in the customer list synchronization method provided in the embodiment of the present application, if the timestamp of the first target list is greater than the timestamp in the target list database, synchronizing the first target list in the online queue to the target list database includes: storing the first target list in the target list database using a key-value storage style, wherein the key-value storage style includes at least keywords and values, wherein the keywords are the customer's certificate information, and the values ​​are various list information integrated into a JSON string.

[0066] In this application, the key-value storage style includes: using the certificate information in the list information as the key, and putting all the lists associated with the certificate information together as the value. The key-value storage style is as follows:

[0067] key: document type#document number;

[0068] value:[{list type 1: effective date ← tag information ← effective flag ← generation timestamp};

[0069] {List type 2: effective date ← tag information ← effective flag ← generation timestamp};

[0070] {List type 2: effective date ← tag information ← effective flag ← generation timestamp}.

[0071] In this embodiment, the target list database is divided into two databases: a distributed cache database and a table storage service database. Both databases store the same list. When searching for information about a target risk list in the target risk list database, the default query is the distributed cache database. If the distributed cache database experiences a read / write anomaly or other failure, the table storage service database is used for the query.

[0072] By using this key-value storage style to store messages, multiple lists for the same customer are consolidated, making it possible to more quickly query the same customer's list information. This also avoids the need to repeatedly traverse the target list database when querying multiple lists for the same customer, saving query time and improving query efficiency. Furthermore, using two data sources to store lists improves the fault tolerance of the entire system, thereby achieving a more reliable service.

[0073] Optionally, in the customer list synchronization method provided in the embodiment of the present application, if the incremental list of customers in the source list database has been successfully updated, the certificate information and list type of the incremental list are synchronized to the de-control cache library, and the effective time of the incremental list in the de-control cache library is set. The method also includes: if a query instruction for the list to be queried is received, the de-control cache library is queried whether there is a list to be queried; if there is a list to be queried in the de-control cache library, and the query time is within the effective time of the list to be queried in the de-control cache library, a first query result is obtained; if there is no list to be queried in the de-control cache library, the target list database is searched to obtain a second query result.

[0074] In this application, when a computer system receives a request to query a list, it first searches the deregulation cache for the pending list. If the pending list exists in the deregulation cache and the query time is within the valid time for the pending list, the query results are returned based on the list in the deregulation cache. If the pending list does not exist in the deregulation cache, or if the pending list exists in the deregulation cache but the query time is outside the valid time for the pending list, the query continues in the target risk database.

[0075] For example, after Zhang San's list is synchronized to the target list database at 12:00, it is also synchronized to the deregistration cache, and its validity period is set to 5 minutes, from 12:00 to 12:05. If the computer system receives Zhang San's query instruction at 12:01, it first searches the deregistration cache for Zhang San's list information. Because 12:01 is within the validity period for Zhang San's updated list, Zhang San's updated list is available in the deregistration cache. The query result is then returned based on Zhang San's updated list in the deregistration cache. If the computer system receives Zhang San's list query instruction at 12:11, it also first searches the deregistration cache for Zhang San's list. However, because 12:11 is not within the validity period for updated list information, Zhang San's updated list is not available in the deregistration cache. The query result is then returned by continuing to search the target risk list database for Zhang San's list information.

[0076] To ensure data consistency between the upstream inlet table and the distributed message queue, we also pull logs from the upstream inlet table and the distributed message queue daily for comparison to prevent partial list information from being unprocessed. We also periodically scan the distributed message queue's work logs, and if any anomalies are detected, we notify you immediately.

[0077] Through the above solution, when querying, the de-control cache library can be accessed first, achieving the purpose of faster querying of customer lists; at the same time, it also avoids the situation where the latest customer list information cannot be returned when the list to be queried is written to the target database, thereby improving the efficiency, accuracy and real-time performance of the query.

[0078] Optionally, in the customer list synchronization method provided in an embodiment of the present application, writing the second target list into the batch queue through the batch queue writing process includes: mapping the certificate information in the second target list into a hash value of a fixed length; dividing the last digit of the hash value by the number of partitions of the batch queue and taking the remainder to obtain a second partition number, wherein the second partition number is the partition number for writing the second target list into the batch queue; and writing the second target list into the batch queue according to the second partition number.

[0079] In this application, the batch queue, like the online queue, is a multi-partition queue. Therefore, when the second target list is added to the batch queue, the number of partitions to which the second target list is written is calculated. This is done by mapping the document information in the second target list to a fixed-length hash value. The remainder of the hash value is divided by the total number of partitions in the batch queue. This remainder is the number of partitions in which the second target list is currently written to the batch queue.

[0080] Through the above solution, the list information of the same customer is integrated together, making the data arrangement more regular. At the same time, the list information is stored at the customer granularity. When the customer's list information is synchronized to the target list database, the integrity of the customer information is guaranteed, which in turn ensures that the query is more efficient and the information of the same customer is more accurate.

[0081] In summary, the customer list synchronization method provided in the embodiment of the present application uses a data replication component to synchronize the incremental list of customers in at least one source list database to a distributed message queue and parse it to obtain a first target list; uses an online queue write service to write the first target list to an online queue; uses a first queue consumer process to parse the first target list in the online queue to obtain a second target list; if the timestamp of the second target list is greater than the timestamp in the target list database, the second target list is synchronized to the target list database, thereby solving the problem of poor timeliness of customer list synchronization technology in related technologies. By introducing a timestamp judgment mechanism and using an online queue to promptly process the incremental list, the incremental list can be queried shortly after the list is updated, thereby shortening the list synchronization cycle and improving the timeliness of customer list synchronization.

[0082] Optional embodiments of this application, such as Figure 2 As shown, the lists in the source list database are synchronized to the risk list database (i.e. the target list database mentioned above), the risk details database and the control release cache library; when a list control release instruction is received, the incremental list is synchronized to the source list database and the control release cache library; when a list query instruction is received, the list is queried in the control release cache library and the source list database.

[0083] When the list in the source list database is synchronized to the target list database, such as Figure 3 As shown, lists from one or more source list databases are written to the target list database via online and batch queues. Existing lists in the source list database are copied to the data lake via the data replication component and then written to the distributed cache database and Table Storage Service database via batch queues. Incremental list information in the distributed message queue is written to the distributed cache database and Table Storage Service database via online queues, including writing detailed data required by the risk detail database from the incremental list to the risk detail database.

[0084] When receiving the list release instruction, if Figure 4As shown, after adding an incremental list to the source list database, the incremental list is synchronized to the deregulation cache, where it is then stored and its validity period is set. Furthermore, when a list query instruction is received, the deregulation cache is first queried using the client-granular storage structure. If no query results are found, the target list database is searched again and the query results are returned.

[0085] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0086] The embodiment of the present application also provides a customer list synchronization device. It should be noted that the customer list synchronization device of the embodiment of the present application can be used to execute the customer list synchronization method provided in the embodiment of the present application. The customer list synchronization device provided in the embodiment of the present application is introduced below.

[0087] Figure 5 Schematic diagram of a synchronization device for a customer list according to an embodiment of the present application. Figure 5 As shown, the device includes: a first parsing unit 201, a first writing unit 202, a first judging unit 203 and a first synchronizing unit 204.

[0088] The first parsing unit 201 is configured to synchronize the incremental list of customers in at least one source list database to the distributed message queue and parse the list to obtain a first target list through a data replication component;

[0089] The first writing unit 202 is used to write the first target list into the online queue through the online queue writing service;

[0090] The first determination unit 203 is configured to use a queue consumption process to determine whether the timestamp of the first target list is greater than the timestamp in the target list database;

[0091] The first synchronization unit 204 is configured to synchronize the first target list in the online queue to the target list database if the timestamp of the first target list is greater than the timestamp in the target list database.

[0092] The customer list synchronization device provided in the embodiment of the present application uses a first parsing unit 201 to synchronize the incremental list of customers in at least one source list database to a distributed message queue and parse it through a data replication component to obtain a first target list; a first writing unit 202 writes the first target list into an online queue through an online queue writing service; a first judgment unit 203 uses a queue consumption process to determine whether the timestamp of the first target list is greater than the timestamp in the target list database; and a first synchronization unit 204 synchronizes the first target list in the online queue to the target list database if the timestamp of the first target list is greater than the timestamp in the target list database, thereby solving the problem of poor timeliness of customer list synchronization technology in related technologies. By introducing a timestamp judgment mechanism and using an online queue to promptly process incremental lists, the list can be queried shortly after it is updated, thereby shortening the list synchronization cycle and improving the timeliness of customer list synchronization.

[0093] Optionally, in the customer list synchronization device provided in the embodiment of the present application, the device also includes: a setting unit, which is used to synchronize the incremental list of customers in at least one source list database to a distributed message queue and parse it through a data replication component, and before obtaining the first target list, if the incremental list of customers in the source list database has been successfully updated, then the certificate information and list type of the incremental list are synchronized to the decontrol cache library, and the effective time of the incremental list in the decontrol cache library is set.

[0094] Optionally, in the customer list synchronization device provided in the embodiment of the present application, the device also includes: a second parsing unit, used to synchronize the incremental list of customers in at least one source list database to the distributed message queue and parse it through the data replication component, and before obtaining the first target list, use a batch preloading method to process the list of customers already existing in at least one source list database to obtain a second target list; a second writing unit, used to write the second target list into the batch queue through a batch queue writing process; a second judgment unit, used to use a queue consumption process to determine whether the timestamp of the second target list is greater than the timestamp in the target list database; a second synchronization unit, used to synchronize the second target list to the target list database if the timestamp of the second target list is greater than the timestamp in the target list database.

[0095] Optionally, in the customer list synchronization device provided in the embodiment of the present application, the first writing unit includes: a mapping subunit, used to map the certificate information in the first target list into a hash value of a fixed length; a calculation subunit, used to divide the last digit of the hash value by the number of partitions of the online queue and take the remainder to obtain a first partition number, wherein the first partition number is the partition number for writing the first target list into the online queue; and a writing subunit, used to write the first target list into the online queue according to the first partition number.

[0096] Optionally, in the customer list synchronization device provided in the embodiment of the present application, the first synchronization unit includes: a storage subunit, used to store the first target list in the target list database using a key-value storage style, wherein the key-value storage style includes at least keywords and values, wherein the keywords are the customer's certificate information, and the values ​​are various list information integrated into a JSON string.

[0097] Optionally, in the customer list synchronization device provided in the embodiment of the present application, the setting unit also includes: a first query unit, which is used to synchronize the certificate information and list type of the incremental list to the de-control cache library if the incremental list of the customer in the source list database has been successfully updated, and set the effective time of the incremental list in the de-control cache library. After that, if a query instruction for the list to be queried is received, the de-control cache library is queried whether there is a list to be queried; a second query unit, which is used to obtain a first query result if there is a list to be queried in the de-control cache library and the query time is within the effective time of the list to be queried in the de-control cache library; a third query unit, which is used to search from the target list database to obtain a second query result if there is no list to be queried in the de-control cache library.

[0098] Optionally, in the customer list synchronization device provided in an embodiment of the present application, the second writing unit includes: a mapping subunit, used to write the second target list into the batch queue through a batch queue writing process, by mapping the certificate information in the second target list into a hash value of a fixed length; a calculation subunit, used to divide the last digit of the hash value by the number of partitions of the batch queue and take the remainder to obtain a second partition number, wherein the second partition number is the partition number for writing the second target list into the batch queue; a writing subunit, used to write the second target list into the batch queue according to the second partition number.

[0099] The customer list synchronization device provided in the embodiment of the present application uses a first parsing unit 201 to synchronize the incremental list of customers in at least one source list database to a distributed message queue and parse it through a data replication component to obtain a first target list; a first writing unit 202 writes the first target list into an online queue through an online queue writing service; a first judgment unit 203 uses a queue consumption process to determine whether the timestamp of the first target list is greater than the timestamp in the target list database; and a first synchronization unit 204 synchronizes the first target list in the online queue to the target list database if the timestamp of the first target list is greater than the timestamp in the target list database, thereby solving the problem of poor timeliness of customer list synchronization technology in related technologies. By introducing a timestamp judgment mechanism and using an online queue to promptly process incremental lists, the list can be queried shortly after it is updated, thereby shortening the list update cycle, ensuring the timeliness of list updates, and improving the efficiency of list queries.

[0100] The synchronization device for the customer list includes a processor and a memory. The above units are all stored in the memory as program units, and the processor executes the above program units stored in the memory to realize corresponding functions.

[0101] The processor includes a kernel, which retrieves corresponding program units from memory. One or more kernels can be provided, and by adjusting kernel parameters, a customer list synchronization method, device, and processor are provided to address the poor timeliness of customer list synchronization technology in related technologies.

[0102] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0103] An embodiment of the present invention provides a computer-readable storage medium having a program stored thereon, which implements the customer list synchronization method when executed by a processor.

[0104] An embodiment of the present invention provides a processor, which is used to run a program, wherein the method for synchronizing the customer list is executed when the program is run.

[0105] like Figure 6As shown, an embodiment of the present invention provides an electronic device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, the following steps are implemented: synchronizing an incremental list of customers in at least one source list database to a distributed message queue and parsing the list through a data replication component to obtain a first target list; writing the first target list to an online queue through an online queue write service; using a queue consumption process to determine whether the timestamp of the first target list is greater than the timestamp in the target list database; if the timestamp of the first target list is greater than the timestamp in the target list database, synchronizing the first target list in the online queue to the target list database.

[0106] When the processor executes the program, the following steps are also implemented: before synchronizing the incremental list of customers in at least one source list database to the distributed message queue and parsing it through the data replication component to obtain the first target list, the method also includes: if the incremental list of customers in the source list database has been successfully updated, then synchronizing the certificate information and list type of the incremental list to the decontrol cache library, and setting the effective time of the incremental list in the decontrol cache library.

[0107] When the processor executes the program, the following steps are also implemented: before synchronizing the incremental list of customers in at least one source list database to the distributed message queue and parsing it through the data replication component to obtain the first target list, the method also includes: using a batch preloading method to process the list of customers already existing in at least one source list database to obtain a second target list; writing the second target list into the batch queue through a batch queue writing process; using a queue consumption process to determine whether the timestamp of the second target list is greater than the timestamp in the target list database; if the timestamp of the second target list is greater than the timestamp in the target list database, synchronizing the second target list to the target list database.

[0108] When the processor executes the program, the following steps are also implemented: writing the first target list into the online queue through the online queue writing service includes: mapping the certificate information in the first target list into a hash value of fixed length; dividing the last digit of the hash value by the number of partitions of the online queue and taking the remainder to obtain the first partition number, wherein the first partition number is the partition number for writing the first target list into the online queue; and writing the first target list into the online queue according to the first partition number.

[0109] When the processor executes the program, the following steps are also implemented: if the timestamp of the first target list is greater than the timestamp in the target list database, then synchronizing the first target list in the online queue to the target list database includes: storing the first target list in the target list database using a key-value storage style, wherein the key-value storage style includes at least keywords and values, wherein the keywords are the customer's certificate information, and the values ​​are various list information integrated into a JSON string.

[0110] When the processor executes the program, the following steps are also implemented: if the incremental list of the customer in the source list database has been successfully updated, the certificate information and list type of the incremental list are synchronized to the de-control cache library, and the effective time of the incremental list in the de-control cache library is set. The method also includes: if a query instruction for the list to be queried is received, the de-control cache library is queried whether there is a list to be queried; if there is a list to be queried in the de-control cache library, and the query time is within the effective time of the list to be queried in the de-control cache library, a first query result is obtained; if there is no list to be queried in the de-control cache library, the target list database is searched to obtain a second query result.

[0111] When executing the program, the processor further implements the following steps: writing the second target list into the batch queue through the batch queue writing process includes: mapping the document information in the second target list into a fixed-length hash value; dividing the last digit of the hash value by the number of partitions in the batch queue and taking the remainder to obtain a second partition number, wherein the second partition number is the partition number for writing the second target list into the batch queue; and writing the second target list into the batch queue according to the second partition number. The device herein may be a server, a PC, a PAD, a mobile phone, etc.

[0112] The present application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program having the following method steps: synchronizing an incremental list of customers in at least one source list database to a distributed message queue and parsing the list through a data replication component to obtain a first target list; writing the first target list to an online queue through an online queue write service; using a queue consumption process to determine whether the timestamp of the first target list is greater than the timestamp in the target list database; if the timestamp of the first target list is greater than the timestamp in the target list database, synchronizing the first target list in the online queue to the target list database.

[0113] When executed on a data processing device, it is also suitable for executing an initialization program having the following method steps: before synchronizing the incremental list of customers in at least one source list database to a distributed message queue and parsing it through a data replication component to obtain a first target list, the method also includes: if the incremental list of customers in the source list database has been successfully updated, then synchronizing the certificate information and list type of the incremental list to the decontrol cache library, and setting the effective time of the incremental list in the decontrol cache library.

[0114] When executed on a data processing device, it is also suitable for executing a program initialized with the following method steps: before synchronizing the incremental list of customers in at least one source list database to a distributed message queue and parsing it through a data replication component to obtain a first target list, the method also includes: using a batch preloading method to process a list of customers already existing in at least one source list database to obtain a second target list; writing the second target list into a batch queue through a batch queue writing process; using a queue consumption process to determine whether the timestamp of the second target list is greater than the timestamp in the target list database; if the timestamp of the second target list is greater than the timestamp in the target list database, synchronizing the second target list to the target list database.

[0115] When executed on a data processing device, it is also suitable for executing an initialization program having the following method steps: writing the first target list to the online queue through the online queue writing service includes: mapping the certificate information in the first target list into a hash value of a fixed length; dividing the last digit of the hash value by the number of partitions of the online queue and taking the remainder to obtain the first partition number, wherein the first partition number is the partition number for writing the first target list to the online queue; and writing the first target list to the online queue according to the first partition number.

[0116] When executed on a data processing device, it is also suitable for executing an initialized program having the following method steps: if the timestamp of the first target list is greater than the timestamp in the target list database, then synchronizing the first target list in the online queue to the target list database includes: storing the first target list in the target list database using a key-value storage style, wherein the key-value storage style includes at least keywords and values, wherein the keywords are the customer's certificate information, and the values ​​are various list information integrated into a JSON string.

[0117] When executed on a data processing device, it is also suitable for executing an initialized program having the following method steps: if the incremental list of the customer in the source list database has been successfully updated, the certificate information and list type of the incremental list are synchronized to the de-control cache library, and the effective time of the incremental list in the de-control cache library is set. The method also includes: if a query instruction for the list to be queried is received, the de-control cache library is queried whether there is a list to be queried; if there is a list to be queried in the de-control cache library, and the query time is within the effective time of the list to be queried in the de-control cache library, a first query result is obtained; if there is no list to be queried in the de-control cache library, the target list database is searched to obtain a second query result.

[0118] When executed on a data processing device, it is also suitable for executing an initialized program having the following method steps: writing the second target list into the batch queue through a batch queue writing process includes: mapping the certificate information in the second target list into a hash value of a fixed length; dividing the last digit of the hash value by the number of partitions of the batch queue and taking the remainder to obtain a second partition number, wherein the second partition number is the partition number for writing the second target list into the batch queue; and writing the second target list into the batch queue according to the second partition number.

[0119] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0120] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0121] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0122] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0123] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0124] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0125] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0126] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a ..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

[0127] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0128] The above are merely embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should all be included within the scope of the claims of the present application.

Claims

1. A method for synchronizing a customer list, characterized in that: include: Synchronize the incremental list of customers in at least one source list database to the distributed message queue and parse it through the data replication component to obtain a first target list; Writing the first target list into the online queue through the online queue writing service; Using a queue consumption process, determining whether the timestamp of the first target list is greater than the timestamp in the target list database; If the timestamp of the first target list is greater than the timestamp in the target list database, synchronizing the first target list in the online queue to the target list database; The step of writing the first target list into the online queue through the online queue writing service includes: By mapping the certificate information in the first target list into a hash value of fixed length; Dividing the last digit of the hash value by the number of partitions of the online queue and taking the remainder to obtain a first partition number, wherein the first partition number is the partition number for writing the first target list into the online queue; The first target list is written into the online queue according to the first partition number.

2. The method according to claim 1, characterized in that Before synchronizing the incremental list of customers in at least one source list database to the distributed message queue and parsing the incremental list of customers in the distributed message queue to obtain the first target list through the data replication component, the method further includes: If the incremental list of the customer in the source list database has been successfully updated, the certificate information and list type of the incremental list will be synchronized to the decontrol cache library, and the validity period of the incremental list in the decontrol cache library will be set.

3. The method according to claim 1, characterized in that Before synchronizing the incremental list of customers in at least one source list database to the distributed message queue and parsing the incremental list of customers in the distributed message queue to obtain the first target list through the data replication component, the method further includes: Processing a list of customers existing in at least one source list database in a batch preloading manner to obtain a second target list; Writing the second target list into the batch queue through the batch queue writing process; Using a queue consumption process, determining whether the timestamp of the second target list is greater than the timestamp in the target list database; If the timestamp of the second target list is greater than the timestamp in the target list database, the second target list is synchronized to the target list database.

4. The method according to claim 1, wherein If the timestamp of the first target list is greater than the timestamp in the target list database, synchronizing the first target list in the online queue to the target list database includes: The first target list is stored in the target list database using a key-value storage style, wherein the key-value storage style includes at least a keyword and a value, wherein the keyword is the customer's certificate information, and the value is various list information integrated into a JSON string.

5. The method according to claim 2, characterized in that If the incremental list of the customer in the source list database has been successfully updated, then synchronizing the certificate information and list type of the incremental list to the decontrol cache, and setting the validity period of the incremental list in the decontrol cache, the method further includes: If the incremental list of the customer in the source list database has been successfully updated, the certificate information and list type of the incremental list are synchronized to the decontrol cache, and the validity period of the incremental list in the decontrol cache is set; If a query instruction for a list to be queried is received, query whether the list to be queried exists in the control release cache; If the list to be queried exists in the control release cache, and the query time is within the valid time of the list to be queried in the control release cache, a first query result is obtained; If the to-be-queried list does not exist in the control release cache, the target list database is searched to obtain a second query result.

6. The method according to claim 3, characterized in that Writing the second target list into the batch queue through the batch queue writing process includes: By mapping the certificate information in the second target list into a hash value of fixed length; Dividing the last digit of the hash value by the number of partitions of the batch queue and taking the remainder, obtaining a second partition number, wherein the second partition number is the partition number for writing the second target list into the batch queue; The second target list is written into the batch queue according to the second partition number.

7. A customer list synchronization device, characterized in that: include: A first parsing unit is configured to synchronize an incremental list of customers in at least one source list database to a distributed message queue and parse the incremental list to obtain a first target list through a data replication component; a first writing unit, configured to write the first target list into an online queue through an online queue writing service; a first determining unit, configured to use a queue consumption process to determine whether a timestamp of the first target list is greater than a timestamp in a target list database; a first synchronization unit, configured to synchronize the first target list in the online queue to the target list database if the timestamp of the first target list is greater than the timestamp in the target list database; Among them, the first writing unit includes a mapping subunit, which is used to map the certificate information in the first target list into a hash value of fixed length; a calculation subunit, which is used to divide the last digit of the hash value by the number of partitions of the online queue and take the remainder to obtain a first partition number, wherein the first partition number is the partition number for writing the first target list into the online queue; and a writing subunit, which is used to write the first target list into the online queue according to the first partition number.

8. A processor, characterized in that: The processor is configured to run a program, wherein the program, when running, executes the customer list synchronization method according to any one of claims 1 to 6.

9. An electronic device, characterized in that: It includes one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the customer list synchronization method described in any one of claims 1 to 6.

Images