Labeled customer group selection system, method and device based on multi-source data
Through the multi-source data-based tag customer group selection system, the rules engine automatically identify and process multiple data sources, the problems of poor consistency of label effective time and incremental updates are solved, efficient and timely data synchronization is achieved, and the improvement of the Marketing effects and user experience.
Patent Information
- Application Number
- CN202410988506.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-15
AI Technical Summary
In the prior art, the label's effective time is poor and incremental updates cannot be performed, resulting in low timeliness of customer group selection, affecting marketing effects and user experience.
A tag customer group selection system based on multi-source data is designed, and multiple data sources are obtained through the data retrieval module. The data structure is automatically identified using a pre-configured rule engine, query statements are generated and incremental data are extracted, and formatted processing is performed. Verify, realize automatic synchronization and alarm mechanisms, and ensure the timeliness and accuracy of data.
It realizes efficient query of any table and any field, improves the timeliness and accuracy of data synchronization, optimizes synchronization efficiency, reduces waste of computing resources, and improves the stability and user experience of the system.
Smart Images

Figure CN118964374A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of big data technology, and specifically relates to a label customer group selection system, method and device based on multi-source data. Background Art
[0002] The application of bank tag customer groups in the field of financial technology has evolved from simple to complex. In the early days, bank tag customer lists were mostly manually generated through data exploration and data extraction for marketing. This method was slow, had a low degree of automation, high labor costs, and no customer group data was accumulated. It was impossible to accumulate customer group strategies and trace historical data for effect analysis.
[0003] With the development of big data technology and the emergence of new technologies such as distributed search engines, banks have begun to implement label combination operations based on ElasticSearch and provide customer group selection services, which have basically achieved offline automation. The business only needs simple front-end selection, and generates a list of customer groups to be marketed through label intersection, union, and difference operations, and reaches users through telemarketing, SMS, PUSH and other channels. However, there are diverse data sources, inconsistent data generation time from different sources, low overall customer timeliness in the large wide table method, and low performance of HIVE-TO-ElasticSearch components writing through external tables. The business cannot process the customer group list normally on T+1, and cannot incrementally update and synchronize the change label data, which has a great impact on business development results and user experience.
[0004] In the marketing field of various industries, the bottom layer is based on customer tag data. Customer group data is generated based on customer tags, customer group stratification is performed, and differentiated marketing is performed for different customer groups. Therefore, the timeliness of data is crucial to the marketing effect. Traditional tag customer groups are based on ElasticSearch for customer group selection services, which can basically achieve customer group selection and crowd estimation services, but there are the following problems:
[0005] 1. Since there are many types of upstream big data HIVE tables that labels rely on, and the data sources come from various business system data, tracking behavior data, external data, etc., the effective time of each data source is also different. Based on the HIVE large wide table method, all labels have the same effective time and poor timeliness.
[0006] 2. Currently, HIVE-TO-ElasticSearch components synchronize data to ElasticSearch through HIVE external tables. However, the large wide table can only be formed after all upstream data is completed, which results in a very late synchronization with ElasticSearch. In addition, this solution cannot perform incremental updates, resulting in poor performance of full user synchronization, a great waste of computing resources, and instability of label services during synchronization. Summary of the invention
[0007] The purpose of the present invention is to provide a label customer group selection system, method and device based on multi-source data to solve the problem that the label effective time is consistent but the timeliness is very poor and incremental updates cannot be made.
[0008] To achieve the above-mentioned purpose, the present invention provides the following technical solution: a label customer group selection system based on multi-source data, comprising:
[0009] The data retrieval module is used to obtain multiple data sources and automatically identify new data sources and their data structures through a pre-configured rule engine to obtain source data information;
[0010] A data identification module is used to obtain source data information, generate a query statement according to the data structure in the source data information, retrieve the source data information through the query statement, obtain the query result, extract the query result, generate metadata information, store the metadata information, and broadcast the metadata information to all execution nodes;
[0011] The data processing module is used to obtain metadata information, generate a comparison script, identify incremental data in the query results through the comparison script, and perform format processing on the incremental data to obtain standard data;
[0012] The data synchronization module is used to obtain the broadcast metadata information and standard data, generate target data, and automatically verify whether the target data meets the requirements of the standard library. If it does, the target data is synchronized. If it does not, the failure data is output and an alarm message is returned.
[0013] Preferably, the data synchronization module includes:
[0014] The verification module is used to obtain the target data and automatically verify whether the target data meets the requirements of the standard library. If it meets the requirements, the target data is synchronized. If it does not meet the requirements, the failure data is output and an alarm message is returned.
[0015] The second attempt module is used to obtain the incremental data of the failed synchronization and try to synchronize the target data again. If successful, the execution ends. If failed, the problem data is output and an alarm message is returned.
[0016] Preferably, the data retrieval module comprises:
[0017] A rule configuration module is used to obtain manually input data recognition rules, data mapping rules and data conversion rules;
[0018] The automatic detection module is used to automatically identify the newly added data source and its data structure and obtain the source data information.
[0019] Preferably, the data identification module includes:
[0020] The automatic generation module is used to generate a query statement according to the data structure in the source data information, retrieve the source data information through the query statement, and obtain the query result;
[0021] The query preset module is used to obtain the novelty search results, extract the query results, obtain the field names and corresponding data types, and generate metadata information;
[0022] A storage module, used to store metadata information in distributed memory;
[0023] The broadcast module is used to broadcast metadata information to all execution nodes.
[0024] A method for selecting a label customer group based on multi-source data includes the following steps:
[0025] Obtain multiple data sources, and automatically identify new data sources and their data structures through a pre-configured rule engine to obtain source data information;
[0026] Obtain source data information, generate query statements according to the data structure in the source data information, retrieve the source data information through the query statements, obtain query results, extract the query results, generate metadata information, store the metadata information, and broadcast the metadata information to all execution nodes;
[0027] Obtain metadata information, generate a comparison script, use the comparison script to identify incremental data in the query results, format the incremental data, and obtain standard data;
[0028] Obtain broadcast metadata information and standard data, generate target data, and automatically verify whether the target data meets the requirements of the standard library. If it does, synchronize the target data. If it does not, output failure data and return an alarm message.
[0029] Preferably, the synchronization of standard data includes the following execution steps:
[0030] Get the target data and automatically check whether the target data meets the requirements of the standard library. If it meets the requirements, synchronize the target data. If it does not meet the requirements, output the failed data and return the alarm information.
[0031] If the target data is synchronized, try to synchronize the target data again. If successful, end the execution. If the second attempt fails, output the problem data and return an alarm message.
[0032] Preferably, the pre-configured rule engine comprises the following steps:
[0033] Obtain manually input data recognition rules, data mapping rules and data conversion rules;
[0034] Automatically identify new data sources and their data structures to obtain source data information.
[0035] Preferably, the storing and broadcasting of the metadata information to all execution nodes further includes the following steps:
[0036] Generate a query statement according to the data structure in the source data information, retrieve the source data information through the query statement, and obtain the query result;
[0037] Get the new search results, extract the query results, get the field names and corresponding data types, and generate metadata information;
[0038] Store metadata information in distributed memory and broadcast metadata information to all execution nodes.
[0039] A device for selecting a label customer group based on multi-source data, comprising:
[0040] A processor and a memory, wherein the memory stores a computer program executable by the processor, and when the processor executes the computer program, a method for selecting a label customer group based on multi-source data is implemented.
[0041] The technical effects and advantages of the present invention are as follows: by acquiring and constructing query conditions, it is possible to query any table and any field, with high versatility, convenient and efficient use; by automatically verifying standard data, it synchronizes standard data that meets the requirements, and can promptly discover standard data that does not meet the requirements, and solves problems by outputting problem data, returning alarm information, and obtaining manual intervention. Through the controllable number of synchronous connections, users are allowed to specify the data range and synchronization frequency for synchronization, and the concurrency and batch size of target data synchronization can be dynamically adjusted according to the load of the ES cluster to optimize synchronization efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 It is a schematic diagram of the method flow of the present invention;
[0043] Figure 2 It is a system schematic diagram of the present invention. DETAILED DESCRIPTION
[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0045] The present invention provides Figure 1-2The system, method and device for selecting a customer group based on multi-source data are shown in the figure, and the above steps are performed through the following modules: a data retrieval module, a data identification module, a data generation module and a data synchronization module, including the following steps:
[0046] S1: Obtain multi-source data through the data retrieval module, and automatically identify the newly added data source and its data structure through the pre-configured rule engine, including: data identification rules, data mapping rules and data conversion rules, to obtain source data information;
[0047] S2: Obtain source data information through the data identification module, generate a query statement according to the data structure in the source data information, retrieve the source data information through the query statement, obtain the query result, extract the query result, obtain metadata information, and broadcast the metadata information to all execution nodes;
[0048] S3: Obtain metadata information through the data generation module, generate a comparison script, identify incremental data in the query results through the comparison script, and store the incremental data. After storage, format the incremental data to obtain standard data.
[0049] S4: Obtain the broadcast metadata information and standard data through the data synchronization module, generate target data, automatically verify the target data, and determine whether the target data meets the requirements of the standard library. If it does, synchronize the target data. If the synchronization fails, start the retry mechanism and try again. If successful, end the execution. If it fails, output the problem data and return an alarm message. If it does not meet the requirements, output the failure data and return an alarm message.
[0050] The above steps are performed as follows:
[0051] S1: Obtain multiple data sources through the data retrieval module, where the sources of the multiple data sources include Hive tables, HDFS file systems, real-time message queues, etc. The contents of the multiple data sources include: user behavior data, natural attribute data, marketing and equity data, deposit and loan application data, overdue and collection data, external data, etc.; after obtaining multiple data sources, manually input data identification rules, data mapping rules and data conversion rules can be obtained through the rule configuration module, the rule engine can be determined, and the new data source and its data structure can be detected through the rule engine to obtain source data information. When multiple data sources are connected to the system, the rule engine has a registration process, and the basic information of multiple data sources (such as name, type, connection information, etc.) is registered in the system to identify the new data source, so that the back-end code can automatically identify it from the system without changing the code; where the data identification rules include: table structure, column data type, table partition information; data type mapping rules include: field type mapping; data conversion rules include: format standardization, data encryption and desensitization, and data splitting. Data type mapping rules are used to solve the consistency and data compatibility between systems. Data conversion rules are used to standardize data for new data sources, convert the data in the new data sources into a unified format to meet business needs, and convert data according to business logic, such as converting amounts from original currency to standard currency. Data identification rules are used to identify and understand the structure and content of new data sources. Its main functions are structure identification, source data information extraction, and data relationship understanding.
[0052] S2: Establish a connection with Hive, dynamically build query statements through table names, field names and filter conditions in the data structure, and execute query statements through Hive SQL to retrieve source data information and obtain matching query results. By extracting the query results, metadata information is obtained, where metadata information includes: field names and data types. The extracted metadata information is stored in distributed memory, and the metadata information is broadcast to all execution nodes using Spark's broadcast variable mechanism, so that metadata information can be accessed efficiently without repeated transmission, thereby improving data processing efficiency. By obtaining and building query conditions, queries can be made to any table and any field, which is highly versatile, convenient and efficient to use.
[0053] S3: Obtain metadata information in distributed memory through the data generation module, generate a comparison script for identifying incremental data based on the metadata information, execute the generated comparison script, identify incremental data in the Hive table through left join operations and field comparison conditions, compare current data and historical data for any added data, if so, the added data is incremental data, store the incremental data, perform format processing based on the data type of the incremental data, and perform format processing on the incremental data to generate standard data in JSON format suitable for the ES cluster.
[0054] S4: Obtain standard data and metadata information after broadcast through the data synchronization module, generate target data according to the quasi-data and broadcast metadata information, set and initialize the connection pool, set the size of the number of connections, and set the connection pool to the maximum number of connections and the maximum number of connections for each route to optimize the use of connection resources and improve access efficiency. Automatically check whether the target data meets the synchronization standards of the standard library. If it meets the synchronization standards, the target data is synchronized to the ES cluster. If it does not meet the synchronization standards, the failed data is output and the alarm information is returned. Among them, for the target data that meets the standard library, when it is synchronized to the ES cluster, there is a synchronization failure. At this time, the retry mechanism can be started to try to synchronize the standard data again. If the attempt is successful, the problem data is output and the alarm information is returned. If the synchronization is successful, the execution is terminated. By automatically verifying the target data, the target data that meets the quality standards of the standard library is synchronized to the ES cluster to ensure that the synchronized data meets the requirements, and the target data with quality problems can be found in time through automatic verification. By outputting the problem data and returning the alarm information, the problem is solved by obtaining manual intervention, and the solution to the problem data is improved. According to the load of the ES cluster, the concurrency and batch size of data synchronization are dynamically adjusted to optimize the synchronization efficiency. Through the controllable number of synchronization connections, users are allowed to specify the data range and synchronization frequency for synchronization, and can dynamically adjust the concurrency and batch size of target data synchronization according to the load of the ES cluster to optimize synchronization efficiency.
[0055] Finally, it should be noted that the above is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, it is still possible for those skilled in the art to modify the technical solutions described in the aforementioned embodiments or to make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
Claims
1. A label customer group selection system based on multi-source data, characterized by: include: The data retrieval module is used to obtain multiple data sources and automatically identify new data sources and their data structures through a pre-configured rule engine to obtain source data information; A data identification module is used to obtain source data information, generate a query statement according to the data structure in the source data information, retrieve the source data information through the query statement, obtain the query result, extract the query result, generate metadata information, store the metadata information, and broadcast the metadata information to all execution nodes; The data processing module is used to obtain metadata information, generate a comparison script, identify incremental data in the query results through the comparison script, and perform format processing on the incremental data to obtain standard data; The data synchronization module is used to obtain the broadcast metadata information and standard data, generate target data, and automatically verify whether the target data meets the requirements of the standard library. If it does, the target data is synchronized. If it does not, the failure data is output and an alarm message is returned.
2. According to claim 1, a label customer group selection system based on multi-source data is characterized by: The data synchronization module includes: The verification module is used to obtain the target data and automatically verify whether the target data meets the requirements of the standard library. If it meets the requirements, the target data is synchronized. If it does not meet the requirements, the failure data is output and an alarm message is returned. The second attempt module is used to obtain the incremental data of the failed synchronization and try to synchronize the target data again. If successful, the execution ends. If failed, the problem data is output and an alarm message is returned.
3. The label customer group selection system based on multi-source data according to claim 1 is characterized by: The data retrieval module comprises: A rule configuration module is used to obtain manually input data recognition rules, data mapping rules and data conversion rules; The automatic detection module is used to automatically identify the newly added data source and its data structure and obtain the source data information.
4. According to claim 1, a label customer group selection system based on multi-source data is characterized by: The data identification module comprises: The automatic generation module is used to generate a query statement according to the data structure in the source data information, retrieve the source data information through the query statement, and obtain the query result; The query preset module is used to obtain the novelty search results, extract the query results, obtain the field names and corresponding data types, and generate metadata information; A storage module, used to store metadata information in distributed memory; The broadcast module is used to broadcast metadata information to all execution nodes.
5. A method for selecting a label customer group based on multi-source data, characterized in that: The steps include: Obtain multiple data sources, and automatically identify new data sources and their data structures through a pre-configured rule engine to obtain source data information; Obtain source data information, generate query statements according to the data structure in the source data information, retrieve the source data information through the query statements, obtain query results, extract the query results, generate metadata information, store the metadata information, and broadcast the metadata information to all execution nodes; Obtain metadata information, generate a comparison script, use the comparison script to identify incremental data in the query results, format the incremental data, and obtain standard data; Obtain broadcast metadata information and standard data, generate target data, and automatically verify whether the target data meets the requirements of the standard library. If it does, synchronize the target data. If it does not, output failure data and return an alarm message.
6. The method for selecting a label customer group based on multi-source data according to claim 5, characterized in that: The synchronization standard data includes the following execution steps: Get the target data and automatically check whether the target data meets the requirements of the standard library. If it meets the requirements, synchronize the target data. If it does not meet the requirements, output the failed data and return the alarm information. If the target data is synchronized, try to synchronize the target data again. If successful, end the execution. If the second attempt fails, output the problem data and return an alarm message.
7. The method for selecting a label customer group based on multi-source data according to claim 6, characterized in that: The pre-configured rule engine comprises the following steps: Obtain manually input data recognition rules, data mapping rules and data conversion rules; Automatically identify new data sources and their data structures to obtain source data information.
8. The method for selecting a label customer group based on multi-source data according to claim 7, characterized in that: The step of storing and broadcasting the metadata information to all execution nodes also includes the following steps: Generate a query statement according to the data structure in the source data information, retrieve the source data information through the query statement, and obtain the query result; Get the new search results, extract the query results, get the field names and corresponding data types, and generate metadata information; Store metadata information in distributed memory and broadcast metadata information to all execution nodes.
9. A device for selecting customer groups based on multi-source data, characterized in that: include: A processor and a memory, wherein the memory stores a computer program executable by the processor, and when the processor executes the computer program, the method according to any one of claims 5 to 8 is implemented.
Citation Information
Cited By
Lightweight multi-data-source dynamic management method and system, terminal equipment and computer readable storage medium
CN120872978A
