Blacklist data processing method, blacklist data processing system, blacklist data processing equipment and readable storage medium

Abstract

The invention discloses a blacklist data processing method, system and device and a readable storage medium, reprocessing full-amount blacklist data processed by a preset AI model in a data warehouse, determining incremental blacklist data, and directly synchronizing the incremental blacklist data to a blacklist system according to a preset period by the data warehouse. The situation that one more data is stored through a risk control system in the middle is avoided, and system calling links are reduced; furthermore, the blacklist system can be compared with the data volume of the data warehouse based on each scheduling, whether the problem of number leakage exists or not is searched, in addition, a data compensation processing mechanism is further provided, number complementation is flexibly carried out, and the efficiency is improved. The problem of resource waste caused by full data synchronization and the problem of data redundancy caused by the fact that multiple systems store the same data in an existing scheme are solved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, is applicable to the financial field, and in particular, to a blacklist data processing method, system, device and readable storage medium. Background Art

[0002] In the insurance industry, the blacklist management system of the risk control platform is a key component, which improves the efficiency and accuracy of the underwriting process by integrating advanced technical means. As the core of data processing, risk control system A is responsible for synchronizing the full blacklist data from the data warehouse and identifying and processing incremental updates. In addition, blacklist system B, as the main system providing blacklist verification services, also needs to obtain the latest blacklist data from risk control system A for real-time query by the underwriting system.

[0003] In general, the current process is:

[0004] Risk control system A uses the big data synchronization tool Sqoop to regularly synchronize the full blacklist data from the data warehouse to its own database.

[0005] The scheduling module of risk control system A is responsible for identifying incremental data and synchronizing this incremental data to related systems, such as blacklist system B.

[0006] Blacklist system B receives the blacklist data synchronized by risk control system A and saves it in its own database so that the underwriting system can perform real-time verification.

[0007] However, the existing methods have certain defects, including:

[0008] Full data synchronization leads to resource waste: Risk control system A needs to synchronize the full blacklist data every time, which not only increases storage costs, but also increases the consumption of computing resources because of the need to distinguish between new and old data.

[0009] Data redundancy: Since both risk control system A and blacklist system B store the same data, storage resources are wasted.

[0010] Synchronization delay and risk of missing data: If there is a delay in big data synchronization, the scheduling module may not be able to obtain the latest data in time, resulting in missing data, which is difficult for developers to detect.

[0011] Lack of flexibility: The existing data synchronization mechanism lacks flexibility and cannot push data based on a specific point in time, which is particularly inconvenient when a specific batch of data needs to be pushed. Summary of the invention

[0012] The present application provides a blacklist data processing method, system, device and readable storage medium, which solves the resource waste problem caused by full data synchronization in the existing solution and the data redundancy problem existing when multiple systems store the same data.

[0013] In view of this, the first aspect of the present application provides a blacklist data processing method, the method comprising:

[0014] Constructing a data partition table in a data warehouse, wherein the data partition table stores initial blacklist data;

[0015] When executing a data processing task, the data warehouse receives the full blacklist data processed by a preset AI model, determines the incremental blacklist data in the full blacklist data according to a preset identifier, adds a target partition field to the incremental blacklist data, and stores it in a database table of the data warehouse;

[0016] The data warehouse synchronizes the incremental blacklist data within the period to the blacklist system according to the target partition field according to a preset period, and the blacklist system synchronizes the incremental blacklist data to the associated party system.

[0017] Optionally, before executing the data processing task, the step further includes:

[0018] The data preprocessing system obtains original insurance policy data, wherein the original insurance policy data includes information of several original insurance policies;

[0019] The data preprocessing system converts the original policy data into a target format and stores the converted data in a graph database;

[0020] The graph database creates entity corresponding nodes according to the original insurance policy data, establishes relationships between the nodes based on the information of the original insurance policy, and marks the nodes with the same information accordingly;

[0021] The data preprocessing system calls a preset AI model to perform feature extraction and similarity calculation based on the nodes created in the graph database and the relationships between the nodes, and determines the full amount of blacklist data in the original policy data.

[0022] Optionally, the data preprocessing system calls a preset AI model, performs feature extraction and similarity calculation based on the nodes created in the graph database and the relationships between the nodes, and determines that the full amount of blacklist data in the original insurance policy data is specifically:

[0023] The data preprocessing system calls a preset AI model, wherein a preset feature list is constructed in the preset AI model;

[0024] The data preprocessing system extracts the target features of the nodes created in the graph database through the preset AI model, and calculates the similarity between the original insurance policies based on the relationship between the nodes;

[0025] If the similarity is higher than a preset threshold, the data preprocessing system marks the information of the two or more original insurance policies correspondingly associated as blacklist data;

[0026] The data preprocessing system counts all blacklist data in the original insurance policy data.

[0027] Optionally, the data warehouse synchronizes the incremental blacklist data within a period to the blacklist system according to the target partition field according to a preset period, specifically:

[0028] The data warehouse uses the target partition field as a mapping to establish a database connection with the blacklist system;

[0029] The blacklist system sends an incremental data scheduling instruction to the data warehouse according to a preset period, wherein the incremental data scheduling instruction carries a first target partition field value;

[0030] The data warehouse synchronizes the corresponding incremental blacklist data to the blacklist system according to the first target partition field value.

[0031] Optionally, before the blacklist system synchronizes the incremental blacklist data to the associated party system, the process further includes:

[0032] The blacklist system sends a data volume query instruction to the data warehouse, wherein the data volume query instruction carries a second target partition field value;

[0033] The data warehouse counts the first data volume of the stored incremental blacklist data according to the second target partition field value and returns the data to the blacklist system;

[0034] The blacklist system counts and stores a second amount of the incremental blacklist data according to the second target partition field value;

[0035] The blacklist system compares the first data volume with the second data volume for consistency, and if they are inconsistent, performs data missing warning and data compensation processing.

[0036] Optionally, the data compensation process is specifically:

[0037] The blacklist system deletes the incremental blacklist data stored corresponding to the second target partition field value;

[0038] The blacklist system re-sends the incremental data scheduling instruction to the data warehouse, and the incremental data scheduling instruction carries the second target partition field value, so that the data warehouse synchronizes the corresponding incremental blacklist data to the blacklist system according to the second target partition field value.

[0039] Optionally, the blacklist system synchronizing the incremental blacklist data to the associated party system specifically includes:

[0040] The blacklist system synchronizes the incremental blacklist data to the associated party system in real time according to the real-time call request of the associated party system;

[0041] or

[0042] The blacklist system sends a message containing the incremental blacklist data to a message queue, and the associated party system consumes the message containing the incremental blacklist data from the message queue.

[0043] A second aspect of the present application provides a blacklist data processing system, the system comprising: a data warehouse and a blacklist system, the data warehouse being communicatively connected to the blacklist system, wherein:

[0044] Constructing a data partition table in the data warehouse, wherein the data partition table stores initial blacklist data;

[0045] When executing a data processing task, the data warehouse receives the full blacklist data processed by a preset AI model, determines the incremental blacklist data in the full blacklist data according to a preset identifier, adds a target partition field to the incremental blacklist data, and stores it in a database table of the data warehouse;

[0046] The data warehouse synchronizes the incremental blacklist data within the period to the blacklist system according to the target partition field according to a preset period, and the blacklist system synchronizes the incremental blacklist data to the associated party system.

[0047] Optionally, the system further comprises a data preprocessing system, and the data preprocessing system is communicatively connected with the data warehouse, wherein: before executing the data processing task, the system further comprises:

[0048] The data preprocessing system obtains original insurance policy data, wherein the original insurance policy data includes information of several original insurance policies;

[0049] The data preprocessing system converts the original policy data into a target format and stores the converted data in a graph database;

[0050] The graph database creates entity corresponding nodes according to the original insurance policy data, establishes relationships between the nodes based on the information of the original insurance policy, and marks the nodes with the same information accordingly;

[0051] The data preprocessing system calls a preset AI model to perform feature extraction and similarity calculation based on the nodes created in the graph database and the relationships between the nodes, and determines the full amount of blacklist data in the original policy data.

[0052] Optionally, the data preprocessing system calls a preset AI model, performs feature extraction and similarity calculation based on the nodes created in the graph database and the relationships between the nodes, and determines that the full amount of blacklist data in the original insurance policy data is specifically:

[0053] The data preprocessing system calls a preset AI model, wherein a preset feature list is constructed in the preset AI model;

[0054] The data preprocessing system extracts the target features of the nodes created in the graph database through the preset AI model, and calculates the similarity between the original insurance policies based on the relationship between the nodes;

[0055] If the similarity is higher than a preset threshold, the data preprocessing system marks the information of the two or more original insurance policies correspondingly associated as blacklist data;

[0056] The data preprocessing system counts all blacklist data in the original insurance policy data.

[0057] Optionally, the data warehouse synchronizes the incremental blacklist data within a period to the blacklist system according to the target partition field according to a preset period, specifically:

[0058] The data warehouse uses the target partition field as a mapping to establish a database connection with the blacklist system;

[0059] The blacklist system sends an incremental data scheduling instruction to the data warehouse according to a preset period, wherein the incremental data scheduling instruction carries a first target partition field value;

[0060] The data warehouse synchronizes the corresponding incremental blacklist data to the blacklist system according to the first target partition field value.

[0061] Optionally, before the blacklist system synchronizes the incremental blacklist data to the associated party system, the process further includes:

[0062] The blacklist system sends a data volume query instruction to the data warehouse, wherein the data volume query instruction carries a second target partition field value;

[0063] The data warehouse counts the first data volume of the stored incremental blacklist data according to the second target partition field value and returns the data to the blacklist system;

[0064] The blacklist system counts and stores a second amount of the incremental blacklist data according to the second target partition field value;

[0065] The blacklist system compares the first data volume with the second data volume for consistency, and if they are inconsistent, performs data missing warning and data compensation processing.

[0066] Optionally, the data compensation process is specifically:

[0067] The blacklist system deletes the incremental blacklist data stored corresponding to the second target partition field value;

[0068] The blacklist system re-sends the incremental data scheduling instruction to the data warehouse, and the incremental data scheduling instruction carries the second target partition field value, so that the data warehouse synchronizes the corresponding incremental blacklist data to the blacklist system according to the second target partition field value.

[0069] Optionally, the blacklist system synchronizing the incremental blacklist data to the associated party system specifically includes:

[0070] The blacklist system synchronizes the incremental blacklist data to the associated party system in real time according to the real-time call request of the associated party system;

[0071] or

[0072] The blacklist system sends a message containing the incremental blacklist data to a message queue, and the associated party system consumes the message containing the incremental blacklist data from the message queue.

[0073] A third aspect of the present application provides a blacklist data processing device, the device comprising a processor and a memory:

[0074] The memory is used to store program code and transmit the program code to the processor;

[0075] The processor is used to execute the steps of the method for processing blacklist data as described in the first aspect according to the instructions in the program code.

[0076] A fourth aspect of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store program code, and the program code is used to execute the method described in the first aspect above.

[0077] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0078] In the present application, a blacklist data processing method, system, device and readable storage medium are provided. The full blacklist data processed by a preset AI model is reprocessed in a data warehouse to determine the incremental blacklist data. The data warehouse directly synchronizes the incremental blacklist data to the blacklist system according to a preset period, thereby avoiding the situation where an extra copy of the data is stored in the middle through the risk control system, and reducing the system call link. Furthermore, the blacklist system can compare the data volume based on each scheduling with the data warehouse to find out whether there is a problem of missing numbers. In addition, a data compensation processing mechanism is provided to flexibly make up numbers, which solves the resource waste problem caused by full data synchronization in the existing solution and the data redundancy problem existing in multiple systems storing the same data. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] Figure 1 This is a method flow chart of the blacklist data processing method in the embodiment of the present application;

[0080] Figure 2 This is a schematic diagram of the structure of the blacklist data processing system in the embodiment of the present application;

[0081] Figure 3 It is a structural diagram of the blacklist data processing device in the embodiment of the present application. DETAILED DESCRIPTION

[0082] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0083] The present application designs a blacklist data processing method, system, device and readable storage medium, which solves the resource waste problem caused by full data synchronization in the existing solution and the data redundancy problem existing when multiple systems store the same data.

[0084] For easier understanding, see Figure 1 , Figure 1 Flow chart of the method for processing blacklist data in the embodiment of the present application. Figure 1 As shown, specifically:

[0085] S101, constructing a data partition table in a data warehouse, wherein the data partition table stores initial blacklist data;

[0086] It should be noted that a partition table is created in the data warehouse to store the initial blacklist data. The partition table can be partitioned by date, time or other logic.

[0087] Example: Create a partition table named blacklist_data, use partition_date as the partition field, and store the initial blacklist data.

[0088] S102. When executing a data processing task, the data warehouse receives the full blacklist data processed by the preset AI model, determines the incremental blacklist data in the full blacklist data according to the preset identifier, adds a target partition field to the incremental blacklist data, and stores it in the database table of the data warehouse;

[0089] It should be noted that when a data processing task is triggered, the data warehouse receives the full blacklist data from the preset AI model.

[0090] For example, the AI ​​model generates a file containing 1,000 blacklist records after processing. The data warehouse receives this data and identifies the 100 new records added since the last update as incremental data.

[0091] S103. The data warehouse synchronizes the incremental blacklist data within the period to the blacklist system according to the target partition field according to the preset period, and the blacklist system synchronizes the incremental blacklist data to the associated party system.

[0092] It should be noted that the data warehouse synchronizes incremental blacklist data to the blacklist system according to a preset period (such as 1 a.m. every day).

[0093] For example: The data warehouse automatically runs a synchronization task at 1 a.m. to send the previous day's incremental blacklist data to the blacklist system.

[0094] Furthermore, before executing the data processing task, the following steps are also included:

[0095] The data preprocessing system obtains original insurance policy data, which contains information of several original insurance policies;

[0096] It should be noted that the data preprocessing system obtains original policy data from multiple data sources (such as the business system of an insurance company).

[0097] Example: Export a CSV file containing all insurance policies for the past year from an insurance company's business system.

[0098] The data preprocessing system converts the original policy data into the target format and stores it in the graph database;

[0099] The graph database creates entity corresponding nodes based on the original policy data, establishes the relationship between the nodes based on the original policy information, and marks the nodes with the same information accordingly;

[0100] It should be noted that the original policy data is converted into a format supported by the graph database and stored.

[0101] For example: Convert the insurance policy data in a CSV file into nodes and edges in a graph database. Each insurance policy is a node, and the relationships between nodes (such as the same policyholder) are connected by edges.

[0102] The data preprocessing system calls the preset AI model to extract features and calculate similarities based on the nodes created in the graph database and the relationships between the nodes to determine the full amount of blacklist data in the original policy data.

[0103] It should be noted that the AI ​​model analyzes the nodes and edges in the graph database, extracts features and calculates similarities.

[0104] For example, the model identifies that two policies were purchased by the same policyholder at similar times, and calculates the similarity to be 0.9, which is higher than the preset threshold of 0.8.

[0105] Policies with high similarity are marked as blacklist data.

[0106] For example: The system marks the policy records with a similarity higher than 0.8 as blacklisted and updates the blacklist_data table.

[0107] Furthermore, the data preprocessing system calls the preset AI model to extract features and calculate similarities based on the nodes created in the graph database and the relationships between the nodes, and determines that the full amount of blacklist data in the original policy data is as follows:

[0108] The data preprocessing system calls a preset AI model, and a preset feature list is built in the preset AI model;

[0109] The data preprocessing system extracts the target features of the nodes created in the graph database through a preset AI model, and calculates the similarity between the original policies based on the relationship between the nodes;

[0110] If the similarity is higher than the preset threshold, the data preprocessing system will mark the information of two or more correspondingly associated original insurance policies as blacklist data;

[0111] The data preprocessing system counts the full amount of blacklist data in the original policy data.

[0112] It should be noted that the following algorithms or technologies are usually used when performing feature extraction and similarity calculation in AI models:

[0113] Deep learning models: such as convolutional neural networks (CNN) for image feature extraction, recurrent neural networks (RNN) or Transformer models for text feature extraction. These models can automatically learn complex feature representations from data.

[0114] Feature vectorization: Convert text or images into numerical vectors. Common methods include Bag of Words, TF-IDF, Word Embeddings (such as Word2Vec, GloVe), and image histograms.

[0115] When calculating similarity, there are many methods, for example:

[0116] Cosine similarity: Calculates the cosine value of the angle between two vectors, which is often used to calculate the similarity of text data.

[0117] Euclidean distance: measures the straight-line distance between two vectors in Euclidean space, and is often used to calculate the similarity of image feature vectors.

[0118] Jaccard similarity: used to measure the ratio of the intersection and union of two sets, suitable for comparing the similarity of set data.

[0119] Structural Similarity Index (SSIM): used to measure the similarity between two images in terms of structure, brightness, and contrast.

[0120] Metric learning: The similarity metric between samples is learned by training the model so that similar samples are closer in the feature space and dissimilar samples are farther away.

[0121] Siamese network: used to learn the similarity between input data pairs, often used to train feature extractors to cluster similar samples in the feature space.

[0122] Faiss: Facebook AI Similarity Search, a library for efficient similarity search and dense vector clustering, supporting similarity search on large-scale datasets.

[0123] TensorFlow Similarity: A library dedicated to similarity learning or metric learning, providing a complete set of algorithms and tools for metric learning.

[0124] Furthermore, the data warehouse synchronizes the incremental blacklist data within the period to the blacklist system according to the target partition field according to the preset period, specifically:

[0125] The data warehouse uses the target partition field as a mapping to establish a database connection with the blacklist system;

[0126] The blacklist system sends an incremental data scheduling instruction to the data warehouse according to a preset period, and the incremental data scheduling instruction carries the first target partition field value;

[0127] The data warehouse synchronizes the corresponding incremental blacklist data to the blacklist system according to the first target partition field value.

[0128] It should be noted that a database connection is established in order to synchronize data.

[0129] For example: Use JDBC to connect to the data warehouse and blacklist system, and configure the database driver and connection string.

[0130] The blacklist system sends a dispatch instruction requesting data for a specific partition field value.

[0131] Example: Send a request to synchronize the blacklist data with partition_date = '2024-08-29'.

[0132] Furthermore, before the blacklist system synchronizes the incremental blacklist data to the associated party system, the following steps are also included:

[0133] The blacklist system sends a data volume query instruction to the data warehouse, wherein the data volume query instruction carries a second target partition field value;

[0134] The data warehouse counts the first data volume of the stored incremental blacklist data according to the second target partition field value and returns it to the blacklist system;

[0135] The blacklist system counts a second amount of incremental blacklist data stored according to the second target partition field value;

[0136] The blacklist system compares the first data volume with the second data volume for consistency. If they are inconsistent, a data missing warning and data compensation processing will be performed.

[0137] Furthermore, the data compensation process is specifically as follows:

[0138] The blacklist system deletes the incremental blacklist data stored corresponding to the second target partition field value;

[0139] The blacklist system resends the incremental data scheduling instruction to the data warehouse, and the incremental data scheduling instruction carries the second target partition field value, so that the data warehouse synchronizes the corresponding incremental blacklist data to the blacklist system according to the second target partition field value.

[0140] It should be noted that when comparing the data volumes, warnings and compensation will be issued if inconsistencies are found.

[0141] For example, if the data volume comparison on August 29 is found to be inconsistent, an alert is triggered, and then the missing data is resynchronized from the data warehouse.

[0142] Furthermore, the blacklist system synchronizes incremental blacklist data to the associated party system, specifically including:

[0143] The blacklist system synchronizes incremental blacklist data to the associated party system in real time based on the real-time call request of the associated party system;

[0144] or

[0145] The blacklist system sends a message containing incremental blacklist data to the message queue, and the associated party system consumes the message containing incremental blacklist data from the message queue.

[0146] It should be noted that data is synchronized in real time according to the request of the associated party system, or asynchronously through a message queue.

[0147] For example: the associated party system initiates a real-time call request, and the blacklist system immediately returns the latest blacklist data; or sends data to the message queue, and the associated party system listens to the queue and processes the message.

[0148] Based on the blacklist data processing method provided in the embodiment of the present application, the specific application process is as follows:

[0149] Data preprocessing: First, the data preprocessing system obtains original policy data from multiple data sources of the insurance company (such as customer management system, claims system, etc.). This data may contain customer personal information, policy details, claims history, etc.

[0150] Data conversion and storage: The preprocessing system converts the original policy data into a unified format and stores it in a graph database. The graph database can create entity nodes such as customers, policies, and claims events based on the original policy data, and establish relationships between these nodes, such as the relationship between customers and policies, and the relationship between policies and claims events.

[0151] Feature extraction and similarity calculation: The data preprocessing system calls a preset AI model, which extracts features based on nodes and relationships in the graph database. The AI ​​model may use deep learning technology to identify patterns and anomalies in policy data. By calculating the similarity between policies, policy combinations that may be fraudulent can be identified.

[0152] Blacklist data determination: If the similarity between policies is higher than the preset threshold, or there are other fraud indicators, these policy information will be marked as blacklist data. This data will be used for subsequent risk assessment and fraud prevention.

[0153] Data synchronization: The data warehouse will synchronize incremental blacklist data to the blacklist system based on a preset period (such as daily or weekly). This step may involve updating the data partition table and establishing a database connection with the blacklist system.

[0154] Data consistency check: Before synchronizing data, the blacklist system may send a data volume query instruction to the data warehouse to ensure data consistency. If the data volume is inconsistent, the system will trigger an early warning and perform data compensation processing.

[0155] Synchronize to the associated system: Finally, the blacklist system will synchronize the incremental blacklist data to the associated system based on the real-time call request. This can be achieved through direct API calls or asynchronous message delivery through message queues.

[0156] In the whole process, message queues (such as RabbitMQ, Apache Kafka, etc.) play an important role. They ensure the reliable transmission of data between different systems, provide asynchronous processing capabilities, and enhance the scalability and fault tolerance of the system. For example, message queues can be used to process order information, log data, cache updates, message notifications and other services, thereby improving the efficiency and stability of the entire insurance system.

[0157] In addition, graph database technologies (such as NebulaGraph, TigerGraph, etc.) also play an important role in the field of insurance anti-fraud. They help insurance companies identify fraud, optimize agent team management, and improve the efficiency of risk control by building a relationship network between entities such as policies, customers, and claims.

[0158] Through such systematic processing, the insurance system can more effectively manage and prevent fraud risks and protect the rights and interests of companies and customers. At the same time, it also provides insurance companies with data-driven decision support to help them stay competitive in the fiercely competitive market.

[0159] See also Figure 2 , Figure 2 Schematic diagram of the structure of the blacklist data processing system in the embodiment of the present application. Figure 2 As shown, it specifically includes: a data warehouse and a blacklist system, the data warehouse 201 is connected in communication with the blacklist system 202, wherein:

[0160] Constructing a data partition table in the data warehouse 201, wherein the data partition table stores initial blacklist data;

[0161] When executing a data processing task, the data warehouse 201 receives the full amount of blacklist data processed by the preset AI model, and the data warehouse 201 determines the incremental blacklist data in the full amount of blacklist data according to the preset identifier, and adds the target partition field to the incremental blacklist data, and stores it in the database table of the data warehouse 201;

[0162] The data warehouse 201 synchronizes the incremental blacklist data within the period to the blacklist system 202 according to the target partition field according to the preset period, and the blacklist system 202 synchronizes the incremental blacklist data to the associated party system 204 .

[0163] Furthermore, the system further includes a data preprocessing system 203, which is in communication with the data warehouse 201, wherein: before executing the data processing task, the system further includes:

[0164] The data preprocessing system 203 obtains original insurance policy data, which includes information of several original insurance policies;

[0165] The data preprocessing system 203 converts the original policy data into a target format and stores it in a graph database;

[0166] The graph database creates entity corresponding nodes based on the original policy data, establishes the relationship between the nodes based on the original policy information, and marks the nodes with the same information accordingly;

[0167] The data preprocessing system 203 calls the preset AI model, performs feature extraction and similarity calculation based on the nodes created in the graph database and the relationships between the nodes, and determines the full amount of blacklist data in the original insurance policy data.

[0168] Furthermore, the data preprocessing system 203 calls the preset AI model, performs feature extraction and similarity calculation based on the nodes created in the graph database and the relationships between the nodes, and determines that the full amount of blacklist data in the original insurance policy data is specifically:

[0169] The data preprocessing system 203 calls a preset AI model, in which a preset feature list is constructed;

[0170] The data preprocessing system 203 extracts target features of nodes created in the graph database through a preset AI model, and calculates the similarity between the original insurance policies based on the relationship between the nodes;

[0171] If the similarity is higher than a preset threshold, the data preprocessing system 203 marks the information of the corresponding two or more original insurance policies as blacklist data;

[0172] The data preprocessing system 203 counts the full amount of blacklist data in the original insurance policy data.

[0173] Furthermore, the data warehouse 201 synchronizes the incremental blacklist data within the period to the blacklist system 202 according to the target partition field according to the preset period, specifically:

[0174] The data warehouse 201 uses the target partition field as a mapping to establish a database connection with the blacklist system 202;

[0175] The blacklist system 202 sends an incremental data scheduling instruction to the data warehouse 201 according to a preset period, and the incremental data scheduling instruction carries the first target partition field value;

[0176] The data warehouse 201 synchronizes the corresponding incremental blacklist data to the blacklist system 202 according to the first target partition field value.

[0177] Furthermore, before the blacklist system 202 synchronizes the incremental blacklist data to the associated party system 204, the following steps are also included:

[0178] The blacklist system 202 sends a data volume query instruction to the data warehouse 201, wherein the data volume query instruction carries the second target partition field value;

[0179] The data warehouse 201 counts the first data volume of the stored incremental blacklist data according to the second target partition field value and returns it to the blacklist system 202;

[0180] The blacklist system 202 counts a second data volume of the stored incremental blacklist data according to the second target partition field value;

[0181] The blacklist system 202 compares the first data volume with the second data volume for consistency, and if they are inconsistent, performs data missing warning and data compensation processing.

[0182] Furthermore, the data compensation process is specifically as follows:

[0183] The blacklist system 202 deletes the incremental blacklist data stored corresponding to the second target partition field value;

[0184] The blacklist system 202 resends the incremental data scheduling instruction to the data warehouse 201 , and the incremental data scheduling instruction carries the second target partition field value, so that the data warehouse 201 synchronizes the corresponding incremental blacklist data to the blacklist system 202 according to the second target partition field value.

[0185] Furthermore, the blacklist system 202 synchronizes the incremental blacklist data to the associated party system 204 specifically including:

[0186] The blacklist system 202 synchronizes incremental blacklist data to the associated party system 204 in real time according to the real-time call request of the associated party system 204;

[0187] or

[0188] The blacklist system 202 sends a message containing incremental blacklist data to the message queue, and the associated party system 204 consumes the message containing incremental blacklist data from the message queue.

[0189] The present application embodiment also provides another blacklist data processing device, such as Figure 3 For the sake of convenience, only the parts related to the embodiments of the present application are shown. For specific technical details not disclosed, please refer to the method part of the embodiments of the present application. The terminal can be any terminal device including a computer, a tablet computer, a personal digital assistant (English full name: Personal Digital Assistant, English abbreviation: PDA), a sales terminal (English full name: Point of Sales, English abbreviation: POS), a car computer, etc., taking the terminal as a computer as an example:

[0190] Figure 3 FIG. 1 is a block diagram showing a partial structure of a computer related to a terminal provided in an embodiment of the present application. Figure 3 The computer includes: a radio frequency (RF) circuit 1010, a memory 1020, an input unit 1030, a display unit 1040, a sensor 1050, an audio circuit 1060, a wireless fidelity (WiFi) module 1070, a processor 1080, and a power supply 1090. Those skilled in the art can understand that Figure 3 The computer structure shown in the figure does not constitute a limitation of the computer, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0191] Combine the following Figure 3 A detailed introduction to the various components of a computer:

[0192] The RF circuit 1010 can be used for receiving and sending signals during information transmission or calls. In particular, after receiving the downlink information of the base station, it is sent to the processor 1080 for processing; in addition, the designed uplink data is sent to the base station. Generally, the RF circuit 1010 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (full name: Low Noise Amplifier, English abbreviation: LNA), a duplexer, etc. In addition, the RF circuit 1010 can also communicate with the network and other devices through wireless communication. The above-mentioned wireless communications can use any communication standard or protocol, including but not limited to Global System of Mobile communications (Global System of Mobile communication, English abbreviation: GSM), General Packet Radio Service (General Packet Radio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, English abbreviation: CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (Long Term Evolution, English abbreviation: LTE), e-mail, Short Messaging Service (SMS), etc.

[0193] The memory 1020 can be used to store software programs and modules. The processor 1080 executes various functional applications and data processing of the computer by running the software programs and modules stored in the memory 1020. The memory 1020 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area can store data created according to the use of the computer (such as audio data, a phone book, etc.), etc. In addition, the memory 1020 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0194] The input unit 1030 can be used to receive input digital or character information, and to generate key signal input related to the user settings and function control of the computer. Specifically, the input unit 1030 may include a touch panel 1031 and other input devices 1032. The touch panel 1031, also known as a touch screen, can collect the user's touch operation on or near it (such as the user's operation on the touch panel 1031 or near the touch panel 1031 using any suitable object or accessory such as a finger, stylus, etc.), and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 1031 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into the touch point coordinates, and then sends it to the processor 1080, and can receive and execute the command sent by the processor 1080. In addition, the touch panel 1031 can be implemented in various types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch panel 1031, the input unit 1030 may further include other input devices 1032. Specifically, the other input devices 1032 may include but are not limited to one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.

[0195] The display unit 1040 can be used to display information input by the user or information provided to the user and various menus of the computer. The display unit 1040 may include a display panel 1041. Optionally, the display panel 1041 may be configured in the form of a liquid crystal display (full name in English: Liquid Crystal Display, English abbreviation: LCD), an organic light-emitting diode (full name in English: Organic Light-Emitting Diode, English abbreviation: OLED), etc. Further, the touch panel 1031 may cover the display panel 1041. When the touch panel 1031 detects a touch operation on or near it, it is transmitted to the processor 1080 to determine the type of touch event. Subsequently, the processor 1080 provides a corresponding visual output on the display panel 1041 according to the type of touch event. Although in Figure 3 In the embodiment, the touch panel 1031 and the display panel 1041 are used as two independent components to realize the input and output functions of the computer, but in some embodiments, the touch panel 1031 and the display panel 1041 can be integrated to realize the input and output functions of the computer.

[0196] The computer may also include at least one sensor 1050, such as a light sensor, a motion sensor, and other sensors. Specifically, the light sensor may include an ambient light sensor and a proximity sensor, wherein the ambient light sensor may adjust the brightness of the display panel 1041 according to the brightness of the ambient light, and the proximity sensor may turn off the display panel 1041 and / or the backlight when the computer is moved to the ear. As a type of motion sensor, the accelerometer sensor can detect the magnitude of acceleration in all directions (generally three axes), and can detect the magnitude and direction of gravity when stationary, and can be used for applications that identify computer postures (such as horizontal and vertical screen switching, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometers, tapping), etc.; as for other sensors that can also be configured on the computer, such as gyroscopes, barometers, hygrometers, thermometers, infrared sensors, etc., they will not be repeated here.

[0197] The audio circuit 1060, the speaker 1061, and the microphone 1062 can provide an audio interface between the user and the computer. The audio circuit 1060 can transmit the received audio data to the speaker 1061 after converting the received audio data into an electrical signal, which is converted into a sound signal for output; on the other hand, the microphone 1062 converts the collected sound signal into an electrical signal, which is received by the audio circuit 1060 and converted into audio data, and then the audio data is output to the processor 1080 for processing, and then sent to another computer through the RF circuit 1010, or the audio data is output to the memory 1020 for further processing.

[0198] WiFi is a short-range wireless transmission technology. The computer can help users send and receive emails, browse web pages and access streaming media through the WiFi module 1070. It provides users with wireless broadband Internet access. Figure 3 A WiFi module 1070 is shown, but it is understandable that it is not an essential component of the computer and can be omitted as needed without changing the essence of the invention.

[0199] The processor 1080 is the control center of the computer. It uses various interfaces and lines to connect various parts of the entire computer. By running or executing software programs and / or modules stored in the memory 1020 and calling data stored in the memory 1020, it executes various functions of the computer and processes data, thereby monitoring the computer as a whole. Optionally, the processor 1080 may include one or more processing units; preferably, the processor 1080 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 1080.

[0200] The computer also includes a power supply 1090 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 1080 through a power management system, so that the power management system can manage functions such as charging, discharging, and power consumption.

[0201] Although not shown, the computer may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0202] In the embodiment of the present application, the processor 1080 included in the terminal also has the following functions:

[0203] A data partition table is constructed in the data warehouse, and the data partition table stores initial blacklist data;

[0204] When executing a data processing task, the data warehouse receives the full blacklist data processed by the preset AI model, determines the incremental blacklist data in the full blacklist data according to the preset identifier, adds the target partition field to the incremental blacklist data, and stores it in the database table of the data warehouse;

[0205] The data warehouse synchronizes the incremental blacklist data within the period to the blacklist system according to the target partition field according to the preset period, and the blacklist system synchronizes the incremental blacklist data to the associated party system.

[0206] An embodiment of the present application also provides a computer-readable storage medium for storing program code, wherein the program code is used to execute any one of the implementation methods of a blacklist data processing method described in the aforementioned embodiments.

[0207] In an embodiment of the present application, a blacklist data processing method, system, device and readable storage medium are provided. The full blacklist data processed by a preset AI model is reprocessed in a data warehouse to determine incremental blacklist data. The data warehouse directly synchronizes the incremental blacklist data to the blacklist system according to a preset period, thereby avoiding the situation where an extra copy of the data is stored in the middle through the risk control system, and reducing the system call link. Furthermore, the blacklist system can compare the data volume based on each scheduling with the data warehouse to find out whether there is a problem of missing numbers. In addition, a data compensation processing mechanism is also provided to flexibly make up numbers, which solves the resource waste problem caused by full data synchronization in the existing solution and the data redundancy problem existing in multiple systems storing the same data.

[0208] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0209] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein, for example. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0210] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0211] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0212] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0213] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0214] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (full name in English: Read-Only Memory, English abbreviation: ROM), random access memory (full name in English: Random Access Memory, English abbreviation: RAM), disk or optical disk and other media that can store program codes.

[0215] As described above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A blacklist data processing method, characterized in that: include: Constructing a data partition table in a data warehouse, wherein the data partition table stores initial blacklist data; When executing a data processing task, the data warehouse receives the full blacklist data processed by a preset AI model, determines the incremental blacklist data in the full blacklist data according to a preset identifier, adds a target partition field to the incremental blacklist data, and stores it in a database table of the data warehouse; The data warehouse synchronizes the incremental blacklist data within the period to the blacklist system according to the target partition field according to a preset period, and the blacklist system synchronizes the incremental blacklist data to the associated party system.

2. The blacklist data processing method according to claim 1, characterized in that: Before executing the data processing task, the following steps are also included: The data preprocessing system obtains original insurance policy data, wherein the original insurance policy data includes information of several original insurance policies; The data preprocessing system converts the original policy data into a target format and stores the converted data in a graph database; The graph database creates entity corresponding nodes according to the original insurance policy data, establishes relationships between the nodes based on the information of the original insurance policy, and marks the nodes with the same information accordingly; The data preprocessing system calls a preset AI model to perform feature extraction and similarity calculation based on the nodes created in the graph database and the relationships between the nodes, and determines the full amount of blacklist data in the original policy data.

3. The blacklist data processing method according to claim 2, characterized in that: The data preprocessing system calls the preset AI model, performs feature extraction and similarity calculation based on the nodes created in the graph database and the relationships between the nodes, and determines that the full amount of blacklist data in the original insurance policy data is specifically: Calling a preset AI model, wherein the preset AI model has a preset feature list; Extracting the target features of the nodes created in the graph database through the preset AI model, and calculating the similarity between the original insurance policies based on the relationship between the nodes; If the similarity is higher than a preset threshold, the information of the two or more correspondingly associated original insurance policies is marked as blacklist data; The total amount of blacklist data in the original policy data is counted.

4. The blacklist data processing method according to claim 1, characterized in that: The data warehouse synchronizes the incremental blacklist data within the period to the blacklist system according to the target partition field according to the preset period, specifically: The data warehouse uses the target partition field as a mapping to establish a database connection with the blacklist system; The blacklist system sends an incremental data scheduling instruction to the data warehouse according to a preset period, wherein the incremental data scheduling instruction carries a first target partition field value; The data warehouse synchronizes the corresponding incremental blacklist data to the blacklist system according to the first target partition field value.

5. The blacklist data processing method according to claim 1, characterized in that: Before the blacklist system synchronizes the incremental blacklist data to the associated party system, the following further includes: The blacklist system sends a data volume query instruction to the data warehouse, wherein the data volume query instruction carries a second target partition field value; The data warehouse counts the first data amount stored in the incremental blacklist data according to the second target partition field value and returns it to the blacklist system; The blacklist system counts a second amount of data stored in the incremental blacklist data according to the second target partition field value; The blacklist system compares the first data volume with the second data volume for consistency, and if they are inconsistent, performs data missing warning and data compensation processing.

6. The blacklist data processing method according to claim 5, characterized in that: The data compensation process is specifically as follows: The blacklist system deletes the incremental blacklist data stored corresponding to the second target partition field value; The blacklist system re-sends the incremental data scheduling instruction to the data warehouse, and the incremental data scheduling instruction carries the second target partition field value, so that the data warehouse synchronizes the corresponding incremental blacklist data to the blacklist system according to the second target partition field value.

7. The blacklist data processing method according to claim 1, characterized in that: The blacklist system synchronizing the incremental blacklist data to the associated party system specifically includes: The blacklist system synchronizes the incremental blacklist data to the associated party system in real time according to the real-time call request of the associated party system; or The blacklist system sends a message containing the incremental blacklist data to a message queue, and the associated party system consumes the message containing the incremental blacklist data from the message queue.

8. A blacklist data processing system, characterized in that: include: A data warehouse and a blacklist system, wherein the data warehouse is in communication with the blacklist system, wherein: Constructing a data partition table in the data warehouse, wherein the data partition table stores initial blacklist data; When executing a data processing task, the data warehouse receives the full blacklist data processed by a preset AI model, determines the incremental blacklist data in the full blacklist data according to a preset identifier, adds a target partition field to the incremental blacklist data, and stores it in a database table of the data warehouse; The data warehouse synchronizes the incremental blacklist data within the period to the blacklist system according to the target partition field according to a preset period, and the blacklist system synchronizes the incremental blacklist data to the associated party system.

9. A blacklist data processing device, characterized in that: The device comprises a processor and a memory: The memory is used to store program codes and transmit the program codes to the processor; The processor is used to execute the blacklist data processing method according to any one of claims 1 to 7 according to the instructions in the program code.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program code, and the program code is used to execute the blacklist data processing method according to any one of claims 1 to 7.

Images