Data extraction method, device, equipment and storage medium

By extracting methods in the intelligent risk control system, the original data is obtained, converted and encrypted, and the extraction rules are loaded to extract feature indicators, the problem of slow data processing speed is solved, fast access and efficient processing are achieved, and system performance is improved.

CN112463799BActive Publication Date: 2024-11-08TIANMIAN INFORMATION TECHNOLOGY (SHENZHEN) CO LTD
View PDF 6 Cites 0 Cited by

Patent Information

Application Number
CN202011453102.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-12-11
Publication Date
2024-11-08
Estimated Expiration
2040-12-11

AI Technical Summary

Technical Problem

In the intelligent risk control system of personal credit products, it is difficult for the existing technology to quickly extract characteristic indicator data, resulting in slow data processing speed and different data from different authorization channels, resulting in high data processing complexity.

Method used

By obtaining raw data from the message queue every preset time period, converting it to a standard format and encrypting it, storing it to a database, loading the extraction rules to extract feature indicator data, and performing backup and storage operations based on configuration information, decoupling data processing and writing Enter the operation to improve the data processing speed.

Benefits of technology

It realizes fast data access and processing, reduces programming complexity, improves the system's processing capabilities, and can complete the extraction of feature indicators in milliseconds and output risk control results in seconds.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112463799B_ABST
    Figure CN112463799B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of data processing, and provides a data extraction method, device, equipment and storage medium. The method comprises: obtaining original data and the data type of the original data from a preset message queue at preset time intervals; converting the original data into standard format data based on the data type of the original data, performing encryption operation on sensitive information in the standard format data, and storing the encrypted standard format data in a first data table of a preset database; based on the data type of the standard format data, loading the extraction rules corresponding to the data type, and extracting the characteristic indicator data of the standard format data based on the extraction rules. The present invention can decouple data standardization and characteristic indicator extraction, realize rapid data access, reduce programming complexity, and thus quickly respond to business changes.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of data processing, and in particular to a data extraction method, device, equipment and storage medium. Background Art

[0002] In the intelligent risk control system applied to personal credit products, it is necessary to analyze the user's economic status from multiple dimensions to predict the user's repayment ability, repayment willingness and fraud risk. For example, it is necessary to analyze the user's age, education, occupation, family, credit information, etc.

[0003] Currently, the system mainly has data filled in by users themselves, third-party data obtained with user authorization, or user behavior data collected with user authorization. However, these raw data cannot be used directly as feature indicators, and the same type of information may have many different authorization channels, resulting in different raw data for the same type of information. Due to factors such as the large user base and the diversity of data types, the system needs to process a large amount of data, and the processed data is highly dependent on other related systems, resulting in slow data processing. Summary of the invention

[0004] In view of the above, the present invention provides a data extraction method, device, equipment and storage medium, which aims to solve the technical problem of slow data processing speed when extracting characteristic index values ​​in the prior art.

[0005] To achieve the above object, the present invention provides a data extraction method, which comprises:

[0006] Obtaining original data and the data type of the original data from a preset message queue at every preset time period;

[0007] Converting the original data into standard format data based on the data type of the original data, performing encryption operation on sensitive information in the standard format data, and storing the encrypted standard format data in a first data table of a preset database;

[0008] Based on the data type of the standard format data, an extraction rule corresponding to the data type is loaded, and characteristic indicator data of the standard format data is extracted based on the extraction rule.

[0009] Preferably, after obtaining the original data and the data type of the original data, the method further includes:

[0010] Determining whether it is necessary to perform a backup operation and a storage operation on the original data according to the preset configuration information, and when it is determined that it is necessary to perform a backup operation on the original data, sending the original data to a preset backup path of the message queue;

[0011] When it is determined that a storage operation needs to be performed on the original data, an encryption operation is performed on the sensitive information in the original data, and the encrypted original data is stored in a second data table of a preset database.

[0012] Preferably, after converting the original data into standard format data based on the data type of the original data, the method further comprises:

[0013] It is determined whether there is data of a preset type in the standard format data, and when it is determined that there is data of a preset type in the standard format data, the data of the preset type is sent to the message queue.

[0014] Preferably, after extracting the characteristic indicator data of the standard format data based on the extraction rule, the method further includes:

[0015] It is determined whether the characteristic indicator data needs to be sent to a preset storage path according to the message header of the original data corresponding to the characteristic indicator data, and if so, the characteristic indicator data is sent to the preset storage path.

[0016] Preferably, the performing encryption operation on the sensitive information in the standard format data includes:

[0017] Sensitive information that needs to be encrypted is acquired according to a pre-configured encryption configuration table, and the encryption operation is performed on the sensitive information, wherein the encryption configuration table records the fields that need to be encrypted.

[0018] To achieve the above object, the present invention further provides a data extraction device, the data extraction device comprising:

[0019] Acquisition module: used to acquire original data and the data type of the original data from a preset message queue every preset time period;

[0020] A conversion module: used for converting the original data into standard format data based on the data type of the original data, performing an encryption operation on sensitive information in the standard format data, and storing the encrypted standard format data in a first data table of a preset database;

[0021] Extraction module: used for loading extraction rules corresponding to the data type based on the data type of the standard format data, and extracting characteristic indicator data of the standard format data based on the extraction rules.

[0022] To achieve the above object, the present invention further provides an electronic device, the electronic device comprising:

[0023] at least one processor; and,

[0024] a memory communicatively connected to the at least one processor; wherein,

[0025] The memory stores a program executable by the at least one processor, and the program is executed by the at least one processor so that the at least one processor can perform the following steps:

[0026] Obtaining original data and the data type of the original data from a preset message queue at every preset time period;

[0027] Converting the original data into standard format data based on the data type of the original data, performing encryption operation on sensitive information in the standard format data, and storing the encrypted standard format data in a first data table of a preset database;

[0028] Based on the data type of the standard format data, an extraction rule corresponding to the data type is loaded, and characteristic indicator data of the standard format data is extracted based on the extraction rule.

[0029] Preferably, after obtaining the original data and the data type of the original data, the following steps are also included:

[0030] Determining whether it is necessary to perform a backup operation and a storage operation on the original data according to the preset configuration information, and when it is determined that it is necessary to perform a backup operation on the original data, sending the original data to a preset backup path of the message queue;

[0031] When it is determined that a storage operation needs to be performed on the original data, an encryption operation is performed on the sensitive information in the original data, and the encrypted original data is stored in a second data table of a preset database.

[0032] Preferably, after converting the original data into standard format data based on the data type of the original data, the method further includes the following steps:

[0033] It is determined whether there is data of a preset type in the standard format data, and when it is determined that there is data of a preset type in the standard format data, the data of the preset type is sent to the message queue.

[0034] To achieve the above object, the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a data extraction program, and when the data extraction program is executed by a processor, any step of the data extraction method described above is implemented.

[0035] The data extraction method, device, equipment and storage medium proposed in the present invention only need to standardize data from different sources and extract feature data to complete the calculation of a certain type of data, which greatly reduces the complexity and workload of the code. When it is necessary to access data from a new channel, the corresponding data standardization class can be developed to successfully complete the data extraction process. Since the system often needs to adjust the calculation logic of the risk control index, it only needs to modify the feature data extraction logic. Decoupling data standardization and feature index extraction can achieve rapid data access, reduce programming complexity, and quickly respond to business changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 A schematic diagram of a flow chart of a preferred embodiment of the data extraction method of the present invention;

[0037] Figure 2 A schematic diagram of a module of a preferred embodiment of a data extraction device of the present invention;

[0038] Figure 3 A schematic diagram of a preferred embodiment of the electronic device of the present invention;

[0039] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0041] The present invention provides a data extraction method. Figure 1 FIG. 1 is a flow chart of a method of an embodiment of a data extraction method of the present invention. The method may be executed by an electronic device, and the electronic device may be implemented by software and / or hardware. The data extraction method includes:

[0042] Step S10: acquiring original data and the data type of the original data from a preset message queue at every preset time period.

[0043] The application scenario of this solution can be to extract characteristic indicator data from data obtained from multiple data sources for risk assessment. It should be noted that the actual application scenario of this solution is not limited to this. In the risk control system of personal credit products, it is necessary to analyze the user's economic situation from multiple dimensions to predict the user's repayment ability, repayment willingness and fraud risk. For example, it is necessary to analyze the user's age, education, occupation, family, assets, liabilities, social security information, provident fund information, and credit information of the People's Bank of China.

[0044] There are three main types of data sources for obtaining user information: loan application data filled out by users themselves, third-party data obtained with user authorization, and user behavior data collected with user authorization. These raw data cannot be used directly as risk control feature indicators, and the same type of information may have many different authorization channels, resulting in different raw data for the same type of information. For example, for provident fund data, user A chooses to authorize data A1 from K1, and user B chooses to authorize data A2 from K2. For the risk control system, it is necessary to process the same type of data from different data sources into a unified data format, and then extract relevant feature indicator data from the unified format data for risk assessment.

[0045] In this embodiment, after the upstream system obtains the user's authorization data in real time, it constructs a corresponding message header according to the characteristics of the authorization data, puts the authorization data into the message body, and sends the authorization data to the message queue (for example, the corresponding Topic of Kafka), which can achieve decoupling between systems. After that, it monitors the authorization data in the corresponding Topic in Kafka. Topic is the basic unit of Kafka data writing operation, and uses micro batch processing to obtain a batch of authorization data as raw data for processing every preset time period (for example, x milliseconds).

[0046] In one embodiment, after obtaining the original data and the data type of the original data, the method further includes:

[0047] Determine whether to perform a backup operation and a storage operation on the original data according to the preset configuration information, and when it is determined that the backup operation needs to be performed on the original data, send the original data to a preset backup path (for example, a Kafka backup Topic) of the message queue;

[0048] When it is determined that a storage operation needs to be performed on the original data, an encryption operation is performed on the sensitive information in the original data, and the encrypted original data is stored in a second data table of a preset database.

[0049] The preset configuration information can be configured according to specific business needs. The configuration center stores a list of data types that do not need to be backed up. The data types in the list do not need to be backed up, and the rest of the data needs to be backed up by default. Send the data to the Kafka backup Topic, and another auxiliary system is responsible for writing the data to HDFS. The HDFS write operation can be decoupled from data processing to increase the speed of data processing. Sensitive information can be personal privacy data such as user ID number, mobile phone number, address, etc. The second data table of the preset database can refer to the original data table corresponding to cassandra.

[0050] Step S20: converting the original data into standard format data based on the data type of the original data, performing encryption operation on sensitive information in the standard format data, and storing the encrypted standard format data in a first data table of a preset database.

[0051] In this embodiment, the original data is converted into standard format data according to the data type of the original data, for example, the unified provident fund data format, social security data format, etc. within the company. After the original data is standardized, the sensitive information in the standard format data is encrypted and then the entire standard format data is stored in the corresponding standardized data table of Cassandra.

[0052] In one embodiment, performing encryption operation on sensitive information in the standard format data includes:

[0053] Sensitive information that needs to be encrypted is acquired according to a pre-configured encryption configuration table, and the encryption operation is performed on the sensitive information, wherein the encryption configuration table records the fields that need to be encrypted.

[0054] Sensitive information can be personal privacy data such as user ID number, mobile phone number, address, etc. You can determine whether the data needs to be encrypted based on the pre-configured encryption configuration table. The encryption configuration table records which fields in which table need to be encrypted. When storing data, it will first determine whether the table contains information corresponding to the encryption configuration table. If so, the sensitive information will be encrypted and the entire standard format data will be stored in the database. When reading data, after reading the data, it will determine whether the read data contains information corresponding to the encryption configuration table. If so, the data will be decrypted and returned. The encryption and decryption operation is processed through the system's proxy mode. Business processing does not need to pay attention to this operation. The data that needs to be encrypted and decrypted only needs to be configured in the encryption configuration table.

[0055] In one embodiment, after converting the original data into standard format data based on the data type of the original data, the method further includes:

[0056] It is determined whether there is data of a preset type in the standard format data, and when it is determined that there is data of a preset type in the standard format data, the data of the preset type is sent to the message queue.

[0057] If the standard format data contains information used for a certain type of data search (for example, address information), the information used for data search is extracted and sent to the message queue of the Kafka search topic. Another auxiliary system writes the data to the search server (for example, Elastic Search) for information search. This can decouple the write operation of Elastic Search from data processing, further improving the speed of data processing.

[0058] Step S30: Based on the data type of the standard format data, an extraction rule corresponding to the data type is loaded, and characteristic indicator data of the standard format data is extracted based on the extraction rule.

[0059] In this embodiment, according to the data type of the standardized standard format data, the corresponding feature indicator extraction rules are loaded to extract the feature indicator data, wherein the feature indicator extraction rules correspond to different feature indicator extraction processing classes, and each type of standard format data has a corresponding mapping relationship with the feature indicator extraction processing class.

[0060] The processing class corresponding to the data type will eventually generate a data model, and the corresponding feature indicator extraction processing class will be loaded according to the model. The extraction processing class will declare which data model to process when it is defined. When the system starts, it will scan the extraction processing class with a specific naming specification, put the corresponding relationship between the model and the processing class into the map and load it into the memory. Subsequently, the feature processing class is directly obtained from the map in the memory according to the model. The system loads the corresponding feature indicator extraction processing class, extracts the corresponding risk control feature indicator data, encrypts the sensitive information in the data, and then writes the data into the feature indicator data table in Cassandra. The data in this table is an important reference data for the system to make risk control decisions.

[0061] In one embodiment, after extracting the characteristic indicator data of the standard format data based on the extraction rule, the method further includes:

[0062] It is determined whether the characteristic indicator data needs to be sent to a preset storage path according to the message header of the original data corresponding to the characteristic indicator data, and if so, the characteristic indicator data is sent to the preset storage path.

[0063] The preset storage path may refer to a storage path of a downstream system (eg, a system that needs to be evaluated based on characteristic indicator data).

[0064] The message header contains the data type of the original data. Whether the characteristic indicator data needs to be sent to the downstream system can be determined based on the data type. The data type of the characteristic indicator data that needs to be sent to the downstream system can be pre-configured according to actual business needs. For example, if callback is required for processing provident fund data, the header information needs to include the data type, data sequence ID, Kafka Topic name corresponding to the callback data, and callback parameters. If callback is not required for processing address book data, the header information needs to include the data type and data sequence ID.

[0065] The data standardization and feature data extraction processing of this solution only requires standardization of data from different sources and feature data extraction to complete the calculation of a certain type of data, which greatly reduces the complexity and workload of the code. When it is necessary to access data from a new channel, it is only necessary to develop the corresponding data standardization class to successfully complete the data processing. Since the system often needs to adjust the calculation logic of risk control indicators, it is only necessary to modify the feature data extraction logic. Separating data standardization from risk control feature indicator extraction can achieve rapid data access, reduce programming complexity, and quickly respond to business changes.

[0066] In the subsequent data processing of the risk control system, other systems perform analysis and processing based on the standardized data, which greatly reduces the complexity of data processing and improves the processing capacity of the overall system. The distributed computing architecture based on spark stream processing is easy to scale horizontally. The number of computing nodes can be adjusted according to the amount of data to achieve effective resource utilization and high availability, high performance, and high concurrency of the system. By decoupling HDFS write operations and Elastic Search write operations from data processing, the processing speed of the system is greatly improved, allowing the system to complete the extraction of feature indicators within milliseconds and cooperate with other systems to output risk control results within seconds.

[0067] Reference Figure 2 FIG. 1 is a schematic diagram of the functional modules of the data extraction device 100 of the present invention.

[0068] The data extraction device 100 of the present invention can be installed in an electronic device. According to the functions to be implemented, the data extraction device 100 may include an acquisition module 110, a conversion module 120 and an extraction module 130. The module of the present invention may also be referred to as a unit, which refers to a series of computer program segments that can be executed by a processor of an electronic device and can complete fixed functions, and is stored in the memory of the electronic device.

[0069] In this embodiment, the functions of each module / unit are as follows:

[0070] The acquisition module 110 is used to acquire original data and the data type of the original data from a preset message queue every preset time period.

[0071] The application scenario of this solution can be to extract characteristic indicator data from data obtained from multiple data sources for risk assessment. It should be noted that the actual application scenario of this solution is not limited to this. In the risk control system of personal credit products, it is necessary to analyze the user's economic situation from multiple dimensions to predict the user's repayment ability, repayment willingness and fraud risk. For example, it is necessary to analyze the user's age, education, occupation, family, assets, liabilities, social security information, provident fund information, and credit information of the People's Bank of China.

[0072] There are three main types of data sources for obtaining user information: loan application data filled out by users themselves, third-party data obtained with user authorization, and user behavior data collected with user authorization. These raw data cannot be used directly as risk control feature indicators, and the same type of information may have many different authorization channels, resulting in different raw data for the same type of information. For example, for provident fund data, user A chooses to authorize data A1 from K1, and user B chooses to authorize data A2 from K2. For the risk control system, it is necessary to process the same type of data from different data sources into a unified data format, and then extract relevant feature indicator data from the unified format data for risk assessment.

[0073] In this embodiment, after the upstream system obtains the user's authorization data in real time, it constructs a corresponding message header according to the characteristics of the authorization data, puts the authorization data into the message body, and sends the authorization data to the message queue (for example, the corresponding Topic of Kafka), which can achieve decoupling between systems. After that, it monitors the authorization data in the corresponding Topic in Kafka. Topic is the basic unit of Kafka data writing operation, and uses micro batch processing to obtain a batch of authorization data as raw data for processing every preset time period (for example, x milliseconds).

[0074] In one embodiment, after obtaining the original data and the data type of the original data, the method further includes:

[0075] Determine whether to perform a backup operation and a storage operation on the original data according to the preset configuration information, and when it is determined that the backup operation needs to be performed on the original data, send the original data to a preset backup path (for example, a Kafka backup Topic) of the message queue;

[0076] When it is determined that a storage operation needs to be performed on the original data, an encryption operation is performed on the sensitive information in the original data, and the encrypted original data is stored in a second data table of a preset database.

[0077] The preset configuration information can be configured according to specific business needs. The configuration center stores a list of data types that do not need to be backed up. The data types in the list do not need to be backed up, and the rest of the data needs to be backed up by default. Send the data to the Kafka backup Topic, and another auxiliary system is responsible for writing the data to HDFS. The HDFS write operation can be decoupled from data processing to increase the speed of data processing. Sensitive information can be personal privacy data such as user ID number, mobile phone number, address, etc. The second data table of the preset database can refer to the original data table corresponding to cassandra.

[0078] The conversion module 120 is used to convert the original data into standard format data based on the data type of the original data, perform encryption operation on sensitive information in the standard format data, and store the encrypted standard format data in the first data table of the preset database.

[0079] In this embodiment, the original data is converted into standard format data according to the data type of the original data, for example, the unified provident fund data format, social security data format, etc. within the company. After the original data is standardized, the sensitive information in the standard format data is encrypted and then the entire standard format data is stored in the corresponding standardized data table of Cassandra.

[0080] In one embodiment, performing encryption operation on sensitive information in the standard format data includes:

[0081] Sensitive information that needs to be encrypted is acquired according to a pre-configured encryption configuration table, and the encryption operation is performed on the sensitive information, wherein the encryption configuration table records the fields that need to be encrypted.

[0082] Sensitive information can be personal privacy data such as user ID number, mobile phone number, address, etc. You can determine whether the data needs to be encrypted based on the pre-configured encryption configuration table. The encryption configuration table records which fields in which table need to be encrypted. When storing data, it will first determine whether the table contains information corresponding to the encryption configuration table. If so, the sensitive information will be encrypted and the entire standard format data will be stored in the database. When reading data, after reading the data, it will determine whether the read data contains information corresponding to the encryption configuration table. If so, the data will be decrypted and returned. The encryption and decryption operation is processed through the system's proxy mode. Business processing does not need to pay attention to this operation. The data that needs to be encrypted and decrypted only needs to be configured in the encryption configuration table.

[0083] In one embodiment, after converting the original data into standard format data based on the data type of the original data, the method further includes:

[0084] It is determined whether there is data of a preset type in the standard format data, and when it is determined that there is data of a preset type in the standard format data, the data of the preset type is sent to the message queue.

[0085] If the standard format data contains information used for a certain type of data search (for example, address information), the information used for data search is extracted and sent to the message queue of the Kafka search topic. Another auxiliary system writes the data to the search server (for example, Elastic Search) for information search. This can decouple the write operation of Elastic Search from data processing, further improving the speed of data processing.

[0086] The extraction module 130 is used to load an extraction rule corresponding to the data type of the standard format data based on the data type of the standard format data, and extract characteristic indicator data of the standard format data based on the extraction rule.

[0087] In this embodiment, according to the data type of the standardized standard format data, the corresponding feature indicator extraction rules are loaded to extract the feature indicator data, wherein the feature indicator extraction rules correspond to different feature indicator extraction processing classes, and each type of standard format data has a corresponding mapping relationship with the feature indicator extraction processing class.

[0088] The processing class corresponding to the data type will eventually generate a data model, and the corresponding feature indicator extraction processing class will be loaded according to the model. The extraction processing class will declare which data model to process when it is defined. When the system starts, it will scan the extraction processing class with a specific naming specification, put the corresponding relationship between the model and the processing class into the map and load it into the memory. Subsequently, the feature processing class is directly obtained from the map in the memory according to the model. The system loads the corresponding feature indicator extraction processing class, extracts the corresponding risk control feature indicator data, encrypts the sensitive information in the data, and then writes the data into the feature indicator data table in Cassandra. The data in this table is an important reference data for the system to make risk control decisions.

[0089] In one embodiment, after extracting the characteristic indicator data of the standard format data based on the extraction rule, the method further includes:

[0090] It is determined whether the characteristic indicator data needs to be sent to a preset storage path according to the message header of the original data corresponding to the characteristic indicator data, and if so, the characteristic indicator data is sent to the preset storage path.

[0091] The preset storage path may refer to a storage path of a downstream system (eg, a system that needs to be evaluated based on characteristic indicator data).

[0092] The message header contains the data type of the original data. Whether the characteristic indicator data needs to be sent to the downstream system can be determined based on the data type. The data type of the characteristic indicator data that needs to be sent to the downstream system can be pre-configured according to actual business needs. For example, if callback is required for processing provident fund data, the header information needs to include the data type, data sequence ID, Kafka Topic name corresponding to the callback data, and callback parameters. If callback is not required for processing address book data, the header information needs to include the data type and data sequence ID.

[0093] The data standardization and feature data extraction processing of this solution only requires standardization of data from different sources and feature data extraction to complete the calculation of a certain type of data, which greatly reduces the complexity and workload of the code. When it is necessary to access data from a new channel, it is only necessary to develop the corresponding data standardization class to successfully complete the data processing. Since the system often needs to adjust the calculation logic of risk control indicators, it is only necessary to modify the feature data extraction logic. Separating data standardization from risk control feature indicator extraction can achieve rapid data access, reduce programming complexity, and quickly respond to business changes.

[0094] In the subsequent data processing of the risk control system, other systems perform analysis and processing based on the standardized data, which greatly reduces the complexity of data processing and improves the processing capacity of the overall system. The distributed computing architecture based on spark stream processing is easy to scale horizontally. The number of computing nodes can be adjusted according to the amount of data to achieve effective resource utilization and high availability, high performance, and high concurrency of the system. By decoupling HDFS write operations and Elastic Search write operations from data processing, the processing speed of the system is greatly improved, allowing the system to complete the extraction of feature indicators within milliseconds and cooperate with other systems to output risk control results within seconds.

[0095] Reference Figure 3 FIG. 1 is a schematic diagram of a preferred embodiment of an electronic device 1 of the present invention.

[0096] The electronic device 1 includes but is not limited to: a memory 11, a processor 12, a display 13 and a network interface 14. The electronic device 1 is connected to a network through the network interface 14 to obtain raw data. The network may be a wireless or wired network such as an intranet, the Internet, a global system of mobile communications (GSM), a wideband code division multiple access (WCDMA), a 4G network, a 5G network, Bluetooth, Wi-Fi, a call network, etc.

[0097] Among them, the memory 11 includes at least one type of readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 11 can be an internal storage unit of the electronic device 1, such as a hard disk or memory of the electronic device 1. In other embodiments, the memory 11 can also be an external storage device of the electronic device 1, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped with the electronic device 1. Of course, the memory 11 can also include both the internal storage unit of the electronic device 1 and its external storage device. In this embodiment, the memory 11 is generally used to store the operating system and various application software installed on the electronic device 1, such as the program code of the data extraction program 10, etc. In addition, the memory 11 can also be used to temporarily store various types of data that have been output or are to be output.

[0098] In some embodiments, the processor 12 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips. The processor 12 is generally used to control the overall operation of the electronic device 1, such as performing control and processing related to data interaction or communication. In this embodiment, the processor 12 is used to run the program code stored in the memory 11 or process data, such as running the program code of the data extraction program 10.

[0099] The display 13 may be referred to as a display screen or a display unit. In some embodiments, the display 13 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an organic light-emitting diode (OLED) touch device. The display 13 is used to display information processed in the electronic device 1 and to display a visual working interface, such as displaying the results of data statistics.

[0100] The network interface 14 may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), and the network interface 14 is generally used to establish a communication connection between the electronic device 1 and other electronic devices.

[0101] Figure 3Only the electronic device 1 having components 11 - 14 and the data extraction program 10 is shown, but it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0102] Optionally, the electronic device 1 may further include a user interface, which may include a display (Display), an input unit such as a keyboard (Keyboard), and the optional user interface may also include a standard wired interface and a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an organic light-emitting diode (Organic Light-Emitting Diode, OLED) touch device, etc. Among them, the display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device 1 and to display a visual user interface.

[0103] The electronic device 1 may further include a radio frequency (RF) circuit, a sensor, an audio circuit, etc., which will not be described in detail here.

[0104] In the above embodiment, the processor 12 may implement the following steps when executing the data extraction program 10 stored in the memory 11:

[0105] Obtaining original data and the data type of the original data from a preset message queue at every preset time period;

[0106] Converting the original data into standard format data based on the data type of the original data, performing encryption operation on sensitive information in the standard format data, and storing the encrypted standard format data in a first data table of a preset database;

[0107] Based on the data type of the standard format data, an extraction rule corresponding to the data type is loaded, and characteristic indicator data of the standard format data is extracted based on the extraction rule.

[0108] The storage device may be the memory 11 of the electronic device 1 , or may be another storage device that is communicatively connected to the electronic device 1 .

[0109] For a detailed description of the above steps, please refer to the above Figure 2 Functional module diagram of the embodiment of the data extraction device 100 and Figure 1 An illustration of a flow chart of an embodiment of a data extraction method.

[0110] In addition, an embodiment of the present invention further proposes a computer-readable storage medium, which may be non-volatile or volatile. The computer-readable storage medium may be any one or any combination of a hard disk, a multimedia card, an SD card, a flash memory card, an SMC, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a USB memory, etc. The computer-readable storage medium includes a data storage area and a program storage area, the data storage area stores data created according to the use of the blockchain node, and the program storage area stores a data extraction program 10, and the data extraction program 10 is executed by the processor to implement the following operations:

[0111] Obtaining original data and the data type of the original data from a preset message queue at every preset time period;

[0112] Converting the original data into standard format data based on the data type of the original data, performing encryption operation on sensitive information in the standard format data, and storing the encrypted standard format data in a first data table of a preset database;

[0113] Based on the data type of the standard format data, an extraction rule corresponding to the data type is loaded, and characteristic indicator data of the standard format data is extracted based on the extraction rule.

[0114] The specific implementation of the computer-readable storage medium of the present invention is substantially the same as the specific implementation of the above-mentioned data extraction method, and will not be described in detail here.

[0115] It should be noted that the serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. And the terms "including", "comprising" or any other variants thereof in this article are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "including a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0116] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, including a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, an electronic device, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0117] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A data extraction method, characterized in that: The method comprises: Obtaining original data and the data type of the original data from a preset message queue at every preset time period; Converting the original data into standard format data based on the data type of the original data, performing encryption operation on sensitive information in the standard format data, and storing the encrypted standard format data in a first data table of a preset database; Based on the data type of standard format data, extraction rules corresponding to the data type are loaded, and feature indicator data of the standard format data is extracted based on the extraction rules, wherein the standard format data of each data type corresponds to a feature indicator extraction processing class, and the feature indicator data of the standard format data is extracted based on the extraction rules, including: generating a data model according to the feature indicator processing class corresponding to the data type, loading the corresponding feature indicator extraction processing class according to the data model, extracting the feature indicator data from the standard format data using the feature indicator extraction processing class, and encrypting sensitive information in the feature indicator data and writing it into a preset feature indicator data table.

2. The data extraction method according to claim 1, characterized in that: After obtaining the original data and the data type of the original data, the method further includes: Determining whether it is necessary to perform a backup operation and a storage operation on the original data according to the preset configuration information, and when it is determined that it is necessary to perform a backup operation on the original data, sending the original data to a preset backup path of the message queue; When it is determined that a storage operation needs to be performed on the original data, an encryption operation is performed on the sensitive information in the original data, and the encrypted original data is stored in a second data table of a preset database.

3. The data extraction method according to claim 1, characterized in that: After converting the original data into standard format data based on the data type of the original data, the method further includes: It is determined whether there is data of a preset type in the standard format data, and when it is determined that there is data of a preset type in the standard format data, the data of the preset type is sent to the message queue.

4. The data extraction method according to claim 1, characterized in that: After extracting the characteristic indicator data of the standard format data based on the extraction rule, the method further includes: It is determined whether the characteristic indicator data needs to be sent to a preset storage path according to the message header of the original data corresponding to the characteristic indicator data, and if so, the characteristic indicator data is sent to the preset storage path.

5. The data extraction method according to claim 1, characterized in that: The performing of encryption operation on the sensitive information in the standard format data comprises: Sensitive information that needs to be encrypted is acquired according to a pre-configured encryption configuration table, and the encryption operation is performed on the sensitive information, wherein the encryption configuration table records the fields that need to be encrypted.

6. A data extraction device, characterized in that: The device comprises: Acquisition module: used to acquire original data and the data type of the original data from a preset message queue every preset time period; A conversion module: used for converting the original data into standard format data based on the data type of the original data, performing encryption operation on sensitive information in the standard format data, and storing the encrypted standard format data in a first data table of a preset database; Extraction module: used for loading the extraction rules corresponding to the data type of standard format data based on the data type of the standard format data, and extracting the characteristic indicator data of the standard format data based on the extraction rules, wherein the standard format data of each data type corresponds to a characteristic indicator extraction processing class, and extracting the characteristic indicator data of the standard format data based on the extraction rules includes: generating a data model according to the characteristic indicator processing class corresponding to the data type, loading the corresponding characteristic indicator extraction processing class according to the data model, extracting the characteristic indicator data from the standard format data using the characteristic indicator extraction processing class, and encrypting the sensitive information in the characteristic indicator data and writing it into a preset characteristic indicator data table.

7. An electronic device, characterized in that: The electronic device comprises: at least one processor; and, a memory communicatively connected to the at least one processor; wherein, The memory stores a program executable by the at least one processor, and the program is executed by the at least one processor to enable the at least one processor to perform the following steps: Obtaining original data and the data type of the original data from a preset message queue at every preset time period; Converting the original data into standard format data based on the data type of the original data, performing encryption operation on sensitive information in the standard format data, and storing the encrypted standard format data in a first data table of a preset database; Based on the data type of standard format data, extraction rules corresponding to the data type are loaded, and feature indicator data of the standard format data is extracted based on the extraction rules, wherein the standard format data of each data type corresponds to a feature indicator extraction processing class, and the feature indicator data of the standard format data is extracted based on the extraction rules, including: generating a data model according to the feature indicator processing class corresponding to the data type, loading the corresponding feature indicator extraction processing class according to the data model, extracting the feature indicator data from the standard format data using the feature indicator extraction processing class, and encrypting sensitive information in the feature indicator data and writing it into a preset feature indicator data table.

8. The electronic device according to claim 7, characterized in that: After obtaining the original data and the data type of the original data, the following steps are also included: Determining whether it is necessary to perform a backup operation and a storage operation on the original data according to the preset configuration information, and when it is determined that it is necessary to perform a backup operation on the original data, sending the original data to a preset backup path of the message queue; When it is determined that a storage operation needs to be performed on the original data, an encryption operation is performed on the sensitive information in the original data, and the encrypted original data is stored in a second data table of a preset database.

9. The electronic device according to claim 7, characterized in that: After converting the original data into standard format data based on the data type of the original data, the following steps are also included: It is determined whether there is data of a preset type in the standard format data, and when it is determined that there is data of a preset type in the standard format data, the data of the preset type is sent to the message queue.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a data extraction program, and when the data extraction program is executed by a processor, the steps of the data extraction method according to any one of claims 1 to 5 are implemented.

Citation Information

Patent Citations

  • Network transmission data storage system based on ElasticSearch and Hbase technologies

    CN109614401A

  • Desensitization rule configuration method, data desensitization method and system, and computer equipment

    CN110348239A

  • Distributed batch data cleaning method and system

    CN111522805A

  • Multi-source heterogeneous data aggregation method and device, storage medium and terminal

    CN111782690A

  • Construction method of data model and device and computer readable storage medium

    CN111813770A