Data processing method, system and equipment and storage medium

Abstract

The invention provides a data processing method, system and device and a storage medium, and relates to the technical field of computer application. The data processing method comprises the steps that a strategy information table is obtained, a list template table and a list template field information table corresponding to the list template table are obtained according to list template numbers in the strategy information table, the list template field information table comprises to-be-stored fields, the list template table is generated according to a basic template table comprising list classification information, and the list template table and the to-be-stored fields are stored in the basic template table. Generating a to-be-stored table according to the list template field information table; filling a to-be-stored field in the customer data into a to-be-stored table; analyzing and reading the filled single-row data of the to-be-stored table, and generating a list object according to the strategy information table and the single-row data; and according to the list classification information, determining list sub-tables to be stored and storing the list sub-tables. By adopting the data processing method provided by the invention, the effect of performing data storage on the specified list table can be realized, and meanwhile, development and online of storage logic can be avoided, so that the data storage efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer application technology, and more specifically, to a data processing method, system, device and storage medium. Background Art

[0002] With the development of business, more and more customer information data, such as customer information data from banks, online business platforms, etc., need to be queried as a verification basis. However, the current processing of customer information data such as warehousing has the following problems:

[0003] 1. Due to different data formats and different hit conditions, currently, for each business / each type of data, a corresponding data table must be designed, and the corresponding warehousing logic and query interface must be developed, which makes the demand docking, development, testing, and online cycle longer, and there is a problem that the data warehousing logic cannot be reused.

[0004] 2. The existing import progress display is only an approximate progress. For example, it is 30% for a long time and then becomes 50% directly after the next step, or it quickly rises from 1% to 99% and then remains in loading until the end. It cannot reflect the actual data processing progress.

[0005] 3. In the prior art, data is stored in the same table, or is routed to a specific sub-table after being processed by a hash algorithm (hash algorithm or hash algorithm) according to a certain field, resulting in excessive dependence on the field in the future. When adding other fields, the routing becomes more complicated, making subsequent expansion and management inconvenient.

[0006] 4. In the prior art, once the data is imported, it is permanently valid. If the data needs to be removed later, it can only be done by manual deletion. However, in reality, a lot of data is time-sensitive. For example, a customer is a premium customer this month, but no longer a premium customer next month.

[0007] 5. Since the data provider provides the full amount of data every time, the data volume will expand too quickly, and the data can only be stored incrementally in the existing technology.

[0008] 6. In the prior art, once the hit conditions are set for the data, they cannot be modified.

[0009] 7. In the existing technology, data can only be stored in plain text, so there is a risk of leaking sensitive data.

[0010] 8. In the existing technology, all data are stored in the warehouse, and non-compliant data cannot be filtered out, which may result in a large amount of dirty data being stored in the warehouse.

[0011] 9. In the prior art, the processing of failure data is log printing, and the failure data cannot be viewed intuitively and completely. Summary of the invention

[0012] In order to solve the problems existing in the above-mentioned prior art or at least part of the problems, the embodiments of the present invention provide a data processing method, system, storage medium, device and computer program product, which obtains a policy information table and a corresponding list template table, and then determines the list classification and the list table sub-table that the list object needs to be stored in the library according to the basic template table in the list template table. It can achieve the effect of data entry into the library by specifying the list table, so that data from the same source are stored in the same table. At the same time, it can avoid the development and online launch of the warehousing logic, thereby improving the efficiency of data entry.

[0013] According to the first aspect of the present invention, an embodiment of the present invention provides a data processing method, which includes: obtaining a policy information table, the policy information table including: a list template number, a policy code, and a policy name; obtaining a list template table and a list template field information table corresponding to the list template table according to the list template number, the list template field information table including fields to be stored, wherein the list template table is generated according to a basic template table, the basic template table includes list classification information, the list template table includes: a basic template table number, a list template number, and a list template name; generating a table to be stored according to the list template field information table; obtaining customer data and filling the fields to be stored in the customer data into the table to be stored; parsing and reading a single row of data of the filled table to be stored, and generating a list object according to the policy information table and the single row of data; determining a sub-table of the list table to be stored according to the list classification information, and storing the list object into the sub-table of the list table.

[0014] According to the above-mentioned embodiment of the present invention, by obtaining the policy information table and the corresponding list template table, and then determining the list classification and the list table sub-table that the list object needs to be stored in according to the basic template table in the list template table, the effect of data storage in the specified list table can be achieved, so that data from the same source are stored in the same table. At the same time, for different customer information data, users can quickly and conveniently realize data storage by configuring corresponding storage templates and strategies, avoiding the development and launch of storage logic, thereby improving data storage efficiency.

[0015] In some embodiments of the present invention, the policy information table also includes at least one of the following: list level, list source, validity period, whether to delete old list field, hit condition; the list template field information table includes verification rules.

[0016] According to the above-mentioned implementation mode of the present invention, the function of specifying the validity period of data and dynamically configuring the hit conditions can be realized through the setting of the policy information table, and the effect of data increment and full storage can be achieved by configuring whether to delete the old list. In addition, by configuring the verification rules for each field, non-compliant data can be filtered out during data import to avoid dirty data in the database.

[0017] In some embodiments of the present invention, the data processing method further includes: when the list object does not comply with the verification rule, writing the list object into an invalid data file.

[0018] According to the above-mentioned embodiment of the present invention, by writing the list objects that do not meet the verification rules into the invalid data file, the failed data can be exported, which facilitates the intuitive and complete viewing of the failed data.

[0019] In some embodiments of the present invention, the data processing method also includes: obtaining a query request, the query request including the policy code; querying a list table according to the query request to obtain hit data, the list table including the list table sub-table; when the number of hit data is greater than one, determining a query result data according to the list level and data creation time of the hit data.

[0020] According to the second aspect of the present invention, an embodiment of the present invention provides a data processing system, which includes: a policy information table acquisition module, used to obtain a policy information table, the policy information table includes: a list template number, a policy code, and a policy name; a list template table acquisition module, used to obtain a list template table and a list template field information table corresponding to the list template table according to the list template number, the list template field information table includes fields to be stored, wherein the list template table is generated according to a basic template table, the basic template table includes list classification information, the list template table includes: a basic template table number, a list template number, and a list template name; a data storage module, used to perform the following operations: generate a table to be stored according to the list template field information table; obtain customer data and fill the fields to be stored in the customer data into the table to be stored; parse and read the single-row data of the filled table to be stored, and generate a list object according to the policy information table and the single-row data; determine the list table sub-table to be stored according to the list classification information, and store the list object into the list table sub-table.

[0021] According to the above-mentioned embodiment of the present invention, by obtaining the policy information table and the corresponding list template table, and then determining the list classification and the list table sub-table that the list object needs to be stored in according to the basic template table in the list template table, the effect of data storage in the specified list table can be achieved, so that data from the same source are stored in the same table. At the same time, for different customer information data, users can quickly and conveniently realize data storage by configuring corresponding storage templates and strategies, avoiding the development and launch of storage logic, thereby improving data storage efficiency.

[0022] In some embodiments of the present invention, the policy information table also includes at least one of the following: list level, list source, validity period, whether to delete old list field, hit condition; the list template field information table includes verification rules.

[0023] According to the above-mentioned implementation mode of the present invention, the function of specifying the validity period of data and dynamically configuring the hit conditions can be realized through the setting of the policy information table, and the effect of data increment and full storage can be achieved by configuring whether to delete the old list. In addition, by configuring the verification rules for each field, non-compliant data can be filtered out during data import to avoid dirty data in the database.

[0024] In some embodiments of the present invention, the data storage module is further used to write the list object into an invalid data file when the list object does not meet the verification rule.

[0025] According to the above-mentioned embodiment of the present invention, by writing the list objects that do not meet the verification rules into the invalid data file, the failed data can be exported, which facilitates the intuitive and complete viewing of the failed data.

[0026] In some embodiments of the present invention, the data processing system also includes a data query module for performing the following operations: obtaining a query request, the query request including the policy code; querying a list table according to the query request to obtain hit data, the list table including the list table sub-table; when the number of hit data is greater than one, determining a query result data according to the list level and data creation time of the hit data.

[0027] According to the third aspect of the present invention, an embodiment of the present invention provides a computer-readable storage medium having computer-readable instructions stored thereon. When the computer-readable instructions are executed by a processor, the computer performs the following operations: the operations include the steps included in the data processing method described in any one of the above embodiments.

[0028] According to the fourth aspect of the present invention, an embodiment of the present invention provides a computer device including a memory and a processor, wherein the memory is used to store one or more computer-readable instructions, wherein the one or more computer-readable instructions, when executed by the processor, can implement the data processing method described in any one of the above embodiments.

[0029] According to a fifth aspect of the present invention, an embodiment of the present invention provides a computer program product including a computer program, and when the computer program is executed by a processor, the data processing method described in any one of the above embodiments is implemented.

[0030] As can be seen from the above, the data processing method, system, storage medium, device and computer program product provided by the embodiments of the present invention can achieve the effect of data entry by specifying a list table by obtaining a policy information table and a corresponding list template table, and then determining the list classification and the list table sub-table that the list object needs to be stored in according to the basic template table in the list template table, so that data from the same source are stored in the same table. At the same time, for different customer information data, users can quickly and conveniently implement data entry by configuring corresponding entry templates and strategies, avoiding the development and launch of entry logic, thereby improving data entry efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a flow chart of a data processing method according to Embodiment 1 of the present invention;

[0032] Figure 2 is a flow chart of a data processing method according to Embodiment 2 of the present invention;

[0033] Figure 3 Schematic diagram of the architecture of a data processing system according to Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0034] The various aspects of the present invention are described in detail below in conjunction with the accompanying drawings and specific embodiments. Among them, well-known modules, units and their connections, links, communications or operations are not shown or described in detail. In addition, the described features, architectures or functions can be combined in any way in one or more embodiments. It should be understood by those skilled in the art that the various embodiments described below are only used for illustration and are not intended to limit the scope of protection of the present invention. It can also be easily understood that the modules or units or processing methods in each embodiment described herein and shown in the accompanying drawings can be combined and designed in various different configurations.

[0035] The following is a brief description of the terms used in this article.

[0036] Template: The template can map the fields contained in the customer information data source and adapt the corresponding Excel table. For example, the Jisudai pre-credit data source has the corresponding template fields of ID number, mobile phone number, pre-credit customer classification, and pre-credit limit.

[0037] Strategy: Set the attributes of a batch of customer information data, such as list level, effective time, etc., and configure the hit conditions. For example, the list level of Jisudai pre-credit data is level one, the effective time is 365 days, and the hit condition is "ID number and mobile phone number", where "and" is the hit condition relationship operator.

[0038] Whitelist: represents a type of customer information data, i.e., the subject to be managed by the present invention. The high-quality lending list determined by the existing data within the financial institution, such as the pre-credit data of Jisudai, is a whitelist data, which will be stored in the whitelist table. The fields of the blacklist and whitelist tables are consistent, and the use of sub-tables can not only achieve the classification and isolation of business data, but also improve the query performance.

[0039] Blacklist: represents a type of customer information data, namely, bad customers with extremely bad nature, whose repayment ability, repayment willingness, borrowing purpose, etc. do not meet the standards of normal customers.

[0040] Basic fields: basically universal customer information, there are corresponding fields in the list table. For example, different list sources basically have basic fields such as ID number (card_no), mobile phone number (mobile), name (customer_name), etc.

[0041] Private fields: fields that do not exist in the list table. Since different customer information data has fields with their own special meanings, if all are added to the list table, a large wide table that is inconvenient to expand will be formed, and the performance of the table will be low. Therefore, for this type of less common data, it is uniformly put in the other_json field of the list table in json format. For example, fields such as pre-credit customer classification (ysxCustomerType) and pre-credit limit (ysxLimit) are only used in Jisudai pre-credit data.

[0042] Excel table: A spreadsheet file created using Excel software.

[0043] MD5: Message Digest Algorithm 5, a file encryption and authentication method.

[0044] Regular Expression: Regular Expression is a tool used to match, find, and replace specific patterns in text.

[0045] poiExcel: Apache POI library for operations on excel files.

[0046] XSSFWorkbook: One of the core classes in the POI library for processing Excel 2007 and above (i.e. xlsx format) files. XSSFWorkbook represents an Excel workbook, which is equivalent to a complete Excel file.

[0047] Sheet: XSSFSheet or HSSFSheet class, corresponding to the worksheet in XSSFWorkbook and HSSFWorkbook respectively. XSSFSheet represents a worksheet in an Excel workbook.

[0048] easyExcel: A Java library for reading and writing excel files conveniently and efficiently.

[0049] AnalysisEventListener: An abstract class in easyExcel used to listen to the excel file reading process.

[0050] EasyExcelFactory.write: Method in easyExcel library used to write data to excel file.

[0051] Writer: refers to the related objects used to perform Excel file writing operations.

[0052] EasyExcelFactory.writerSheet: A functional interface used in easyExcel to create, specify, and configure related properties for a worksheet (Sheet) when writing Excel files.

[0053] [Example 1]

[0054] Figure 1 It is a flowchart of a data processing method according to Embodiment 1 of the present invention.

[0055] like Figure 1 As shown, in Embodiment 1 of the present invention, the data processing method may include: step S11, step S12, step S13, step S14, step S15 and step S16, and the above steps are described in detail below.

[0056] In step S11, a policy information table is obtained, wherein the policy information table includes: list template number id, policy code, and policy name.

[0057] In some implementations, the policy information table further includes, but is not limited to, at least one of the following: list level, list source, effective duration, whether to delete old list fields, hit conditions, etc. Thus, by setting the policy information table, the functions of specifying data effective duration and dynamically configuring hit conditions can be realized, and the effects of data increment and full storage can be realized by configuring whether to delete old lists.

[0058] In step S12, a list template table and a list template field information table corresponding to the list template table are obtained according to the list template number, wherein the list template field information table includes fields to be stored, wherein the list template table is generated according to the basic template table, the basic template table includes list classification information, and the list template table includes: basic template table number, list template number, and list template name.

[0059] In this embodiment, the basic template table: the list template table is 1:N, and the list classification information is the code value of the list classification; the basic template table includes list classification, basic template table name, basic template table description and other information; the basic template field information table includes: basic parameter template id, parameter serial number, basic field Chinese name, basic field English name, parameter status (0-not enabled; 1-enabled) and other information; the list template table includes: basic template table id (number), list template id (number), list template name, list template description and other information; the list template field information table includes: list template number, list template field serial number, list template field Chinese name, list template field English name, parameter type (basic field, private field), list template field verification rule, whether encrypted and other information.

[0060] In some implementations, the list template field information table includes validation rules. By configuring validation rules for each field, non-compliant data can be filtered out during data import to avoid dirty data in the database.

[0061] In step S13, a waiting-to-enter table is generated according to the list template field information table. In this embodiment, the waiting-to-enter table is an Excel table.

[0062] In step S14, customer data is obtained and the to-be-stocked fields in the customer data are filled into the to-be-stocked table.

[0063] In step S15, the single row of data in the filled to-be-stored table is parsed and read, and a list object is generated according to the policy information table and the single row of data.

[0064] In step S16, the list sub-table to be stored is determined according to the list classification information, and the list object is stored in the list sub-table.

[0065] In a further embodiment, the data processing method further comprises: when the list object does not meet the verification rule, writing the list object into an invalid data file. By writing the list object that does not meet the verification rule into the invalid data file, the failed data can be exported, which is convenient for intuitively and completely viewing the failed data.

[0066] By adopting the above data processing method of Example 1 of the present invention, by obtaining the policy information table and the corresponding list template table, and then determining the list classification and the list table sub-table that the list object needs to be stored in the library according to the basic template table in the list template table, it is possible to achieve the effect of data storage in the designated list table, so that data from the same source are stored in the same table. At the same time, for different customer information data, users can quickly and conveniently realize data storage by configuring corresponding storage templates and strategies, avoiding the development and launch of storage logic, thereby improving data storage efficiency.

[0067] In a further embodiment, the data processing method further includes: obtaining a query request, the query request includes the policy code; querying the list table according to the query request to obtain hit data, the list table includes the list table sub-table; when the number of hit data is greater than one, determining a query result data according to the list level and data creation time of the hit data. Among them, by providing a query interface, other systems can call the list query function; by assembling the query conditions through the relationship symbol, the data query logic is public and reusable, avoiding the development of a corresponding query interface for each business data.

[0068] In a further embodiment, the data processing method further comprises: when the list object is stored in the list table sub-table, a data processing progress value is calculated and displayed. The data processing progress value is calculated by retaining two decimal places as the ratio of the total amount processed to the total amount to be processed. Thus, the data processing progress can be accurately reflected, allowing the user to understand the progress more clearly.

[0069] [Example 2]

[0070] Figure 2 It is a flowchart of a data processing method according to Embodiment 2 of the present invention.

[0071] like Figure 2 As shown, in the second embodiment of the present invention, the data processing method may include: step S21 to obtain the storage configuration; step S22 to obtain the storage configuration; and step S23 to query the list. The above steps are described in detail below.

[0072] In step S21, the storage configuration is obtained. The template is specified and the policy is created through the storage configuration. The step S21 specifically includes the following steps S211 to S214:

[0073] Step S211, basic template configuration: pre-configure basic fields for subsequent list template selection and specify the name of the list table.

[0074] Among them, the list classification is specified when configuring the basic template. In this embodiment, the list classification corresponding to the basic template is determined according to the list classification enumeration. Optionally, the list classification includes but is not limited to blacklist, whitelist, and graylist; the list classification enumeration defines code values, code value names, and list table names. For example, blacklist, whitelist, and graylist are three different code value names; the list classification enumeration includes: code value: W001, code value name: whitelist, list table name: roster_white_info. Therefore, the specific list table sub-table that needs to be stored in the list object generated based on the list template that specifies the basic template can be determined through the correspondence between the list classification and the sub-table name in the list classification enumeration.

[0075] In this embodiment, the name of the basic template cannot be repeated. Before entering the database, it is necessary to query whether there is data with the same name. If there is data with the same name, it will be prompted that the name is repeated. Furthermore, a basic field is added to the detailed information of the basic template according to the name of the basic template. Each basic template can be configured with multiple basic template field information to predefine the basic field. The basic template field information includes: the Chinese name of the basic field and the English name of the basic field. In addition, the Chinese name of the basic field is the same as the field annotation in the list table. For example, if the annotation of the card_no field in the list table is "ID card number", the Chinese name of the basic field is also "ID card number"; the English name of the basic field conforms to the camel case naming rule and corresponds to the field in the list table. For example, the card_no field in the list table has lowercase letters and underscores, and the English name of the basic field is named cardNo in camel case. Among them, the list table is embedded data, and the list table contains embedded fields.

[0076] Furthermore, in order to make the basic template fields correspond to the fields in the list table, the basic template field information in this embodiment does not support custom added fields. When creating a new basic template, the embedded data (i.e., the English fields and Chinese comments in the list table) are inserted at one time and are not enabled by default. Then, relevant fields are enabled as needed. Specifically, whether the relevant fields are enabled is controlled by param_state (i.e., parameter status, 0-not enabled, 1-enabled) in the basic template field information table (basic_template_info). When the param_state value is 1, that is, when enabled, the field is valid and is then used by subsequent template management.

[0077] At the same time, the present invention gives an example of creating a whitelist basic template: creating a whitelist basic template, the list classification corresponding to the whitelist basic template is W001, and the enabled basic template field information table data (format: basic field English name-basic field Chinese name) includes: cardNo-ID number, mobile-mobile phone number.

[0078] Step S212, template management: specify / configure specific fields of customer information data, define whether to encrypt, define verification rules, and specify template field numbers.

[0079] In this embodiment, different list templates are created according to different customer information data sources. Specifically, by selecting / specifying the basic template configured in step S211, and selecting the data in the basic template field information table data corresponding to the basic template to obtain the basic field data in the detailed information of the single template (list template field information table), each list template can be configured with multiple list template field information. Among them, the name of the list template cannot be repeated; one list template can only correspond to one basic template, and one basic template can correspond to multiple list templates; the list template number is unique. Optionally, the list template number is composed of the letter T + 5 digits. When generating the list template, the largest list template number in the database is queried (if there is no list template number in the database, the default is 0), and then the digital part is intercepted and added by one to generate the list template number of the currently created list template.

[0080] At the same time, the present invention provides an example of creating a list template: the Jisudai pre-credit template is the first configured list template, then its list template number is T00001, and the corresponding basic template is the whitelist basic template.

[0081] In a further implementation method, the list template details, i.e., the list template field information (template_info), are obtained by adding specific fields of the list template. The specific fields of the list template include but are not limited to: parameter type, Chinese name of the template field, English name of the template field, template field serial number, whether encrypted, verification rules, etc.

[0082] When adding a specific field of the list template, you need to specify whether the parameter type is a private field or a basic field. If the parameter type of the field is a basic field, the Chinese name and English name of the template field corresponding to the field are both from the Chinese name and English name of the basic field in the basic template field information table corresponding to the basic template specified by the list template; if the parameter type of the field is a private field, the Chinese name and English name of the template field corresponding to the field are determined by customization.

[0083] When adding specific fields to the list template, you can manually specify the template field number. In addition, the fields on the list template are listed according to the template field number, and by default they are automatically incremented in the order they are added.

[0084] When adding a specific field in the list template, you can specify whether the field is encrypted. If the field is encrypted, the data is ciphertext (such as MD5 value) when imported into Excel; when querying, the input parameter is plaintext, MD5 calculation is performed on the input parameter, and then an equivalent query is performed in the database, and the result is returned as ciphertext. In this way, sensitive information of customers can be protected.

[0085] When adding specific fields of the list template, you can specify the verification rules (for example, by using regular expressions to represent the verification rules). If the field needs to be verified, the imported Excel data will be verified. If the verification fails, a log alarm will be sent and the row of data will be written to the invalid data file. By writing the list objects that do not meet the verification rules into the invalid data file, the failed data can be exported, making it easier to view the failed data intuitively and completely.

[0086] At the same time, the present invention provides an example of adding a specific field of a list template to obtain the details of the list template: the name of the list template is "Jisudai Pre-credit Template", and the format of its list template field information is "Template field Chinese name-Template field English name-Field type-Encryption-Verification rule", then its list template details include the following information: ID card number-cardNo-basic field-unencrypted-verification rule; mobile phone number-mobile-basic field-unencrypted-no verification rule; pre-credit customer classification-ysxCustomerType-private field-unencrypted-no verification rule; pre-credit limit-ysxLimit-private field-unencrypted-no verification rule. According to this example, it is necessary to perform regular verification on the ID card number column when importing data.

[0087] Step S213, policy configuration: bind / select template, specify list level, list source, validity period / validity duration, whether to delete old list, and configure hit conditions. The template is determined by specifying the list template table created in step S212, and the policy configuration includes obtaining the policy name. Before the template is stored in the database, the policy name is deduplicated by judging whether there is data with the same name in the database. That is, when there is data with the same name in the database, it is prompted that the current policy name is illegal.

[0088] In this embodiment, specifying the list level includes: determining the list level through configuration information provided in advance by the business, and the configuration information includes: a code value and a name corresponding to the list level. For example, if the code value corresponding to the list level is GRADE_01, then its list level name is level 1. By obtaining the list level, when a subsequent data query hits multiple list data at the same time, the data with a high list level can be preferentially used as the query result data.

[0089] Among them, the effective duration is the validity period of the data, and its unit is day. By setting the effective duration, the expiration time can be calculated based on the date of entry plus the effective duration. The source of the list is where the data that the business can enter comes from, for example, the source of the list is the collection list compiled by the collection department. The hit condition is to query the hit condition corresponding to the data. All the query fields used are defined by the hit condition enumeration. The query field is the same as the field in the list table, for example, cardNo-ID number, mobile-mobile phone number, customerName-name, bankCardNo-bank card number, customerId-customer number, etc. Furthermore, the optional items of the hit condition are the intersection of all fields in the hit condition enumeration and all fields in the list template field information table corresponding to the policy. For example, the aforementioned Jisudai pre-credit template is used in the Jisudai pre-credit strategy, and the optional items of the hit condition are ID number and mobile phone number.

[0090] Among them, when the hit condition has two or more fields, the hit condition is composed of the hit condition using the hit condition relational symbols "and" and "or". When the "and" relational symbol is used, a hit is considered a hit if all fields are matched. If it is an "or" relational symbol, a hit is considered a hit if one rule is matched. For example, if the hit condition is "ID number and mobile phone number", then the ID number and mobile phone number must be matched at the same time for the query to be a hit; if the hit condition is "ID number or mobile phone number", then a hit is considered a hit if one of the ID number or mobile phone number is matched during the query.

[0091] Among them, when the field of whether to delete the original / old list is "Y-Yes", the data imported by the strategy in the previous period will be deleted each time an Excel is imported, which is equivalent to importing the full data each time; when the field of whether to delete the original list is "N-No", the current data will be inserted each time without deleting the previous data, which is equivalent to incremental data import.

[0092] At the same time, the present invention gives an example of policy configuration: the list level corresponding to the Jisudai pre-credit policy is GRADE_01; the effective time is 365; the hit condition fields are ID card number and mobile phone number; the hit condition relationship symbol is "and"; whether to delete the original list is N-no.

[0093] Step S214, downloading an Excel template according to the template: generating an Excel template table corresponding to the table header and the list template field information table, wherein the list template field information table defines the corresponding template field English name, template field Chinese name and template field serial number.

[0094] In this embodiment, XSSFWorkbook in poiExcel is used to create a sheet, and the English name of the template field is written into the cells of the corresponding order in the first row of Excel (the smaller the field number, the earlier it is written), and set to hidden attributes. Then the Chinese name of the template field is written into the second row, and it is not set to be hidden, that is, the Chinese name of the template field in the second row is the header seen by the business personnel. Finally, the processed file stream is downloaded to the local computer, and an Excel table corresponding to the list template field information table can be obtained.

[0095] At the same time, the present invention provides an example of obtaining an Excel template: the Jisudai pre-credit template corresponds to the detailed list template field information with four fields, which are arranged in order of the template field number: cardNo-ID number, mobile-mobile phone number, ysxCustomerType-pre-credit customer classification, ysxLimit-pre-credit limit. The Excel template table generated according to the detailed list template field information has two rows and four columns, the first row is cardNo, mobile, ysxCustomerType, ysxLimit, and the row is hidden. The second row is the ID number, mobile phone number, pre-credit customer classification, pre-credit limit, and the row is displayed as a header row in Excel.

[0096] In step S22, the list is imported. The data is loaded into the Excel template, and a strategy is selected to import the data into the database. The step S22 specifically includes the following steps S221 to S223:

[0097] Step S221, obtain the policy and the excel file filled with data. The policy is the policy configured in the aforementioned step S213, and the corresponding list template field information can be obtained according to the policy information; the excel file filled with data is an excel file obtained by the business personnel filling the customer data obtained by the third-party company or other related mechanisms into the excel template in the aforementioned step S214.

[0098] In this embodiment, a unique batch number is generated for the Excel file filled with data, each data import into Excel corresponds to a batch, and all data in the current Excel file has the batch number.

[0099] At the same time, the present invention provides an example of an acquisition strategy and an excel file filled with data: the template corresponding to the Jisudai pre-credit data is the Jisudai pre-credit template, which has four fields, including two basic fields (ID number and mobile phone number) and two private fields (pre-credit customer classification and pre-credit limit), thereby generating a four-column excel; the data in the excel comes from high-quality customers of other products and customer information obtained from offline marketing. After collecting the customer information, the business personnel fill in the customer's ID number, mobile phone number, pre-credit customer classification, pre-credit limit and other four fields into the aforementioned excel. For example, a certain customer information is stored in excel and represented as: 13052119930912092, 15798231822, 2, 2345.12.

[0100] Step S222, use easyExcel to parse the Excel table obtained in step S221, read a single row of data, generate the list data according to the level and effective time defined by the policy, and write the data that does not meet the verification rules into the invalid data file. The background uses the invokeHead (processing header), invoke (processing each row of data), and doAfterAllAnalysed (processing end) methods in AnalysisEventListener of easyExcel to parse the Excel file.

[0101] In this embodiment, invokeHead checks whether the excel file matches the template. Specifically, the first row of the generated excel template is hidden and contains English field names. InvokeHead compares the first row of the excel template with the English names of the template fields in the corresponding list template field information table. When there is a discrepancy between the first row of the excel template and the English names of the template fields in the corresponding list template field information table, that is, the selected strategy does not match the incoming excel, an exception is thrown.

[0102] In a further implementation, invokeHead is also used to generate an invalid data file sheet page. Specifically, the Chinese name of the template field corresponding to the template is obtained as a list (column), and the failure reason is added after the list, and then a writer is created through EasyExcelFactory.write, and an error data file sheet page is created through EasyExcelFactory.writerSheet. When the data does not meet the verification rules in the template, it will be written to the error data file sheet page. For example, for the invalid data file of the Jisudai pre-credit template, its header has a total of five fields, namely: ID number, mobile phone number, pre-credit customer classification, pre-credit limit, and failure reason. Among them, the invalid data file has no hidden rows.

[0103] In a further embodiment, the accurate data processing progress is obtained by calling an asynchronous method for calculating the progress. Specifically, first, the total number of Excel rows, i.e., the number to be imported (toInsertCount), is obtained in the invokeHead method of easyExcel. Then, when the field "Whether to delete the original list" in the policy is "Y-Yes", the following step a is executed, and when the field "Whether to delete the original list" in the policy is "N-No", the following step b is executed.

[0104] Step a: The field "Whether to delete the original list" in the policy is "Y-Yes". First, query and obtain the total number of data that are not in the batch number under the policy, that is, the total number to be deleted (totalToDelCount). The sum of the number to be imported and the total number to be deleted is the total amount to be processed; then query and obtain the number of imported data (insertedCount) under the batch every 5 seconds (Thread.sleep, configurable) in the while(true) loop, and the total number of non-batch data under the policy is the number to be deleted at the current time (currentToDelCount). The total number to be deleted minus the number to be deleted at the current time is the deleted number; the imported number plus the deleted number is the processed total. The ratio of the processed total to the total amount to be processed is rounded to two decimal places as the data processing progress value (processRate). Among them, the data processing progress value (processRate) is calculated by the following formula (1):

[0105]

[0106] Step b: The field "Whether to delete the original list" in this strategy is "N-No", and the number to be imported is the total amount to be processed; then in the while(true) loop, the number of imported items (insertedCount) under this batch is checked every 5 seconds (Thread.sleep, configurable), which is the total amount processed. The ratio of the total amount processed to the total amount to be processed is rounded to two decimal places as the data processing progress value (processRate). The data processing progress value (processRate) is calculated by the following formula (2):

[0107]

[0108] Finally, when the progress value is equal to 1.00, the loop is exited. The progress value of the current batch is placed in the redis library. The unique string key used to identify the data stored in redis is a fixed prefix plus the batch number, and the value is the progress value. In addition, the progress value is taken out from redis every 5 seconds for display through polling on the web or other places (for example, through the window.setInterval function on the web). Therefore, the data processing progress value with a percentage of two decimal places can accurately reflect the data processing progress, allowing users to understand the data processing progress more clearly.

[0109] In this embodiment, invoke processes each row of data in the excel file, wherein the second row storing the Chinese field is filtered out. By obtaining the list template field information corresponding to the template, invoke processes each cell data in each row of the excel file, including: when the template specifies that the header field corresponding to the cell has a verification rule, the cell data is verified. Among them, the verification includes: using the regular expression Pattern.matcher to match the cell data. If the match fails, the data file sheet page created in the aforementioned invokeHead of the row of data is written into the error data file, and the cause of the error is specified, that is, a certain field does not match according to a certain verification rule. For example, when the pre-credit data of Jisudai is imported, the ID number, mobile phone number, pre-credit customer classification, and pre-credit limit of a certain row are: 13052119930912092, 15798231822, 2, 2345.12, respectively. According to the verification rule defined for the ID number in the template, the ID number in the cell is judged to be 17 digits, that is, it is not compliant, and a row of data with five columns will be generated in the invalid data file: 13052119930912092, 12398231822, 2, 2345.12, and the ID number verification fails.

[0110] Among them, by obtaining the list template field information corresponding to the template, invoke processes the data of each cell in each row of the excel file and also includes: when the template field parameter type corresponding to the cell is a basic field, the cell data is placed in a json (for example, named rosterJson) stored in the form of a key-value pair, where the key is the English name of the template field corresponding to the field, and the value is the cell value parsed from excel; when the template field parameter type corresponding to the cell is a private field, it is stored in another json (for example, named otherJson), where the key is the English name of the template field corresponding to the field, and the value is the cell value parsed from excel; when each field is processed, otherJson is converted into a string and placed in rosterJson, and its corresponding key is "otherJson". Then, the filled rosterJson is converted into a list object through the toJavaObject method of json.

[0111] At the same time, the present invention provides an example of generating rosterJson: a row imported from the Jisu loan pre-credit template excel is (130521199309120921, 15798231822, 2, 2345.12), then the final format of generating rosterJson is:

[0112] {

[0113] "cardNo":"130521199309120921",

[0114] "mobile":"15798231822",

[0115] "otherJson":"{\"ysxCustomerType\":2,\"ysxLimit\":2345.12}"

[0116] }

[0117] In this embodiment, after reading the Excel data, the policy code, batch number and other fields are supplemented, and the expiration time is calculated based on the validity period in the policy and the current time. Then, the Excel cell data is inserted into the database, where the fields that need to be encrypted are ciphertext in Excel and are also ciphertext in the library. When querying, enter the plain text, calculate it through MD5 first, and then query it in the library. Among them, the basic template information is obtained according to the list template, and then the name of the list table corresponding to the basic template can be obtained based on the list classification enumeration, and then it is stored in the list table.

[0118] Step S223, performing subsequent deletion operations according to whether to delete the original list as defined by the policy.

[0119] In this embodiment, when the "delete original list" field of the policy is "Y-Yes", the data not in the batch under the policy is deleted. Optionally, when there is a lot of data, it is moved into the history table in batches through paging and then physically deleted.

[0120] In step S23, the list query is performed. Dynamic SQL is assembled according to the strategy to query the corresponding sub-table. In this embodiment, by providing a query interface, other systems can call the list query function. The step S23 specifically includes the following steps S231 to S233:

[0121] Step S231, input parameter verification. Obtain input parameters such as the policy code, and verify the input parameters. Among them, the policy number is a required input parameter. There can be multiple policy numbers, separated by commas, and the type is string. Optionally, the input parameters may also include, but are not limited to, cardNo, mobile, and other parameters that correspond to the fields in the hit condition enumeration used when creating the policy.

[0122] Furthermore, the hit condition fields and hit condition relationships are obtained according to the strategy. For the relationship symbol "and", the corresponding hit condition fields must exist, otherwise an input parameter missing error is directly returned; for the relationship symbol "or", at least one corresponding hit condition field must exist, otherwise an input parameter missing error is directly returned.

[0123] Step S232, assembling query conditions according to the strategy.

[0124] In this embodiment, the dynamic SQL (Structured Query Language) of the MyBatis framework is used to query the database, and the query conditions are assembled according to the relationship symbol. If the template field is specified as an encrypted field, it is necessary to perform MD5 calculation on this field before querying. The input parameter supports the input of multiple policy codes. In actual processing, multiple policy codes are separated for single policy query, and then the results are merged. As a result, the data query logic becomes public and reusable, and different query conditions can be assembled according to different strategies, avoiding the development of corresponding query interfaces for each business data.

[0125] At the same time, the present invention provides an example of assembling query conditions: if the hit condition is "ID number and mobile phone number", the converted dynamic SQL is select * from roster_white_info where stra_id =? and card_no =? and mobile =?; if the hit condition is "ID number or mobile phone number", and the ID number is defined as an encrypted field in the template, the converted dynamic SQL is select * from roster_white_info where stra_id =? and (card_no = MD5 (?) or mobile =?).

[0126] Step S233, obtaining hit data according to the query condition, and determining a query result data according to the list level and data creation time of the hit data, wherein the query result data is the data with the highest list level and the most recent creation time.

[0127] In this embodiment, after the hit data is found, the current time is compared with the expiration time to determine whether the hit data is within the validity period; if it is within the validity period, the hit data is legal data, otherwise it is not a hit. Further, when multiple hit data are found, the one with the highest list level and the most recent creation time is taken as the query result data.

[0128] The data processing method of the second embodiment of the present invention can achieve the following effects: 1. The warehousing logic does not need to be developed and put online. For different customer information data, corresponding warehousing templates and strategies can be configured, which is fast and convenient through configuration; 2. The query logic does not need to be developed and put online every time. A unified interface is defined. Each type of customer information data adapts its own strategy. The strategy contains the query conditions of the data. The data is queried according to the dynamic assembly conditions of the strategy; 3. The data processing progress can be accurately reflected, so that users can understand the progress of data processing more clearly; 4. The list table (i.e. the corresponding sub-table) can be specified. ), data from the same source are placed in the same table; 5. The level of the list can be specified for easy selection during use; 6. The validity period can be specified; 7. Whether to delete the old list can be specified, that is, incremental and full data can be stored in the warehouse; 8. The hit conditions can be configured dynamically; 9. Whether encryption can be configured for each field, for example, data such as identity cards can be desensitized; 10. The parameter serial number can be specified for each field for easy typesetting; 11. The verification rules can be configured for each field to filter out non-compliant data during import, so that dirty data will not appear in the database; 12. Failed data can be exported.

[0129] [Example 3]

[0130] Figure 3Schematic diagram of the architecture of a data processing system according to Embodiment 3 of the present invention.

[0131] like Figure 3 As shown, the data processing system includes: a policy information table acquisition module 310, a list template table acquisition module 320, a data storage module 330, a data query module 340, and a progress display module 350.

[0132] Among them, the policy information table acquisition module 310 is used to obtain the policy information table, and the policy information table includes: list template number, policy code, and policy name. In some embodiments, the policy information table also includes but is not limited to at least one of the following: list level, list source, effective time, whether to delete the old list field, hit conditions and other information. Therefore, through the setting of the policy information table, the function of specifying the effective time of data and dynamically configuring the hit conditions can be realized, and the effect of data increment and full storage can be achieved by configuring whether to delete the old list.

[0133] The list template table acquisition module 320 is used to acquire the list template table and the list template field information table corresponding to the list template table according to the list template number, the list template field information table includes fields to be stored, wherein the list template table is generated according to the basic template table, the basic template table includes list classification information, and the list template table includes: basic template table number, list template number, list template name.

[0134] The data entry module 330 is used to perform the following operations: generate a table to be entered into the warehouse based on the list template field information table; obtain customer data and fill the fields to be entered in the customer data into the table to be entered into the warehouse; parse and read the single-row data of the filled table to be entered into the warehouse, and generate a list object based on the policy information table and the single-row data; determine the sub-table of the list table to be entered into the warehouse based on the list classification information, and enter the list object into the sub-table of the list table.

[0135] The data query module 340 is used to perform the following operations: obtain a query request, the query request includes the policy code; query the list table according to the query request to obtain hit data, the list table includes the list table sub-table; when the number of hit data is greater than one, determine a query result data according to the list level and data creation time of the hit data.

[0136] The progress display module 350 is used to calculate and display the data processing progress value when storing the list object into the list table sub-table.

[0137] The data processing system of Embodiment 3 of the present invention can achieve the effect of data entry by specifying a list table by obtaining a policy information table and a corresponding list template table, and then determining the list classification and the list table sub-table that the list object needs to enter according to the basic template table in the list template table, so that data from the same source are stored in the same table. At the same time, for different customer information data, users can quickly and conveniently enter data into the library by configuring corresponding entry templates and policies, avoiding the development and launch of the entry logic, thereby improving the efficiency of data entry.

[0138] Through the description of the above implementation modes, those skilled in the art can clearly understand that the present invention can be implemented by combining software with a hardware platform. Based on such an understanding, all or part of the contribution of the technical solution of the present invention to the background technology can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in various implementation modes of the present invention or some parts of the implementation modes.

[0139] Correspondingly, an embodiment of the present invention further provides a computer-readable storage medium on which computer-readable instructions or programs are stored. When the computer-readable instructions or programs are executed by a processor, the computer performs the following operations: the operations include the steps included in the data processing method described in any of the above embodiments, which are not repeated here. The storage medium may include, for example, an optical disk, a hard disk, a floppy disk, a flash memory, a magnetic tape, etc.

[0140] In addition, an embodiment of the present invention further provides a computer device including a memory and a processor, wherein the memory is used to store one or more computer-readable instructions or programs, wherein the one or more computer-readable instructions or programs can implement the data processing method described in any of the above embodiments when executed by the processor. The computer device can be, for example, a server, a desktop computer, a notebook computer, a tablet computer, etc.

[0141] The present invention also provides a computer program product including a computer program, wherein the computer program includes a program code for executing the data processing method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the data processing method provided by the embodiment of the present disclosure.

[0142] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect through the Internet).

[0143] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention is described in detail with reference to the above embodiments, a person skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some of the technical features can be replaced by equivalents; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the various embodiments of the present invention. Therefore, the protection scope of the present invention shall be subject to the claims.

Claims

1. A data processing method, characterized in that: The data processing method comprises: Obtain a policy information table, the policy information table including: list template number, policy code, and policy name; According to the list template number, a list template table and a list template field information table corresponding to the list template table are obtained, wherein the list template field information table includes fields to be stored, wherein the list template table is generated according to a basic template table, the basic template table includes list classification information, and the list template table includes: a basic template table number, a list template number, and a list template name; Generate a table to be stored according to the list template field information table; Acquire customer data and fill the to-be-stored fields in the customer data into the to-be-stored table; Parse and read the populated single-row data of the to-be-stored table, and generate a list object according to the policy information table and the single-row data; The list sub-table to be stored is determined according to the list classification information, and the list object is stored in the list sub-table.

2. The data processing method according to claim 1, characterized in that: The policy information table also includes at least one of the following: list level, list source, validity period, whether to delete the old list field, and hit conditions; The list template field information table includes verification rules.

3. The data processing method according to claim 2, characterized in that: The data processing method further includes: when the list object does not conform to the verification rule, writing the list object into an invalid data file.

4. The data processing method according to claim 1, characterized in that: The data processing method further includes: Obtaining a query request, wherein the query request includes the policy code; querying a list table according to the query request to obtain hit data, wherein the list table includes the list table sub-tables; When the number of hit data is greater than one, a query result data is determined according to the list level and data creation time of the hit data.

5. A data processing system, characterized in that: The data processing system comprises: A policy information table acquisition module is used to acquire a policy information table, wherein the policy information table includes: a list template number, a policy code, and a policy name; A list template table acquisition module is used to acquire a list template table and a list template field information table corresponding to the list template table according to the list template number, wherein the list template field information table includes fields to be stored, wherein the list template table is generated according to a basic template table, the basic template table includes list classification information, and the list template table includes: a basic template table number, a list template number, and a list template name; The data entry module is used to perform the following operations: Generate a table to be stored according to the list template field information table; Acquire customer data and fill the to-be-stored fields in the customer data into the to-be-stored table; Parse and read the populated single-row data of the to-be-stored table, and generate a list object according to the policy information table and the single-row data; The list sub-table to be stored is determined according to the list classification information, and the list object is stored in the list sub-table.

6. The data processing system according to claim 5, characterized in that The data storage module is also used to write the list object into an invalid data file when the list object does not meet the verification rule.

7. The data processing system according to claim 5, characterized in that The data processing system also includes a data query module, which is used to perform the following operations: Obtaining a query request, wherein the query request includes the policy code; querying a list table according to the query request to obtain hit data, wherein the list table includes the list table sub-tables; When the number of hit data is greater than one, a query result data is determined according to the list level and data creation time of the hit data.

8. A computer-readable storage medium storing computer-readable instructions, characterized in that: The computer-readable instructions are executed by a processor to implement the data processing method according to any one of claims 1 to 4.

9. A computer device comprising a memory and a processor, The memory stores computer-readable instructions, characterized in that: The processor executes the computer-readable instructions to implement the data processing method according to any one of claims 1 to 4.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the data processing method according to any one of claims 1 to 4 is implemented.

Images