Data processing method based on dynamic configuration

Abstract

According to the dynamic configuration-based data processing method provided by the invention, the data configuration page is constructed based on each type of basic data, the collection information configuration page is set, and all terminal users are uniformly specified for each type of basic data through the user specifying module; uniformly setting corresponding verification rules for all fields of the basic data through a verification rule selection module, and setting a rule management window to dynamically display all verification rules configured by a verification rule configuration page in a rule configuration module; once the number of fields of certain basic data or the verification rules corresponding to the fields changes, modification is carried out according to requirements through the verification rule configuration page, the modified result can be dynamically displayed through the rule management window, the collection configuration page of each piece of basic data does not need to be modified, and the user experience is improved. And the working efficiency of system configuration is greatly improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a data processing method based on dynamic configuration. Background Art

[0002] With the development of information digitalization technology, the unified uploading of various business-related data is one of the foundations of many statistical work. For example, the two systems shown in application numbers CN202011474245.X and CN202011511579.X are data management systems built and maintained by our company. These two systems need to collect basic data reported by different government departments or different companies every day, and the system will perform subsequent analysis and statistics based on these basic data. In order to ensure the accuracy of the uploaded data, it is necessary to verify different types of data through preset data verification methods when filling in the data, and data that does not meet the verification rules is not allowed to be uploaded. Common verification methods include: regular expression verification, enumeration value verification, numerical range verification, name verification, date format verification, maximum value verification, ID card verification, etc. These verification methods are all implemented by very mature technologies in the prior art.

[0003] In actual applications, each type of basic data that needs to be uploaded may include multiple fields, and each field requires a different verification method; for example, the basic data that needs to be uploaded is: double public administrative penalties, and the corresponding fields that need to be verified include: administrative counterpart name, administrative counterpart code, legal representative name, administrative penalty decision document number, penalty name, penalty category, penalty reason, penalty decision date, penalty deadline, etc. Once the regulations related to "double public administrative penalties" change, such as the ID card number changes from 16 digits to 18 digits, the associated fields must be adjusted synchronously, and the relevant verification methods need to be adjusted; or if the latest relevant regulations require adding a few more fields to the "double public administrative penalties", then the system needs to add new fields and related verification rules. It is even possible that the "double public administrative penalties" will be deleted and changed to other types of basic data. Once these changes occur, the system needs to promptly and synchronously modify the data entry related modules.

[0004] The system structure of a general data management platform is as follows: Figure 1As shown in the figure, the terminal user accesses the data entry module page of the data platform based on the PC or mobile phone. When the user logs in, the page that opens is the data entry module page corresponding to the user's ID. Users with different permissions or different types of users will correspond to different data entry pages. In order to ensure that users can enter data accurately, before the terminal user logs in, the configuration administrator will configure the data entry page corresponding to each user based on the configuration module. The configuration content includes: the basic data that the user needs to enter and the fields included in the basic data, as well as the verification rules required for each field.

[0005] The traditional configuration method is based on terminal users. The configuration administrator opens the configuration page of each terminal user and configures the basic data that needs to be uploaded for each user. Once a certain basic data changes, the configuration administrator needs to open all the terminal user configuration pages related to this basic data in turn to configure them separately. However, because the number of terminal users who need to submit to the system is very large, there are dozens or hundreds of users in the existing system. If the original data management method is still used, because there are many users involved in the adjustment, and the users involved in each basic data adjustment may be different, each configuration administrator needs to spend a lot of time to handle the configuration change work, and it is easy to make mistakes. Summary of the invention

[0006] In order to solve the problem that when the number of system users is large and the data validation rules need to be modified, the existing configuration method is implemented with low efficiency, the present invention provides a data processing method based on dynamic configuration, which can accurately and efficiently respond to changes in configuration content.

[0007] The technical solution of the present invention is as follows: a data processing method based on dynamic configuration, characterized in that it comprises the following steps: S1: Set up a separate data configuration page for each type of basic data that needs to be uploaded; The data configuration page includes: a basic information configuration page and a collection information configuration page; The basic information configuration page includes: all fields included in the basic data; The collection information configuration page includes: a verification rule selection module and a user specified module; The verification rule selection module is used to configure all fields that need to be verified and the verification rules corresponding to each field; The user specifying module is used to specify all terminal users who need to upload the basic data; S2: Build the verification rule configuration page and the user information configuration page; In the verification rule configuration page, configure the corresponding verification rule for each field that needs to be verified; In the user information configuration page, configure corresponding basic information for each terminal user; S3: construct a rule management window, display all the verification rules configured in the verification rule configuration page in the rule management window, and open the rule management window when the verification rule of each field is selected in the verification rule selection module; The rule management window includes all the verification rules, and each of the verification rules corresponds to a selection box; The selection box in the rule management window is a selection box that supports multiple selections, and allows each field to select more than or equal to 1 verification rule at the same time; S4: When it is necessary to configure the basic data, according to specific needs, add or modify the corresponding verification rules through the verification rule configuration page, and complete the addition or modification of user information through the user information configuration page; S5: Confirm the type of configuration operation to be performed according to the requirements; If a new type of basic data needs to be added, step S6 is executed; If it is necessary to modify the fields and rules of the existing basic data, execute step S7; If the corresponding terminal user is to be modified for the existing basic data, step S8 is executed; S6: When adding basic data, perform the following steps: a1: Based on the basic information configuration page in the data configuration page, configure all the fields included in the newly added basic data; a2: Based on the verification rule selection module in the collection information configuration page, set all the fields that need to be verified, and select the verification rule corresponding to the field based on the rule management window to complete the operation; a3: Based on the user specified module in the collection information configuration page, select all terminal users who need to upload the newly added basic data; a4: execute step S9; S7: When making changes to existing basic data, perform the following steps; b1: Confirm the rules for adding new fields or modifying existing fields; If it is a new field, go to step b2; If it is a rule to modify an existing field, execute step b3; b2: Enter the collection information configuration page of basic data, add a new field based on the verification rule selection module, select the verification rule corresponding to the new field based on the rule management window to complete the operation; execute step S9; b3: Enter the collection information configuration page of the basic data, find the existing field that needs to be modified, open its corresponding rule management window, and reselect the verification rule corresponding to the existing field as required; execute step S9; S8: When modifying the corresponding terminal user for the existing basic data, perform the following steps: c1: Open the data configuration page of the basic data and modify the terminal user based on the user specified module; c2: execute step S9; S9: Check whether there are other configurations. If yes, execute steps S5 to S8 in a loop. If all configuration work is completed, a data upload template is generated for the end user to complete this configuration operation.

[0008] It is further characterized by: It also includes the following steps: d1: Create a directory table and a field table for all basic data; The directory table includes: directory primary key, directory name and directory code; One record in the directory table corresponds to one basic data; The field table stores the contents of all fields included in all basic data; the field table includes: field table primary key, field name column_name, field comments comments_name, field rule column_rule_id, directory primary key; The number of fields included in each basic data is greater than or equal to 1; d2: Create a user table for all end users to record all end user information; The user table: user table primary key, user full name, user code; d4: construct a directory user correspondence table, establish a relationship between the user table and the directory table, and record all basic data that each terminal user needs to submit; the directory user correspondence table includes: the primary key of the relationship table, the primary key of the user table, and the primary key of the directory; d5: For each basic data record in the directory table, read the corresponding field table content and create a physical table for each basic data; The physical table is used to store the data content actually uploaded by the terminal user, which includes: the physical table primary key, user code, user full name, field name, and field comment; A rule table is created for all verification rules, wherein the rule table includes: a rule table primary key, a rule name, and a rule content; The verification rules configured for each field in the verification rule selection module include: required verification, enumeration verification, regular verification, one or more combinations of enumeration and regular verification and plug-in verification; In the verification rule configuration page, the configuration pages of enumeration verification and regularization verification are placed in the same configuration page; based on the verification rule configuration page, the configuration of enumeration verification, regularization verification, and enumeration and regularization verification is implemented; The verification rule configuration page includes: a rule name input box, an enumeration regular expression input box, a separator input box and an enumeration value input box; The data upload template includes: user code, user full name, directory code, directory name, field name, and field comment; Each of the data upload templates corresponds to one type of basic data; in the data upload template, each row includes all the fields of the basic data that need to be checked; It also includes the data uploading process, which specifically includes the following steps: e1: The terminal user uploads data to the system based on the data upload template; e2: Start a thread to parse the files uploaded by the terminal user; e3: Get the data included in each row of the data upload template uploaded by the terminal user, recorded as: data to be confirmed; The data to be confirmed include: user code, user full name, directory code, directory name, field name, field comment and field value; e4: according to the fields included in the data to be confirmed, find the verification items that need to be performed for each field in the data to be confirmed, and put all the verification rules corresponding to the data to be confirmed into a set, which is recorded as: verification items to be executed; e5: Confirm the physical table corresponding to the terminal user according to the user code and user full name, recorded as: physical table to be inserted; e6: Execute all the verification items to be executed for each row of the data to be confirmed; e7: Confirm the execution result of the verification item to be executed; If all the verification items to be executed meet the verification requirements and the result is true, the data to be confirmed is inserted into the physical table to be inserted; Otherwise, as long as the verification result of any field is false, the data to be confirmed will be retained and the error reason for the verification failure will be recorded; In step e6, the following steps are specifically included: f1: read the fields in the data to be confirmed one by one, and record the user uploaded data corresponding to the field as: field data to be verified; Read the verification rule corresponding to the field in the verification item to be executed, and record it as: verification rule to be executed; f2: Identify the verification rule to be executed; If the verification rule to be executed includes enumeration verification or regularization verification, it is recorded as: verification rule to be confirmed, and step f3 is executed; Otherwise, execute step f4; f3: Read the verification rules to be confirmed and perform the following steps: f31: confirm whether the enumeration check is recognized, if so, execute step f32; Otherwise, execute step f33; f32: Determine whether the regular expression is empty. If so, it indicates that the verification to be confirmed is a separate enumeration verification; execute step f34; Otherwise, it is determined that the verification rule to be confirmed is enumeration and regularization verification, and step f36 is executed; f33: Determine whether the enumeration check is empty. If so, it means that the check to be confirmed is a separate regular check; execute step f35; Otherwise, it is determined that the verification rule to be confirmed is enumeration and regularization verification, and step f36 is executed; f34: perform enumeration check on the field data to be checked, and then execute step f5; f35: perform regular check on the field data to be checked, and then execute step f5; f36: for the field data to be checked, perform enumeration check and regular check in sequence according to the order in the preset expression, and then execute step f5; f4: executing the verification rule to be executed for the field data to be verified; f5: confirm the number of verification rules to be executed corresponding to the field data to be verified; If the number is greater than 1, all verification results must be true before the verification result of the field data to be verified can be set to true; Otherwise, as long as there is a verification result that is false, the verification result of the field data to be verified is set to false.

[0009] The present application provides a data processing method based on dynamic configuration, which constructs a data configuration page based on each basic data, sets a collection information configuration page, uniformly specifies all terminal users for each basic data through a user-specified module, uniformly sets corresponding verification rules for all fields of the basic data through a verification rule selection module, and sets a rule management window at the same time so that all verification rules configured on the verification rule configuration page can be dynamically displayed in the rule configuration module; once the number of fields of a certain basic data or the verification rules corresponding to the fields change, as long as the verification rule configuration page is modified according to the needs, the modified results can be dynamically displayed through the rule management window without modifying the collection configuration page of each basic data, which greatly improves the work efficiency of system configuration; once the terminal users of a certain basic data change, all terminal users can be uniformly modified directly through the user-specified module in the collection page, without modifying the configuration of the terminal users one by one as in the prior art, which is not only efficient but also reduces the probability of errors. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a schematic diagram of the system structure of a data management platform in the prior art; Figure 2 This is a schematic diagram of the data form relationship in this application; Figure 3 This is a schematic diagram of the structure of the configuration module in this application; Figure 4 Configure the page example for basic information; Figure 5 An example of configuring a page for collection information; Figure 6 This is an example of a rule management window; Figure 7 Example of configuration page for validation rules; Figure 8 This is an example of rule table data; Fig. 9 For example, physical table; Fig.10 For example, the field table; Fig.11 An example of a template for data upload. DETAILED DESCRIPTION

[0011] In actual applications, every time the industry regulations are modified, the fields or field validation rules of the corresponding basic data are changed, while the end users who submit the data do not change frequently. For example, the basic data that needs to be submitted is: double-publicity administrative penalties. Users who need to submit this type of basic data include: Urban Management Bureau, Supervision Bureau, Environmental Protection Bureau and other departments. When the regulations related to administrative penalties change, what usually changes is the specific content of the administrative penalties, while the departments that need to submit administrative penalty data rarely change. Therefore, in this method, in order to achieve dynamic configuration, the idea of ​​data configuration is improved. In this application, the data configuration page is constructed with basic data as the core. In order to implement the data configuration page at the database level, it is also necessary to create related data tables, such as Figure 2 The specific contents shown include the following.

[0012] d1: The system will create a catalog table and a field table for all basic data; The directory table includes: directory primary key, directory name, directory code and other contents, such as: rules for filtering data when reporting statistics; A record in the catalog table corresponds to one basic data; The field table stores the contents of all fields included in all basic data; the field table includes: field table primary key, field name column_name, field comments comments_name, field rule column_rule_id, directory primary key; For example, the basic data is: double public administrative penalties, and the corresponding fields include: full name of the organization, penalty category, and legal person ID number; The number of fields included in each basic data is greater than or equal to 1.

[0013] d2: Create a user table for all end users to record all end user information; User table: user table primary key, user full name, user code.

[0014] d4: Build a directory-user correspondence table, establish a relationship between the user table and the directory table, and record all the basic data that each terminal user needs to submit; the directory-user correspondence table includes: the primary key of the relationship table, the primary key of the user table, and the primary key of the directory.

[0015] d5: The system will automatically read the corresponding field table content for each basic data record in the directory table and create a physical table for each basic data; for example, the basic data "double public administrative penalties" corresponds to a physical table, including 3 fields; The physical table is used to store the data content actually uploaded by the end user, which includes: physical table primary key, user code, user full name, field name, field comment. All data corresponding to the same basic data uploaded by all end users will be saved in the same physical table. In the physical table, the user code and user full name are used to distinguish which end user uploaded the data.

[0016] In practical applications, the actual storage tables in the data warehouse include entity tables, fact tables, dimension tables, etc. Catalog tables: contain field descriptions and field rules that correspond to the physical tables. The relationship between tables will be used in subsequent operations such as data cleaning. For example, find the field with the same name in the catalog table based on the field name in the physical table, and then obtain the rule configured for the current field.

[0017] In this application, a rule table is also created for all verification rules. The rule table includes: the rule table primary key, the rule name and the rule content. The field table and the rule table are correspondingly related through the rule ID. In practical applications, if the number of fields and rules is large, a rule field and rule relationship table can be created to establish a relationship between the two through the field table primary key and the rule table primary key. In this embodiment, the rule table primary key is directly saved in the field table to record the corresponding relationship between the two.

[0018] The present application includes a data processing method based on dynamic configuration, which includes the following steps.

[0019] S1: Set up a separate data configuration page for each type of basic data that needs to be uploaded, such as Figure 3 As shown; The data configuration page includes: basic information configuration page and collection information configuration page, such as Figure 4 and Figure 5 As shown; The basic information configuration page includes: all fields included in the basic data; The collection information configuration page includes: a verification rule selection module and a user-specified module; The validation rule selection module is used to configure all fields that need to be validated and the validation rules corresponding to each field; The user-specified module is used to specify all terminal users who need to upload this basic data.

[0020] The validation rules configured for each field in the validation rule selection module include: required validation, enumeration validation, regular expression validation, one or more combinations of enumeration and regular expression validation, and plug-in validation.

[0021] S2: Build the verification rule configuration page and the user information configuration page; In the validation rule configuration page, configure the corresponding validation rule for each field that needs to be validated; On the user information configuration page, configure the corresponding basic information for each terminal user.

[0022] Because in actual work, many fields and the corresponding verification rules of the fields can be reused in different basic data, such as the verification of ID card numbers and ID card numbers; or the verification of phone numbers and phone numbers. In order to prevent code redundancy and achieve code reuse, this method constructs a separate verification rule configuration page for the verification rules, and all the verification rules configured through the verification rule configuration page are stored separately in the rule table. In this way, the configuration page of each basic data can call all the verification rules through the rule management window, and flexibly realize the reuse of verification rules.

[0023] Similarly, because each terminal user may need to upload more than one basic data, a separate user table is built for the terminal user. Then, in the user-specified module on the data configuration page of different basic data, the relevant terminal users can be found among the terminal users recorded in the user table, and the configuration of the terminal users related to each basic data can be completed.

[0024] Once a field of basic data or the validation rule of a field changes, there is no need to adjust it for each user. Instead, after the adjustment is completed on the data configuration page of the basic data, the change will be directly reflected to each end user through the data upload target, which greatly improves the configuration efficiency.

[0025] like Figure 7 As shown, in the verification rule configuration page of this embodiment, the configuration pages of enumeration verification and regular verification are placed in the same configuration page; based on the verification rule configuration page, the configuration of three verification forms of enumeration verification, regular verification and enumeration and regular verification can be realized at the same time. The verification rule configuration page includes: a rule name input box, an enumeration regular expression input box, a separator input box and an enumeration value input box.

[0026] S3: Build a rule management window, display all the verification rules configured in the verification rule configuration page in the rule management window, and open the rule management window when selecting the verification rule for each field in the verification rule selection module; the rule management window is as follows: Figure 6 As shown; The rule management window includes all the verification rules, and each verification rule corresponds to a selection box; The selection box in the rule management window supports multiple selections, allowing each field to have more than or equal to 1 validation rule selected at the same time.

[0027] For example, for the basic data: double public administrative penalty table, the corresponding fields include: full name of the organization, penalty category, legal person ID number; the "legal person ID number" field is a required item in the long data transmission process, and the ID number also needs to be accurately verified. Therefore, the verification rules corresponding to this field include: required verification and ID number compliance verification. In the specific method of required verification, it can be determined whether the field is filled in by confirming whether the data is empty; the ID number verification can be implemented based on any verification method in the existing technology; in this embodiment, the compliance verification of the ID number is implemented based on regular verification. When the configuration administrator configures the verification rules for the "legal person ID number" field, the two verification rules of "required verification" and "ID number compliance verification" will be selected in the rule management window. Figure 5 In the example shown, the "Required" check is set in the collection information configuration page.

[0028] In this application, by setting the rule management window, in the verification rule selection module of the collection information configuration page, an open button is set behind each field, such as Figure 5 In the example, the verification rule selection module includes: indicator name, indicator code, collection length, required, operation, and annotation, where the indicator name is the field name and the indicator code is the field code.

[0029] A "Rule" button is set in the "Operation" behind each field, and the rule management window is displayed in a pop-up window. The verification rule associated field selected in the rule management window opened this time is marked by connecting the startup ID to ensure that the associated information of the field and the verification rule can be accurately saved in the field table, and the rule window can be flexibly applied to various basic data configuration scenarios with a small amount of code. Based on the rule management window, not only can the corresponding verification rule be flexibly selected for each field, but once the verification rule changes, only the relevant modifications need to be made in the verification rule configuration page. Before each start of the rule management window, the rule table is read first, and the latest rule table content is displayed in the current window. All changes in the verification rules can be dynamically displayed in all data configuration pages, which is especially suitable for application scenarios with a large number of basic data types. It can greatly improve configuration efficiency and reduce the probability of errors.

[0030] S4: When it is necessary to configure the basic data, firstly, the corresponding verification rules will be added or modified through the verification rule configuration page according to specific needs, and the user information will be added or modified through the user information configuration page.

[0031] For example, a new field “Administrative Penalty Category” is added for the basic data “Double Publicity Administrative Penalty”, and its corresponding data content includes: warning; fine; confiscation of illegal income and illegal property; Fig.10 The following is an example of the field table dp_logic_column. In the table, logic_table_id is the primary key of the directory table dp_logic_table.

[0032] The validation rule for the newly added field "Administrative Penalty Category" is enumeration validation: Add new enumeration values, N of which can be configured, such as: Enumeration 1: Warning Enumeration 2: Fine Enumeration 3: Confiscation of illegal gains, confiscation of illegal property. Set separators between enumeration values, such as: ";", "," . The data will be saved in the rule table, as shown below Figure 8 shown.

[0033] Then, according to the configuration operation type involved in this number, subsequent configuration operations are performed. Common configuration tasks include: adding basic data, modifying the fields and verification rules of basic data, or adding or adjusting terminal users.

[0034] S5: Confirm the type of configuration operation to be performed according to the requirements; If a new type of basic data needs to be added, step S6 is executed; If it is necessary to modify the fields and rules of the existing basic data, execute step S7; If the corresponding terminal user is to be modified for the existing basic data, step S8 is executed.

[0035] S6: When adding basic data, perform the following steps: a1: Based on the basic information configuration page in the data configuration page, configure all the fields included in the newly added basic data; When the configuration administrator creates new basic data in the data configuration page, the background will automatically generate the corresponding catalog table; a2: Based on the verification rule selection module in the collection information configuration page, set all the fields that need to be verified, and select the verification rules corresponding to the fields in the rule management window to complete the operation; When the configuration administrator configures all fields and corresponding validation rules for the newly added basic data, the relevant content will be reflected in the self-reading table; a3: Based on the user-specified module in the collection information configuration page, select all terminal users who need to upload new basic data; For example, the basic data is: double public disclosure of administrative penalties, and its corresponding end users are: Urban Management Bureau, Supervision Bureau, Environmental Protection Bureau; The relationship between the terminal user and the basic data is established through the user-specified module, and the specific relationship is reflected and saved in the directory user correspondence table.

[0036] At the same time, every time a new field is added, the system will add a new field to this physical table according to the table name and field name of the directory table configured on the page. The number of physical tables corresponding to each basic data is consistent with the number of directories; at the same time, the physical table also includes the terminal users specified in the directory specified module. In order to ensure that the modification can be implemented in the physical table, in this method, once the content of the directory table and field table changes, the system will update the relevant content of the corresponding physical table in real time.

[0037] a4: Execute step S9.

[0038] S7: When making changes to existing basic data, perform the following steps; b1: Confirm the rules for adding new fields or modifying existing fields; If it is a new field, go to step b2; If it is a rule to modify an existing field, execute step b3; b2: Enter the basic data collection information configuration page, add a new field based on the verification rule selection module, select the verification rule corresponding to the new field based on the rule management window to complete the operation; execute step S9; Establish a relationship between fields and validation rules through the rule management window. The specific content is reflected in the field table. The field table establishes a relationship with the directory through the directory primary key, and establishes a relationship with the rule table through the rule table primary key. b3: Enter the basic data collection information configuration page, find the existing field that needs to be modified, open its corresponding rule management window, and reselect the verification rule corresponding to the existing field as required; execute step S9; If the name of an existing field is modified, the modified field name is directly stored in the field name in the field table. If the verification rule corresponding to the field is modified, the modification is reflected in the content of the primary key of the corresponding rule table in the field table. Execute step S9.

[0039] For example, a new field "Administrative Penalty Decision Document Number" needs to be added to the basic data "Double Publicity Administrative Penalty Table". The configuration administrator opens the data configuration page corresponding to the "Double Publicity Administrative Penalty", and adds the field "Administrative Penalty Decision Document Number" in the basic information configuration page. The content of this field is automatically saved in the field table, and the "Administrative Penalty Decision Document Number" field can also be used in other pages to call this field; the configuration administrator selects the corresponding data verification rule for the "Administrative Penalty Decision Document Number" through the verification rule selection module in the collection information configuration page. If this field is a required item, the required verification must also be selected, and the verification rule information corresponding to the field is also reflected in the field table.

[0040] S8: When modifying the corresponding terminal user for the existing basic data, perform the following steps: c1: Open the data configuration page of the basic data and modify the terminal user based on the user-specified module; The association between end users and basic data is reflected in the directory user correspondence table; c2: Execute step S9.

[0041] S9: Check whether there are other configurations. If yes, execute steps S5 to S8 in a loop. If all configuration work is completed, a data upload template is generated for each terminal user to complete this configuration operation. Fig.11 The following is an example of a data upload template.

[0042] This embodiment takes the basic data: double public administrative penalty table as an example to illustrate the creation process of basic data.

[0043] The corresponding catalog table for the double public disclosure of administrative penalties is: SGS_XZCF. In the data configuration page, configure the relevant content for it: The fields include: organization name jgqc, penalty category cflb, legal person ID number id_card; Its associated end users are: dept_id | dept_name | dept_code d0001 Urban Management Bureau A001 d0002 Market Supervision Bureau A002 d0003 EPA A003; The system generates a new directory data as follows.

[0044] id | table_name |table_code ... 1001 Double Publicity Administrative Penalty SGS_XZCF; The field table automatically adds 3 new data: id |column_name|comments_name |column_rule_id|logic_table_id 001jgqc Organization name empty 1001 002cflb penalty category empty 1001 003id_card Legal person ID card blank 1001 In the data configuration page, verification rules are configured for the penalty category and legal person ID card. The verification rule-related data recorded in the field table are as follows Id |column_name | comments_name | column_rule_id | logic_table_id 001jgqc Organization name empty 1001 002cflb Penalty Category Rule 1 1001 003id_card Legal person ID card rule 2 1001 In this embodiment, the rule table is associated with column_rule_id, and the directory table is associated with logic_table_id.

[0045] When the user designation module is used to designate the associated end user for the double public administrative penalty, three new data items will be added to the directory user correspondence table: id | dept_id | logic_table_id ... x001 d0001 1001 x001 d0002 1001 x001 d0003 1001 In the directory-user correspondence table, the directory and the terminal user are bound.

[0046] The system creates a physical table for the double-publicity administrative penalties, with fields including: full name of the institution, penalty category, and legal person ID number. Fig. 9 The following is an example of a physical table. The table contains the following fields: DEPT_CODE (department code), CF_XDR_MC (full name of the organization), penalty category 1 (CF_CFLB1), etc. All departments use this table to upload administrative penalty data, and the data is distinguished by DEPT_CODE. The primary key and DEPT_CODE are common fields, which exist in every table and are automatically created by the system without the need to configure on the page.

[0047] Finally, based on the directory table, field table and directory user correspondence table, the system generates a data upload template for each terminal user.

[0048] The data upload template includes: user code, user full name, directory code, directory name, field name column_name, and field comments_name.

[0049] Each data upload template corresponds to one kind of basic data; in the data upload template, each row includes all the fields of the basic data that need to be checked.

[0050] If the data uploaded by the user according to the data upload template passes the rule verification, the data will be stored in the corresponding physical table. For example: In this example, three departments reported data, and three new records will be added to the corresponding physical table: id | dept_id | jgqc |cflb | id_card 01 d0001 Company Name 1 Warning 32050xxxxxxxxxxx 02 d0002 Company name 2 Penalty 32050xxxxxxxxxxx 03 d0003 Company name 3 Penalty 32050xxxxxxxxxxx Once a verification rule associated with the double-public administrative penalty changes, you only need to make relevant modifications in the verification rule configuration page and save the changes to the rule table. The configuration administrator can complete the modification by binding the new rule to the corresponding field based on the rule management window. The modified information will overwrite the original field rule relationship.

[0051] In this method, based on the background program, after each modification, the physical table and the data upload template will automatically adjust according to the changes in the directory table and the field table, ensuring that the user uploads data in accordance with the latest regulations.

[0052] When each terminal user reports data, the system first confirms the user's permissions based on the user's login ID, then associates and queries the directory table based on the directory user correspondence table to find the directory content corresponding to the current user, that is, the basic data; and displays all the basic data that the user can upload on the user's upload page. On the upload page, the user selects the data upload template corresponding to the basic data to be uploaded this time, and then downloads it.

[0053] The data uploading process of the terminal user in this method specifically includes the following steps: e1: The end user uploads data to the system based on the data upload template; e2: Start a thread to parse the files uploaded by the terminal user; e3: Get the data included in each row of the data upload template uploaded by the terminal user, recorded as: data to be confirmed; The data to be confirmed include: user code, user full name, directory code, directory name, field name column_name, field comments_name and field value; e4: According to the fields included in the data to be confirmed, find the verification items that need to be performed for each field in the data to be confirmed, and put all the verification rules corresponding to the data to be confirmed into a set, which is recorded as: verification items to be executed; e5: Confirm the physical table corresponding to the terminal user according to the user code and user full name, recorded as: physical table to be inserted; e6: Execute all pending verification items for each row of data to be confirmed; e7: Confirm the execution results of the verification items to be executed; If all the verification items to be executed meet the verification requirements and the result is true, the data to be confirmed will be inserted into the physical table to be inserted; Otherwise, as long as the verification result of any field is false, the data to be confirmed will be retained and the error reason for the verification failure will be recorded.

[0054] The data uploaded by the user will be saved in the specified physical table. The data in the physical table is distinguished by which department the data was collected based on the department code DEPT_CODE.

[0055] For example, for the three fields of double public administrative penalties: full name of the organization, penalty category, and legal person ID number, these three fields are required options, and their corresponding verification rules at least include: required verification and the verification rules of the data itself. When the verification items are finally executed, all verification results must be true, and the corresponding data can be saved in the physical table.

[0056] In step e6, the following steps are specifically included: f1: Read the fields in the data to be confirmed one by one, and record the user uploaded data corresponding to the field as: field data to be verified; Read the verification rule corresponding to the field in the verification item to be executed, and record it as: verification rule to be executed; f2: Identify the verification rules to be executed; If the verification rule to be executed includes enumeration verification or regularization verification, it is recorded as: verification rule to be confirmed, and step f3 is executed; Otherwise, execute step f4; f3: Read the verification rules to be confirmed and perform the following steps: f31: confirm whether the enumeration check is recognized, if so, execute step f32; Otherwise, execute step f33; f32: Determine whether the regular expression is empty. If so, it means that the verification to be confirmed is a separate enumeration verification; execute step f34; Otherwise, it is determined that the verification rule to be confirmed is enumeration and regularization verification, and step f36 is executed; f33: Determine whether the enumeration check is empty. If so, it means that the check to be confirmed is a separate regular check; execute step f35; Otherwise, it is determined that the verification rule to be confirmed is enumeration and regularization verification, and step f36 is executed; f34: Perform enumeration check on the field data to be checked, and then execute step f5; f35: Perform regular check on the field data to be checked, and then execute step f5; f36: For the field data to be checked, perform enumeration check and regular check in sequence according to the order in the preset expression, and then execute step f5; f4: Execute the pending verification rules for the field data to be verified; f5: confirm the number of verification rules to be executed corresponding to the field data to be verified; If the number is greater than 1, all verification results must be true before the verification result of the field data to be verified can be set to true; Otherwise, as long as there is a verification result of false, the verification result of the field data to be verified is set to false.

[0057] In common data validation methods, enumeration validation and regular validation are configured separately. However, in actual applications, it is found that some data cannot meet all validation requirements based on only one validation method. For example: The field "Administrative penalty category" corresponds to the following data: warning; fine; confiscation of illegal gains, confiscation of illegal property; temporary seizure or revocation of permits, temporary seizure or revocation of licenses; administrative detention; other administrative penalties prescribed by laws and administrative regulations (such as: other - administrative detention; other - fine of 1,000, etc.); Usually, for this type of field, the data content is checked through enumeration verification.

[0058] 1) Take out the enumeration value, split it into strings by ";", and put it into the collection mjMap; 2) Take out the field value, split it into strings by ";", put it into the collection columnValueMap, loop through the columnValueMap collection in mjMap, and compare the values ​​in mjMap and columnValueMap one by one. If there is a value in the columnValueMap collection that does not exist in mjMap, it means that the enumeration check fails. If both exist, it means that the enumeration check passes.

[0059] However, in actual work, the actual data content corresponding to "other" in "warning; fine; restriction on employment; confiscation of illegal income; other" is "other-XXXXX", and the content after "other-" is not fixed and unpredictable. For example, it may be: Other-administrative detention for 3 days, or it may be: Other-fine of 1,000 yuan. Therefore, the content of the field "administrative penalty category" cannot be completed by enumeration verification alone to complete data compliance check.

[0060] For example, the enumeration values ​​in the rules are: warning; fine; confiscation of illegal gains and illegal property; restriction on employment; confiscation of illegal gains; confiscation of illegal property; The field values ​​entered by the department are: warning; fine; restriction on employment; confiscation of illegal income; verification passed; The field values ​​entered by the department are: warning; fine; restriction on employment; fine of 100 yuan; verification failed; In this method, the field "Administrative Penalty Category" is verified by enumeration + verification.

[0061] For example: The regular expression in the current example is: Others-[\u4e00-\u9fa5a-zA-Z-z0-9]+((?!;|;).)*$ Indicates that any character must be matched until it ends with another -XXX; If the field value is: warning; fine; restriction on employment; confiscation of illegal income; other-xxx. Verification passed.

[0062] The field value is: warning; fine; restricted employment; other xxxx. The verification failed.

[0063] For the verification of the field "Administrative Penalty Category", both the regular expression and the enumeration must be true to indicate that the verification has passed.

[0064] In actual work, there are also fields that require this combination of methods to complete data verification. In this method, in order to avoid redundancy in the configuration page, Figure 7 In the validation rule configuration page shown in the figure, the configuration pages for enumeration validation and regular validation are set together. If you only need to configure enumeration validation, or only need to configure regular validation, just leave the other validation method blank. If you need to configure both "enumeration + validation", then Figure 7 As shown in , just fill in both "Enumeration Regular Expression" and "Enumeration Value".

[0065] At the same time, in step e6, when executing all the pending verification items for each row of data to be confirmed, it is necessary to consider the three verification forms of regular verification, enumeration verification and regular + enumeration respectively in steps f2 and f3. This method can flexibly meet the data verification requirements of various situations by adding the regular + enumeration verification mode. No human participation in judgment is required, and the complexity of the system is reduced.

Claims

1. A data processing method based on dynamic configuration, characterized in that: It includes the following steps: S1: Set up a separate data configuration page for each type of basic data that needs to be uploaded; The data configuration page includes: a basic information configuration page and a collection information configuration page; The basic information configuration page includes: all fields included in the basic data; The collection information configuration page includes: a verification rule selection module and a user specified module; The verification rule selection module is used to configure all fields that need to be verified and the verification rules corresponding to each field; The user specifying module is used to specify all terminal users who need to upload the basic data; S2: Build the verification rule configuration page and the user information configuration page; In the verification rule configuration page, configure the corresponding verification rule for each field that needs to be verified; In the user information configuration page, configure corresponding basic information for each terminal user; S3: construct a rule management window, display all the verification rules configured in the verification rule configuration page in the rule management window, and open the rule management window when the verification rule for each field is selected in the verification rule selection module; The rule management window includes all the verification rules, and each of the verification rules corresponds to a selection box; The selection box in the rule management window is a selection box that supports multiple selections, and allows each field to select more than or equal to 1 verification rule at the same time; S4: When it is necessary to configure the basic data, according to specific needs, add or modify the corresponding verification rules through the verification rule configuration page, and complete the addition or modification of user information through the user information configuration page; S5: Confirm the type of configuration operation to be performed according to the requirements; If a new type of basic data needs to be added, step S6 is executed; If it is necessary to modify the fields and rules of the existing basic data, execute step S7; If the corresponding terminal user is to be modified for the existing basic data, step S8 is executed; S6: When adding basic data, perform the following steps: a1: Based on the basic information configuration page in the data configuration page, configure all the fields included in the newly added basic data; a2: Based on the verification rule selection module in the collection information configuration page, set all the fields that need to be verified, and select the verification rule corresponding to the field based on the rule management window to complete the operation; a3: Based on the user specified module in the collection information configuration page, select all terminal users who need to upload the newly added basic data; a4: execute step S9; S7: When making changes to existing basic data, perform the following steps; b1: Confirm the rules for adding new fields or modifying existing fields; If it is a new field, go to step b2; If it is a rule to modify an existing field, execute step b3; b2: Enter the collection information configuration page of basic data, add a new field based on the verification rule selection module, select the verification rule corresponding to the new field based on the rule management window to complete the operation; execute step S9; b3: Enter the collection information configuration page of the basic data, find the existing field that needs to be modified, open its corresponding rule management window, and reselect the verification rule corresponding to the existing field as required; execute step S9; S8: When modifying the corresponding terminal user for the existing basic data, perform the following steps: c1: Open the data configuration page of the basic data and modify the terminal user based on the user specified module; c2: execute step S9; S9: Check whether there are other configurations. If yes, execute steps S5 to S8 in a loop. If all configuration work is completed, a data upload template is generated for the end user to complete this configuration operation.

2. The data processing method based on dynamic configuration according to claim 1, characterized in that: It also includes the following steps: d1: Create a directory table and a field table for all basic data; The directory table includes: directory primary key, directory name and directory code; One record in the directory table corresponds to one basic data; The field table stores the contents of all fields included in all basic data; the field table includes: field table primary key, field name column_name, field comments comments_name, field rule column_rule_id, directory primary key; The number of fields included in each basic data is greater than or equal to 1; d2: Create a user table for all end users to record all end user information; The user table: user table primary key, user full name, user code; d4: construct a directory user correspondence table, establish a relationship between the user table and the directory table, and record all basic data that each terminal user needs to submit; the directory user correspondence table includes: the primary key of the relationship table, the primary key of the user table, and the primary key of the directory; d5: For each basic data record in the directory table, read the corresponding field table content and create a physical table for each basic data; The physical table is used to store the data content actually uploaded by the terminal user, which includes: physical table primary key, user code, user full name, field name, and field comment.

3. The data processing method based on dynamic configuration according to claim 1, characterized in that: A rule table is created for all verification rules, and the rule table includes: a rule table primary key, a rule name, and a rule content.

4. The data processing method based on dynamic configuration according to claim 1, characterized in that: The verification rules configured for each field in the verification rule selection module include: required verification, enumeration verification, regular verification, one or more combinations of enumeration and regular verification and plug-in verification.

5. The data processing method based on dynamic configuration according to claim 1, characterized in that: In the verification rule configuration page, the configuration pages of enumeration verification and regularization verification are placed in the same configuration page; based on the verification rule configuration page, the configuration of enumeration verification, regularization verification, and enumeration and regularization verification is implemented; The verification rule configuration page includes: a rule name input box, an enumeration regular expression input box, a separator input box and an enumeration value input box.

6. The data processing method based on dynamic configuration according to claim 1, characterized in that: The data upload template includes: user code, user full name, directory code, directory name, field name, and field comment; Each of the data upload templates corresponds to one type of basic data; in the data upload template, each row includes all the fields of the basic data that need to be checked.

7. The data processing method based on dynamic configuration according to claim 6, characterized in that: It also includes the data uploading process, which specifically includes the following steps: e1: The terminal user uploads data to the system based on the data upload template; e2: Start a thread to parse the files uploaded by the terminal user; e3: Get the data included in each row of the data upload template uploaded by the terminal user, recorded as: data to be confirmed; The data to be confirmed include: user code, user full name, directory code, directory name, field name, field comment and field value; e4: according to the fields included in the data to be confirmed, find the verification items that need to be performed for each field in the data to be confirmed, and put all the verification rules corresponding to the data to be confirmed into a set, which is recorded as: verification items to be executed; e5: Confirm the physical table corresponding to the terminal user according to the user code and user full name, recorded as: physical table to be inserted; e6: Execute all the verification items to be executed for each row of the data to be confirmed; e7: Confirm the execution result of the verification item to be executed; If all the verification items to be executed meet the verification requirements and the result is true, the data to be confirmed is inserted into the physical table to be inserted; Otherwise, as long as the verification result of any field is false, the data to be confirmed is retained and the error reason for the verification failure is recorded.

8. The data processing method based on dynamic configuration according to claim 7, characterized in that: In step e6, the following steps are specifically included: f1: read the fields in the data to be confirmed one by one, and record the user uploaded data corresponding to the field as: field data to be verified; Read the verification rule corresponding to the field in the verification item to be executed, and record it as: verification rule to be executed; f2: Identify the verification rule to be executed; If the verification rule to be executed includes enumeration verification or regularization verification, it is recorded as: verification rule to be confirmed, and step f3 is executed; Otherwise, execute step f4; f3: Read the verification rules to be confirmed and perform the following steps: f31: confirm whether the enumeration check is recognized, if so, execute step f32; Otherwise, execute step f33; f32: Determine whether the regular expression is empty. If so, it indicates that the verification to be confirmed is a separate enumeration verification; execute step f34; Otherwise, it is determined that the verification rule to be confirmed is enumeration and regularization verification, and step f36 is executed; f33: Determine whether the enumeration check is empty. If so, it means that the check to be confirmed is a separate regular check; execute step f35; Otherwise, it is determined that the verification rule to be confirmed is enumeration and regularization verification, and step f36 is executed; f34: perform enumeration check on the field data to be checked, and then execute step f5; f35: perform regular check on the field data to be checked, and then execute step f5; f36: for the field data to be checked, perform enumeration check and regular check in sequence according to the order in the preset expression, and then execute step f5; f4: executing the verification rule to be executed for the field data to be verified; f5: confirm the number of verification rules to be executed corresponding to the field data to be verified; If the number is greater than 1, all verification results must be true before the verification result of the field data to be verified can be set to true; Otherwise, as long as there is a verification result that is false, the verification result of the field data to be verified is set to false.

Images