A data extraction and processing method based on user credit features

Abstract

The application discloses a data extraction and processing method and system based on user credit characteristics, comprising the following steps: S1, configuring variable expression and analysis type according to standard JSON format; S2, request party imports credit investigation report, and the credit investigation report data format is HTML; S3, converting the credit investigation report into standard JSON; S4, querying all variable expressions from a database for cyclic processing; S5, constructing variable expression into an analysis object set according to rules according to the analysis type; S6, performing analysis operation calculation variable value according to the analysis object set according to the imported standard JSON data; S7, encapsulating the analyzed variable value into a warehouse for storage and return. When facing complex and diversified business scenarios and customer demands, new variables can be derived flexibly and dynamically through the configuration of variable expressions, the derived variables can be directly used without customized repeated development, and human development cost can be reduced.

Description

Technical Field

[0001] This invention relates to the field of computers, and more specifically to a method for extracting and processing data based on user credit characteristics. Background Technology

[0002] In the internet industry, more and more companies are focusing on risk control, which requires a large amount of data to support analysis. Credit reports are currently one of the most authoritative data sources. However, for companies to transform the data in credit reports into usable variables for risk control systems to make decisions, they must perform variable screening and development based on complex credit reports, which has significant limitations. The development of each variable requires development, testing, and verification processes before it can be put into use, which requires a significant investment of manpower and time. Furthermore, each iteration and update requires redefining and developing, necessitating the repeated investment of the same amount of manpower in development work. Summary of the Invention

[0003] To address the problems existing in the prior art, this invention provides a data extraction and processing method based on user credit features. When faced with complex and diverse business scenarios and customer needs, new variables can be flexibly and dynamically derived by configuring variable expressions. The derived variables can be used directly without customized and repetitive development, which can reduce human development costs.

[0004] The technical solution of this invention is: a method for data extraction and processing based on user credit features, comprising the following steps:

[0005] Step S1: Configure variable expressions and parsing types according to the standard JSON format; the configured variable expressions are used to create the rule logic strings required for the variables; standard JSON refers to the standard data format defined by the People's Bank of China for the second-generation credit reporting report;

[0006] Step S1 includes the following sub-steps:

[0007] S11. Select the modules that need to be processed in the standard JSON; a module refers to a block of objects that are classified according to different categories based on the data structure of the standard JSON format.

[0008] S12. Perform conditional filtering on the selected modules. Module conditional filtering is to filter and obtain valid values ​​by performing conditional filtering on a certain part of the KEY in the module selected in step S11.

[0009] S13. Perform either or multiple data merging between the filtered modules. When there are multiple modules with values, it is necessary to merge the data between multiple modules or under different filtering conditions in the same module.

[0010] S14. Finally, aggregate functions are applied to the merged data modules to obtain a unique value. Aggregate functions include taking a specific value, summing, summarizing, deduplicating summaries, scaling, maximum value, minimum value, rounding up, and Gini coefficient.

[0011] S2. The requesting party imports a credit report, and the credit report data format is HTML (Hypertext Markup Language);

[0012] S3. Convert the credit report to standard JSON format;

[0013] S4. Retrieve all variable expressions from the database and process them in a loop;

[0014] S5. Construct a set of parsing objects from variable expressions according to the parsing type and rules;

[0015] S6. Perform parsing operations on the incoming standard JSON data according to the parsing object set and calculate variable values;

[0016] S7. Encapsulate the parsed variable values ​​into a database for storage and return them.

[0017] Furthermore, the filtering conditions in step S12 are roughly divided into conditions such as maximum element, minimum element, recent days, recent months, recent years, greater than, less than, equal to, and not equal to, based on the data type. Multiple filtering conditions can be used for or combined filtering.

[0018] Furthermore, step S3 includes the following sub-steps:

[0019] S31. Use Jsoup (parser) to parse the HTML (Hypertext Markup Language) credit report, create a Document (document object) to obtain the parsed text content; then extract the body (data body) from the Document (document object) and return Elements (element list) to achieve the extraction and filtering of specified elements and remove invalid data;

[0020] S32. Map the Elements (list of elements) according to the standard JSON module structure and divide the table (table elements) into blocks;

[0021] S33. Retrieve the corresponding values ​​from the segmented table (table elements) according to the mapping relationship of standard JSON;

[0022] S34. Place the corresponding value into the corresponding JSON location in the hierarchical structure of standard JSON;

[0023] S35. Finally, summarize the values ​​of each sub-module into the final JSON (data format).

[0024] Furthermore, step S5 includes the following sub-steps:

[0025] S51. First, remove and save the module filtering conditions in the variable expression. The form of the module filtering conditions is [].

[0026] S52. Perform OR and AND operations on the removed expression and the filtering conditions respectively to generate the recursive structure ElementStr (element relationship string object) and Inner (filter condition object) of the basic unit block;

[0027] S53. Decompose the basic units in the two recursive structures into the smallest meaningful symbols or values ​​to form recursive objects. Then, put the filtering condition parts into the corresponding positions in the main structure to obtain the final Relation (parsing object set) object.

[0028] Furthermore, in step S6, since Relation (the set of parsed objects) is a recursive object, it will recursively loop down to the lowest level object, parse the result and return it to the parent. Objects at the same level will aggregate the results through a concatenation operator until the top-level structure is returned. The module filtering conditions are encapsulated in Element (element object), which includes three parts: condition value A, symbol, and condition value B. According to the different types of condition values, the corresponding element values ​​are taken for condition values ​​A and B respectively (condition value A is generally a key in standard JSON, and condition value B is generally a specific value, or it may be the same as A). Then, A and B are compared by symbol, and the result set that meets the conditions is returned to the upper layer for further processing. The aggregate function is linked in the form of abc. The results are processed by the corresponding function in sequence, and the unique value obtained is the final variable value.

[0029] Furthermore, variable expression parsing includes filtering based on elimination conditions, splitting basic element blocks, decomposing minimum element values, and aggregating element values.

[0030] The exclusion condition filtering is used to distinguish the condition filtering from the main expression, so as not to cause parsing confusion;

[0031] The basic element block splitting is used to split out the element block within the smallest bracket;

[0032] The minimum element value decomposition is used to further decompose each basic element block into the smallest granularity and type that the system can recognize, including module KEY, arithmetic operators, OR symbols, functions, and specific values.

[0033] The aggregated element value is used to reassemble the smallest element into a single recursive object.

[0034] Furthermore, variable expression operations include arithmetic operations, comparison operations, combination operations, function operations, and value assignment operations:

[0035] The four arithmetic operations are used to process the addition, subtraction, multiplication, and division operations that appear in the expression;

[0036] The comparison operation is used to process the comparison operator operation of condition A and B appearing in the expression filtering conditions;

[0037] The merging operation is used to process the merging of multiple comparison operations that occur in the expression filtering conditions.

[0038] The function operation is used to process the functions that appear in the expression;

[0039] The value retrieval operation is used to process the KEY element of standard JSON that appears in the expression, that is, to extract the corresponding data block in the input JSON.

[0040] The present invention adopts the above technical solution and has the following advantages:

[0041] 1. When facing complex and diverse business scenarios and customer needs, this invention can flexibly and dynamically generate new variables by configuring variable expressions. The generated variables can be used directly without customized and repetitive development, which can reduce human development costs.

[0042] 2. This invention processes credit reports into a standardized JSON structure, helping client organizations organize complex report information and configure derived variables in real time, enabling them to quickly filter out the data needed by the client.

[0043] 3. This invention provides an integrated process from report analysis to variable derivation, which can generate variables according to customer needs, provide customers with risk control decisions, help customers distinguish between good and bad customer groups, and judge lending risks. Attached Figure Description

[0044] Figure 1 The diagram shown is a flowchart of the implementation of this invention;

[0045] Figure 2 The diagram shown is a flowchart of the HTML (Hypertext Markup Language) conversion process implemented in this invention.

[0046] Figure 3 The diagram shown is a visualization configuration diagram of the variable expression implemented in this invention. Detailed Implementation

[0047] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and embodiments of the present invention.

[0048] like Figure 1 ,2 As shown, the present invention includes the following steps S1 to S7:

[0049] Step S1: Configure variable expressions and parsing types according to the standard JSON format; the configured variable expressions are used to create the rule logic strings required for the variables; standard JSON refers to the standard data format defined by the People's Bank of China for the second-generation credit reporting report;

[0050] Step S1 includes the following sub-steps:

[0051] S11. Select the modules that need to be processed in the standard JSON; a module refers to a block of objects that are classified according to different categories based on the data structure of the standard JSON format.

[0052] S12. Perform conditional filtering on the selected modules. Module conditional filtering is to filter and obtain valid values ​​for a certain part of the KEY in the module selected in step S11. The filtering conditions are roughly divided into the maximum element, minimum element, recent days, recent months, recent years, greater than, less than, equal to, not equal to, etc., according to the data type. Multiple filtering conditions can be used or combined for filtering.

[0053] S13. Perform either or multiple data merging between the filtered modules. When there are multiple modules with values, it is necessary to merge the data between multiple modules or under different filtering conditions in the same module.

[0054] S14. Finally, aggregate functions are applied to the merged data modules to obtain a unique value. Aggregate functions include taking a specific value, summing, summarizing, deduplicating summaries, scaling, maximum value, minimum value, rounding up, and Gini coefficient.

[0055] In embodiments of the present invention, such as Figure 3 As shown, the visual configuration is done through the front-end web page, or you can directly write expressions according to the format and store them in the database. An example of a variable expression is as follows:

[0056] P[A==1&&(B<2||C==1,2,3)].P1.sum

[0057] Where P represents the module number, [] represents the filter condition group under module P, A, B, and C represent the KEY under module P, and multiple values ​​of C can be separated by commas. Therefore, this table means to take all P1 elements under module P that meet the conditions [] and then sum them using the sum (sum) function.

[0058] Step S2: The requesting party imports the credit report. The credit report data format is HTML (Hypertext Markup Language).

[0059] Step S3: Convert the credit report into standard JSON;

[0060] Step S3 includes the following sub-steps:

[0061] S31. Use Jsoup (parser) to parse the HTML (Hypertext Markup Language) credit report, create a Document (document object) to obtain the parsed text content; then extract the body (data body) from the Document (document object) and return Elements (element list) to achieve the extraction and filtering of specified elements and remove invalid data;

[0062] S32. Map the Elements (list of elements) according to the standard JSON module structure and divide the table (table element) module elements into blocks;

[0063] S33. Retrieve the corresponding values ​​from the segmented table elements according to the mapping relationship in standard JSON;

[0064] S34. Place the corresponding value into the corresponding JSON location in the hierarchical structure of standard JSON;

[0065] S35. Finally, summarize the values ​​of each sub-module into the final JSON.

[0066] Step S4: Query all variable expressions from the database and process them in a loop;

[0067] Step S5: Construct a set of parsing objects from the variable expressions according to the parsing type and rules;

[0068] Step S5 includes the following sub-steps:

[0069] S51. First, remove and save the module filtering conditions in the variable expression. The form of the module filtering conditions is [].

[0070] S52. Perform OR and AND operations on the removed expression and the filtering conditions respectively to generate the recursive structure ElementStr (element relationship string object) and Inner (filter condition object) of the basic unit block;

[0071] S53. Decompose the basic units in the two recursive structures into the smallest meaningful symbols or values ​​to form recursive objects. Then, put the filtering condition parts into the corresponding positions in the main structure to obtain the final Relation (parsing object set) object.

[0072] Step S6: Perform parsing operations on the incoming standard JSON data according to the parsing object set to calculate variable values.

[0073] In this embodiment of the invention, Relation (the set of parsed objects) is a recursive object, so it will recursively return to the lowest level object, parse the result and return it to the parent. Objects at the same level will summarize the results through a concatenation operator until the top-level structure is returned. The module filtering conditions are encapsulated in Element (element object), which includes three parts: condition value A, symbol, and condition value B. According to the different types of condition values, the corresponding element values ​​of condition values ​​A and B are taken respectively (condition value A is generally a key in standard JSON, and condition value B is generally a specific value, or it may be the same as A). Then, A and B are compared by symbol, and the result set that meets the conditions is returned to the upper layer for further processing. The aggregate function is linked in the form of abc. The results are processed by the corresponding function in sequence, and the unique value is the final variable value.

[0074] Variable expression parsing includes conditional filtering, basic element block splitting, minimum element value decomposition, and aggregated element value: Conditional filtering is used to distinguish conditional filtering from the main expression to avoid parsing confusion; Basic element block splitting is used to split the element blocks within the smallest brackets; Minimum element value decomposition is used to further decompose each split basic element block into the smallest recognizable granularity and type, including module KEY, arithmetic operators, OR symbols, functions, and specific values; Aggregated element value is used to combine the minimum elements and encapsulate them into a whole recursive object.

[0075] Variable expression operations include arithmetic operations, comparison operations, merging operations, function operations, and value retrieval operations: the arithmetic operations are used to handle addition, subtraction, multiplication, and division operations in the expression; the comparison operations are used to handle the comparison of conditions A and B in the expression's filtering conditions; the merging operations are used to handle the merging of multiple comparison operations in the expression's filtering conditions; the function operations are used to handle function processing in the expression; and the value retrieval operations are used to handle the KEY element of standard JSON in the expression, that is, to extract the corresponding data block from the input JSON.

[0076] Step S7: Encapsulate the parsed variable values ​​into the database for storage and return.

[0077] This embodiment addresses complex and diverse business scenarios and customer needs by flexibly and dynamically generating new variables through variable expression configuration. These derived variables can be used directly without requiring repetitive customized development, reducing human development costs. Credit reports are uniformly processed into a standardized JSON structure, helping client institutions organize complex report information. Real-time configuration of derived variables allows for rapid filtering of the data needed by the client. An integrated workflow from report parsing to variable derivation is provided, generating variables based on customer needs for risk control decisions, helping clients differentiate between good and bad customer groups, and assess lending risks.

[0078] The above description is only a preferred embodiment of the present invention. To help readers understand the principle of the present invention, it should be noted that for those skilled in the art, the scope of the present invention is not limited thereto. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for data extraction and processing based on user credit features, characterized in that: Includes the following steps: Step S1: Configure variable expressions and parsing types according to the standard JSON format; The configuration variable expression is used to create the rule logic string required for the variable; Standard JSON refers to the standard data format defined by the People's Bank of China for second-generation credit reports; Step S1 includes the following sub-steps: S11. Select the modules that need to be processed in the standard JSON; a module refers to a block of objects that are classified according to different categories based on the data structure of the standard JSON format. S12. Perform conditional filtering on the selected modules. Module conditional filtering is to filter and obtain valid values ​​by performing conditional filtering on a certain part of the KEY in the module selected in step S11. S13. Perform either or multiple data merging between the filtered modules. When there are multiple modules with values, it is necessary to merge the data between multiple modules or under different filtering conditions in the same module. S14. Finally, aggregate functions are applied to the merged data modules to obtain a unique value. Aggregate functions include taking a specific value, summing, summarizing, deduplicating summaries, scaling, maximum value, minimum value, rounding up, and Gini coefficient. Step S2: The requesting party imports the credit report, and the credit report data format is HTML; Step S3: Convert the credit report into standard JSON; In step S3 It includes the following steps: S31. Use Jsoup to parse the HTML credit report, create a Document to obtain the parsed text content; then extract the body from the Document, return Elements, implement the extraction and filtering of specified elements, and remove invalid data; S32. Map the Elements according to the standard JSON module structure and divide the table into blocks; S33. Retrieve the corresponding values ​​from the segmented table according to the mapping relationship of standard JSON; S34. Place the corresponding value into the corresponding JSON location in the hierarchical structure of standard JSON; S35. Finally, summarize the values ​​of each sub-module into the final JSON; Step S4: Query all variable expressions from the database and process them in a loop; Step S5: Construct a set of parsing objects from the variable expressions according to the parsing type and rules; Step S5 includes the following sub-steps: S51. First, remove and save the module filtering conditions in the variable expression. The form of the module filtering conditions is []. S52. Perform OR and AND operations on the removed expressions and filtering conditions respectively to generate the recursive structures ElementStr and Inner of the basic unit blocks. S53. Decompose the basic units in the two recursive structures into the smallest meaningful symbols or values, form a recursive object, and then put the filtering condition part into the corresponding position of the main structure to obtain the final Relation. Step S6: Perform parsing operations on the incoming standard JSON data according to the parsing object set and calculate the variable values; Step S7: Encapsulate the parsed variable values ​​into the database for storage and return.

2. The method for data extraction and processing based on user credit features according to claim 1, characterized in that: The filtering conditions in step S12 are divided into maximum element, minimum element, recent days, recent months, recent years, greater than, less than, equal to, and not equal to according to data type. Multiple filtering conditions can be used or combined for filtering.

3. The method for data extraction and processing based on user credit features according to claim 1, characterized in that: In step S6, since Relation is a recursive object, it will recursively go to the lowest level object, parse the result and return it to the parent. Objects at the same level will summarize the result through the concatenation operator until the top level structure is returned. The module filtering conditions are encapsulated in Element, which includes three parts: condition value A, symbol, and condition value B. Based on the different types of condition values, the corresponding element values ​​of condition values ​​A and B are retrieved respectively. Then, A and B are compared by symbol, and the result set that meets the conditions is returned to the upper layer for further processing. The aggregate function is linked in the form of abc. The results are processed by the corresponding function in sequence, and the unique value obtained is the final variable value.

4. A method for data extraction and processing based on user credit features according to claim 1 or 3, characterized in that: Variable expression parsing includes filtering based on elimination criteria, splitting basic element blocks, decomposing minimum element values, and aggregating element values. The exclusion condition filtering is used to distinguish the condition filtering from the main expression, so as not to cause parsing confusion; The basic element block splitting is used to split out the element block within the smallest bracket; The minimum element value decomposition is used to further decompose each basic element block into the smallest granularity and type that the system can recognize, including module KEY, arithmetic operators, OR symbols, functions, and specific values. The aggregated element value is used to reassemble the smallest element into a single recursive object.

5. A method for data extraction and processing based on user credit features according to claim 1 or 3, characterized in that: Variable expression operations include arithmetic operations, comparison operations, combination operations, function operations, and value assignment operations: The four arithmetic operations are used to process the addition, subtraction, multiplication, and division operations that appear in the expression; The comparison operation is used to process the comparison operator operation of condition A and B appearing in the expression filtering conditions; The merging operation is used to process the merging of multiple comparison operations that occur in the expression filtering conditions. The function operation is used to process the functions that appear in the expression; The value retrieval operation is used to process the KEY element of standard JSON that appears in the expression, that is, to extract the corresponding data block in the input JSON.

Images