A data cleaning processing method, device and equipment and a computer readable storage medium

Abstract

The application discloses a kind of data cleaning processing method, device equipment and computer readable storage medium, applied to data processing technical field, comprising: obtaining different data format data to be cleaned;Wherein, data to be cleaned is credit data;Data to be cleaned is analyzed, and data to be cleaned analysis data is obtained;Wherein, data analysis is at least one of data type analysis, data content analysis and data separator analysis according to data format;Utilize cleaning rule to clean processing to data to be cleaned analysis data, and obtain cleaned data;Wherein, cleaning rule includes at least one of repeated data cleaning, missing data cleaning, invalid data cleaning and error data cleaning.Compared with current different format data cannot be unified cleaning, the credit data of different format from different enterprises can be unified cleaning processing by the application, and the cleaning data required by each enterprise is obtained, so that the efficiency of data cleaning processing is improved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a data cleaning and processing method, apparatus, and computer-readable storage medium. Background Technology

[0002] Current data cleaning and processing methods typically involve pre-developing cleaning rules and defining a streamlined process to create cleaning definitions across fixed table structures. This approach fails to allow for the reuse of cleaning rules. When cleaning is required for new data formats, the rules cannot be reused and must be redeveloped and configured. Therefore, current data cleaning and processing methods are not reusable across different data formats. Summary of the Invention

[0003] In view of this, the purpose of the present invention is to provide a data cleaning and processing method, apparatus, and computer-readable storage medium, which solves the technical problem in the prior art that it is impossible to uniformly clean and process corporate credit data of different formats, resulting in low cleaning and processing efficiency.

[0004] To solve the above-mentioned technical problems, the present invention provides a data cleaning and processing method, comprising:

[0005] Acquire data to be cleaned in different data formats; wherein, the data to be cleaned is credit data;

[0006] The data to be cleaned is parsed to obtain the parsed data to be cleaned; wherein, the data parsing is at least one of the following: data type parsing, data content parsing, and data delimiter parsing according to the data format;

[0007] The data to be cleaned is processed using cleaning rules to obtain cleaned data; wherein the cleaning rules include at least one of duplicate data cleaning, missing data cleaning, invalid data cleaning, and erroneous data cleaning.

[0008] Optionally, acquiring data to be cleaned in different data formats includes:

[0009] The data to be cleaned is obtained using secure file transfer protocols, file transfer protocols, cloud storage services, and application programming interfaces (APIs) in different data formats.

[0010] Optionally, acquiring data to be cleaned in different data formats includes:

[0011] It can obtain credit information, loan information, detailed information, summary information, loan balance information, disbursement information, repayment information, and loan overdue information in different data formats.

[0012] Optionally, the step of cleaning the data to be cleaned using cleaning rules to obtain cleaned data includes:

[0013] The data to be cleaned is subjected to pre-loan authorization list verification, pre-loan user list verification, and hash function verification to obtain the cleaned data.

[0014] Optionally, after processing the data to be cleaned using cleaning rules to obtain cleaned data, the method further includes:

[0015] The cleaned data is processed using data mapping rules to obtain target cleaned data in different formats corresponding to the cleaned data; wherein, the data mapping rules are data mapping rules set according to different tax bureau data formats;

[0016] The cleaned target data is routed to each tax processor based on the enterprise's tax number.

[0017] Optionally, processing the cleaned data using data mapping rules includes:

[0018] The cleaned data is processed using feedback interface mapping rules, feedback message format mapping rules, feedback field mapping rules, field format validation rules, and data business rule validation rules.

[0019] Optionally, the step of parsing the data to be cleaned to obtain the data to be cleaned and parsed includes:

[0020] The data to be cleaned is parsed using JavaScript technology to obtain the parsed data to be cleaned.

[0021] The present invention also provides a data cleaning and processing apparatus, comprising:

[0022] The data to be cleaned acquisition module is used to acquire data to be cleaned in different data formats; wherein, the data to be cleaned is credit data;

[0023] The data parsing module is used to parse the data to be cleaned to obtain cleaned and parsed data; wherein, the data parsing is at least one of the following: data type parsing, data content parsing, and data delimiter parsing according to the data format;

[0024] The cleaning and processing module is used to clean and process the data to be cleaned and parsed using cleaning rules to obtain cleaned data; wherein, the cleaning rules include at least one of duplicate data cleaning, missing data cleaning, invalid data cleaning, and erroneous data cleaning.

[0025] The present invention also provides a data cleaning and processing apparatus, comprising:

[0026] Memory, used to store computer programs;

[0027] A processor is used to implement the steps of the above-described data cleaning and processing method when executing the computer program.

[0028] The present invention also provides a computer-readable storage medium storing a program that, when executed by a processor, implements the steps of the above-described data cleaning and processing method.

[0029] As can be seen, this invention acquires data to be cleaned in different data formats; wherein the data to be cleaned is credit data. The data to be cleaned is parsed to obtain parsed data to be cleaned; wherein the data parsing includes at least one of the following: data type parsing, data content parsing, and data delimiter parsing based on the data format. The parsed data to be cleaned is then processed using cleaning rules to obtain cleaned data; wherein the cleaning rules include at least one of the following: deduplication cleaning, missing data cleaning, invalid data cleaning, and erroneous data cleaning. Compared to current methods that cannot clean data in different formats, this invention can clean and process credit data in different formats to obtain the cleaned data required by various enterprises.

[0030] In addition, the present invention also provides a data cleaning and processing apparatus, device, and computer-readable storage medium, which also have the above-mentioned beneficial effects. Attached Figure Description

[0031] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0032] Figure 1 A flowchart of a data cleaning and processing method provided in an embodiment of the present invention;

[0033] Figure 2 A flowchart illustrating a data cleaning and processing method provided in an embodiment of the present invention;

[0034] Figure 3 An example diagram illustrating a component-based approach for acquiring data to be cleaned in different data formats, as provided in an embodiment of the present invention;

[0035] Figure 4 This is a flowchart illustrating a list verification process provided in an embodiment of the present invention;

[0036] Figure 5An example diagram illustrating the generation of data messages for various tax bureaus, provided as an embodiment of the present invention;

[0037] Figure 6 A schematic diagram of a feedback anomaly scheduling provided in an embodiment of the present invention;

[0038] Figure 7 This is a schematic diagram of an interface framework corresponding to a data cleaning and processing method provided in an embodiment of the present invention;

[0039] Figure 8 This is a schematic diagram of the structure of a data cleaning and processing apparatus provided in an embodiment of the present invention;

[0040] Figure 9 This is a schematic diagram of the structure of a data cleaning and processing device provided in an embodiment of the present invention. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0042] Post-loan feedback refers to the information and feedback provided by borrowers to lending institutions after a consumer has applied for and received a loan, during the repayment period. The purpose of post-loan feedback is to allow lending institutions to better understand borrowers' repayment behavior and creditworthiness, enabling them to better manage and control risk and provide better services. The following is the background of post-loan feedback: Risk Management: Post-loan feedback provides lending institutions with an opportunity to understand borrowers' repayment behavior and creditworthiness, promptly identify loan default risks, and thus take appropriate measures to control risk and avoid loan defaults. Customer Retention: Timely post-loan feedback can increase trust between lending institutions and borrowers, help lending institutions build better customer relationships, encourage borrowers to choose the same lending institution in future loan applications, and recommend the institution to others. Loan Services: Post-loan feedback can help lending institutions improve and optimize loan services to better meet borrowers' needs and expectations, providing a better service experience. Currently, when cleaning credit data (post-loan feedback data), cleaning is often done only based on key fields, resulting in low efficiency in the credit data cleaning process.

[0043] Please refer to Figure 1 , Figure 1 A flowchart illustrating a data cleaning and processing method provided in an embodiment of the present invention. The method may include:

[0044] S100: Obtain data to be cleaned in different data formats; among them, the data to be cleaned is credit data.

[0045] This embodiment supports obtaining corporate credit data provided by banks through multiple methods, with the data being acquired on the bank's side according to its collection methods. This embodiment does not limit the specific method for obtaining data to be cleaned in different data formats. For example, data to be cleaned can be collected via SFTP (SSH File Transfer Protocol); or via FTP (File Transfer Protocol); or via OSS (Object Storage Service); or via API (Application Program Interface). It should be noted that obtaining data to be cleaned in different data formats can include using secure file transfer protocols, file transfer protocols, cloud storage services, and application programming interfaces to obtain the data in different formats. Because this embodiment can obtain data to be cleaned through various methods, it improves the accuracy of data acquisition.

[0046] It should be noted that the aforementioned acquisition of data to be cleaned in different data formats can include: acquiring credit information, loan information, detailed information, summary information, loan balance information, disbursement information, repayment information, and loan overdue information in different data formats. In this embodiment, the data to be cleaned includes credit information, loan information, detailed information, summary information, loan balance information, disbursement information, repayment information, and loan overdue information. In this embodiment, credit information refers to funds directly provided by commercial banks to non-financial institution clients, or guarantee information regarding potential compensation or payment liabilities in relevant economic activities. This includes on-balance-sheet transactions such as loans, trade finance, bill financing, financial leasing, overdrafts, and various advances, as well as off-balance-sheet transactions such as bill acceptance, issuance of letters of credit, guarantees, standby letters of credit, letter of credit confirmation, bond issuance guarantees, loan guarantees, recourse asset sales, and unused irrevocable loan commitments. In this embodiment, loan information includes data such as loan type, guarantee method, ratio, account status, repayment frequency, repayment month, loan disbursement date, loan maturity date, loan contract amount, and monthly repayment status records. The detailed information in this embodiment can refer to the amount due and actually repaid this month. The summary information in this embodiment can refer to the summary of outstanding loan information. The loan balance information in this embodiment can refer to the total amount of loan the borrower has not yet repaid to the lender as of a certain date. The disbursement information in this embodiment can refer to the loan product type. The repayment information in this embodiment can refer to the repayment date and amount. The loan overdue information in this embodiment can refer to the total number of overdue loans and the duration of overdue loans. It is evident that this embodiment improves the comprehensiveness of data cleaning by acquiring data from different data interfaces.

[0047] S101, perform data parsing on the data to be cleaned to obtain data to be cleaned and parsed; wherein, data parsing is at least one of the following: data type parsing, data content parsing, and data delimiter parsing according to the data format.

[0048] This embodiment does not limit the specific method of parsing the data to be cleaned. For example, large file fragmentation technology can be used to parse the file, improving parsing efficiency and stability, and the parsed file can be stored in a database. Alternatively, a custom lightweight file can be used to parse the data to be cleaned; or JavaScript (JavaScript is a programming language for the Web) scripting technology can be used for standardized cleaning, field completion, and other operations. It should be noted that the above-mentioned data parsing of the data to be cleaned to obtain the data to be cleaned can include: using JavaScript scripting technology to parse the data to be cleaned to obtain the data to be cleaned. This embodiment can improve the efficiency of data parsing because it can use lightweight JavaScript scripting technology to parse the data to be cleaned.

[0049] S102, using cleaning rules to clean and process the data to be parsed, to obtain cleaned data; wherein, the cleaning rules include at least one of duplicate data cleaning, missing data cleaning, invalid data cleaning and erroneous data cleaning.

[0050] This embodiment does not limit the specific cleaning rules. For example, the cleaning rules can be duplicate data cleaning designed based on the authorized list; or the cleaning rules can be missing data cleaning rules designed based on a preset standard data format; or the cleaning rules can be invalid data cleaning and erroneous data cleaning rules designed based on a preset standard data format.

[0051] It should be noted that the above-mentioned cleaning process, which uses cleaning rules to process the data to be cleaned and obtain cleaned data, can include: verifying the data to be cleaned using pre-loan authorization lists, pre-loan user lists, and hash functions. In this embodiment, verifying the pre-loan authorization lists refers to deleting data not included in the authorization lists. Verifying the pre-loan user lists in this embodiment can refer to deleting data from blacklisted users. Hash function verification in this embodiment can refer to filtering out data whose hash function values ​​do not change, thus avoiding duplicate feedback (for the full dataset). The hash function can be MD5 (Message Digest Algorithm Version 5) or MD4 (Message Digest Algorithm Version 4). Because this embodiment can clean the data based on lists and hash function values, it can improve the comprehensiveness of data filtering.

[0052] It should be noted that after cleaning the data to be cleaned using cleaning rules to obtain cleaned data, the process may further include: processing the cleaned data using data mapping rules to obtain target cleaned data in different formats corresponding to the cleaned data; wherein, the data mapping rules are data mapping rules set according to the data formats of different tax bureaus; and routing the target cleaned data to each tax processor according to the enterprise's tax number. In this embodiment, when obtaining cleaned data, the system uses standardized data mapping rules to personalize the message mapping according to the structure of the feedback messages from each tax bureau. By parsing the mapping relationship, the system generates the feedback message content required by the tax bureau, calls the feedback interface to provide data feedback, and ensures that each tax bureau can quickly obtain the cleaned data.

[0053] It should be noted that the above-mentioned processing of cleaned data using data mapping rules includes: processing the cleaned data using feedback interface mapping rules, feedback message format mapping rules, feedback field mapping rules, field format validation rules, and data business rule validation rules. In this embodiment, the feedback interface mapping rule refers to the rule for mapping cleaned data to a feedback interface; the feedback message format mapping rule refers to the rule for mapping the format of cleaned data; the feedback field mapping rule refers to the rule for mapping cleaned data to fields in the formats of various tax bureaus; the field format validation rule refers to the standard field format validation rule corresponding to each tax bureau; and the data business rule validation rule refers to the mapping rule for structured standard business rule data.

[0054] The data cleaning and processing method provided in this invention embodiment may include: acquiring data to be cleaned in different data formats; wherein the data to be cleaned is credit data. Parsing the data to be cleaned yields parsed data to be cleaned; wherein the data parsing includes at least one of data type parsing, data content parsing, and data delimiter parsing based on the data format. Cleaning and processing the parsed data to be cleaned using cleaning rules yields cleaned data; wherein the cleaning rules include at least one of duplicate data cleaning, missing data cleaning, invalid data cleaning, and erroneous data cleaning. Therefore, compared to the current inability to clean data in different formats, this invention can clean and process credit data in different formats to obtain the cleaned data required by various enterprises. Furthermore, because secure file transfer protocols, file transfer protocols, cloud storage services, and application programming interfaces (APIs) can be used to obtain the data to be cleaned in different data formats, the credit data cleaned by this invention is more comprehensive, improving the efficiency of credit data processing. Moreover, because it can uniformly clean credit information, loan information, detailed information, summary information, loan balance information, disbursement information, repayment information, and loan overdue information, the comprehensiveness of data cleaning is improved. Furthermore, because the data to be cleaned undergoes pre-loan authorization list verification, pre-loan user list verification, and hash function verification, the resulting cleaned data is more concise and efficient. It is more comprehensive; moreover, by using data mapping rules to process the cleaned data and obtaining target cleaned data in different formats corresponding to the cleaned data, the cleaned data can be fed back to various tax bureaus in a timely manner; furthermore, by using feedback interface mapping rules, feedback message format mapping rules, feedback field mapping rules, field format validation rules, and data business rule validation rules to process the cleaned data, the data obtained by various tax bureaus can be used directly; and since JS script technology is used to parse the data to be cleaned, and JS script technology is a lightweight technology, the overall cleaning efficiency can be improved.

[0055] For a clearer understanding of this invention, please refer to the following details. Figure 2 , Figure 2 A flowchart illustrating a data cleaning and processing method provided in this embodiment of the invention may specifically include:

[0056] S200: Acquire data to be cleaned in different data formats.

[0057] This embodiment does not limit the data collection method corresponding to different data formats of data to be cleaned. Data collection can include various collection methods, including but not limited to: SFTP (SSH File Transfer Protocol), FTP (File Transfer Protocol), OSS (Object Storage Service), local shared files, API (Application Program Interface), database tables, etc. In this embodiment, each step of the data cleaning process is developed through componentization. Components can use the built-in functions of the product, or the implementation logic of the components can be customized according to the actual business scenario. Switching the implementation of components does not require adjustments to the entire process, reusing other standard process logic, increasing code reuse, reducing the development of repetitive code, improving the quality of process execution, solving the personalized development of different application scenarios, and flexibly implementing plugins according to the scenario. This embodiment can obtain data content files according to the specified data source target address. Each file creates a Job, and multiple files are collected and executed according to multiple jobs. The processing of each Job is independent and does not affect each other. Each Job reads the content of the specified file, customizes the parsing logic of the file content according to the business scenario, and stores each piece of feedback data after parsing. Before data storage, deduplication is performed based on existing cached data; existing data is not stored repeatedly. If a job fails to read data from a specified file, the entire job execution terminates, and the job's latest status and failure description are updated. Please refer to [reference needed]. Figure 3 , Figure 3 This is an example diagram illustrating a component-based approach for acquiring data to be cleaned in different data formats, as provided in an embodiment of the present invention.

[0058] In this embodiment, if the data acquisition process fails to acquire data in different formats to be cleaned, the feedback Job status is set to an abnormal state. By triggering a data acquisition anomaly scheduling event, the specified feedback job is re-initiated in the data acquisition process. 1) Backend triggering: A data anomaly scheduling interface is provided, which triggers a data acquisition anomaly scheduling event by specifying the feedback Job ID, and reschedules data acquisition. 2) Backend timed triggering: The backend sets timed triggering rules to trigger the check for abnormal feedback Jobs awaiting scheduling at a specified time, and calls the data acquisition anomaly scheduling event with the feedback Job ID to reschedule data acquisition.

[0059] S201, perform data parsing on the data to be cleaned in different data formats to obtain parsed cleaned data; wherein, data parsing is to parse according to the data format corresponding to the data to be cleaned.

[0060] This embodiment of data parsing can include various file formats and data delimiters, including but not limited to txt and dat file formats, and delimiters such as |, $, and |$|. This embodiment can utilize components to dynamically support parsing and cleaning logic for different data formats in different scenarios, reducing code intrusion and development, and achieving structured parsing and cleaning of feedback data. Through parsing and cleaning logic, the source data is logically parsed and cleaned, and standard feedback data (parsed and cleaned data) is returned.

[0061] S202 uses a lightweight programming language to perform initial cleaning processing on the parsed and cleaned data to obtain initial cleaned data; the initial cleaning processing involves cleaning the abnormal data in the parsed and cleaned data and converting the normal data into standard fields.

[0062] In this embodiment, the initial cleaning process uses AviatorScript (a high-performance, lightweight scripting language hosted on the JVM) to achieve standardized data cleaning, which is lightweight, efficient, and highly maintainable. This embodiment uses AviatorScript, a statically typed language, to write the data cleaning script. The execution engine of AviatorScript is called via Java (an object-oriented programming language), and the data is passed in for parsing and cleaning.

[0063] In this embodiment, if the feedback data fails during the cleaning process, a cleaning failure event is invoked, setting the status of the feedback data to an abnormal state. The scheduler triggers a data cleaning abnormal scheduling event by specifying the feedback data ID, and re-feeds the data. During the re-feedback process, it will enter the unprocessed stage based on the currently processed stage of the feedback data for feedback operation. 1) Backend triggering: Provides a data cleaning abnormal scheduling interface, which triggers a data cleaning abnormal scheduling event by specifying the feedback data ID, and re-schedules the data for feedback. 2) Backend timed triggering: The backend sets timed triggering rules to trigger the check of abnormal cleaning data to be scheduled at a specified time, and calls the data re-cleaning scheduling event with the feedback data ID to re-schedule the data cleaning.

[0064] S203, perform list verification on the initial cleaning and processing data, filter out duplicate and erroneous data, and obtain list verification data.

[0065] The authorization list verification in this embodiment includes user verification, pre-loan user verification, MD5 verification, and business rule verification. This embodiment can verify feedback data (initial cleansing and processing data). It supports API and local database query methods to verify whether the feedback data message is within the correct feedback list. Data filtering removes unwanted feedback data, which can be used for communication with banks during the feedback process, reducing issues related to feedback list failures. 1) API method: Customizes calls to remote interfaces and returns verification results based on the specified taxpayer identification number. 2) Database query method: Customizes queries for the specified taxpayer identification number using specified databases and tables, returning verification results. 3) Verification failure: If verification fails, a feedback failure event is executed. Please refer to... Figure 4 , Figure 4 This is a flowchart illustrating a list verification process provided in an embodiment of the present invention, in which the initial cleaning and processing data is verified using a list verification method.

[0066] S204 generates target data packets in various formats based on message mapping configuration rules.

[0067] In this embodiment, the message mapping configuration rules include feedback interface configuration, feedback message format configuration, standard field and feedback field mapping, field format validation configuration, and data service rule validation configuration. For ease of understanding, this invention provides a feedback example of feedback data routing mapping: Landscape Architecture Design Co., Ltd. | 91120102770635846H | B | 01 | 120101196709113528 | Lv Yan | 9999602570080100043480 | 20381019 | 3000000.00 | 20381019 | 20401018 | 90000.00 | 60000.00 | Normal | 91120102770635846H | Success | 20381019 | 20401018 | Loan Withdrawal | 1010010 | 24 Months | 16 | Credit Loan | 90000.00 | 3 | 20381019 Figure 5 , Figure 5 This embodiment of the invention provides an example diagram for generating data messages for various tax bureaus. After the component obtains the feedback data, it performs custom data retrieval to generate a map (a container that stores elements by key). The map object is then passed to the implementation of the routing strategy IPlbRouterStrategy interface. Based on the data passed to the map and the logic of the strategy, a mapping key is returned. The mapping key is then used to find the specified tax bureau instance through the instance set of each tax bureau's feedback execution component to execute the data feedback. If the mapping key cannot find a specific tax bureau feedback execution component instance, a feedback failure event is invoked. This embodiment can use agreed-upon XML (Extensible Markup Language) configuration rules as message mapping configuration rules, customize the message mapping according to the structure of each tax bureau's feedback message, parse the mapping relationship to generate the feedback message content required by the tax bureau, and call the feedback interface to perform data feedback. The following XML configuration specification applies: Mapping-dataForm (Mapping-DataForm): The format of the generated message data; Map-from: Fields of the standard message; Map-target: Fields of the message to be generated; The Mapping configuration converts the standard message into a Map format key-value (Non-relational database) data structure. Based on the data format type specified by dataForm (DataForm), different data format generation strategies are used to convert the Map format key-value data structure into a message of the specified data type. A) JSON data format; B) XML data format.

[0068] This embodiment can also configure the quality logic of each feedback field according to the tax bureau's requirements for each field through a standardized XML configuration. The quality logic requirements are configured using the `validator` tag, and the quality logic is configured using static JavaScript syntax. It supports basic data logic judgment and formula calculation, achieving verification of the quality requirements for each field. If the validator validation for any field fails, the entire tax bureau feedback message generation fails, resulting in data feedback failure. This embodiment can also perform tax bureau message rule judgment through a standardized XML configuration. According to the tax bureau's requirements for feedback message data, messages that do not meet the rules will be called to the tax bureau's interface, resulting in direct feedback failure. The XML configuration configures specific rules, and each rule uses static JavaScript syntax to perform rule judgment on the tax bureau's feedback message, supporting basic data logic judgment and formula calculation. If the message does not conform to the rules, the feedback fails. Each rule is independent; if one rule is not met, the feedback fails.

[0069] S205 routes the data to the appropriate tax authority processor based on the company's tax identification number.

[0070] In this embodiment, a feedback execution component can be set up for each tax bureau and developed independently according to the requirements of the tax bureau's feedback interface version. The feedback execution component of the same tax bureau can support multiple versions of feedback from the same tax bureau. The tax bureau requiring feedback, the corresponding tax bureau's strategy logic result in the feedback message mapping, and the required message field structure for the tax bureau are determined through dynamic configuration. The component can dynamically load instances of the relevant tax bureau feedback execution components based on the configured tax bureaus. Each tax bureau feedback execution component is isolated and independently developed, supporting hot deployment and dynamic operation of tax bureau feedback switches. In this embodiment, routing can be performed using the tax bureau feedback execution components. For ease of understanding, please refer to Table 1, which is an example table of feedback routing provided by this embodiment of the invention.

[0071] Table 1. An example of a feedback routing table.

[0072] 1 Guangdong Tax Bureau Check 440000 open 2 Tianjin Tax Bureau Check 120000 open 3 Jiangsu Tax Bureau Check 320000 open

[0073] This embodiment can locate the execution class for tax bureau feedback in the database through the tax bureau configuration, such as Guangdong Tax Bureau: com.vz.plb.tax.gd.PlbGdTaxProcessor. After initializing this class, it is placed into the collection of tax bureau feedback execution components. If the path cannot be found, an error is displayed. The system maps specific instances of tax bureau feedback execution components according to the routing strategy corresponding to the message, and executes the instance's process() method to provide feedback. process is the routing process. Local or remote tax bureau feedback execution component instances are supported. A. Local method: Obtain the class path of the specific tax bureau feedback execution component through the database configuration, initialize the class instance, and perform feedback execution calls. B. RPC (Remote Procedure Call Protocol) method: Configure the remote interface implementation through Hessian (remote communication protocol), and load each tax bureau's remote feedback execution component instance through the Hessian configuration file. This method does not currently support hot deployment. This embodiment can handle new tax bureau feedback via: A) Backend Management Method: New tax bureau feedback configuration is added via the backend management system. After the configuration is submitted, the "Refresh Tax Bureau Feedback Execution Component Set" event is invoked, initializing the new tax bureau feedback configuration instance and adding it to the component set. B) API Method: New tax bureau feedback configuration is added via API calls, simultaneously invoking the "Refresh Tax Bureau Feedback Execution Component Set" event. Tax Bureau Feedback Switch Method: A) Backend Management Method: A switch configuration for specified tax bureau feedback is triggered via the backend management system. Based on the switch's state, the "Switch Tax Bureau Feedback Switch" event is invoked, toggling the addition or destruction of the tax bureau feedback execution component set instance. B) API Method: Based on API calls, the "Switch Tax Bureau Feedback Switch" event is invoked based on the switch's state.

[0074] S206 uses the tax authority processor to encapsulate and encrypt the request message corresponding to the target data message to obtain the feedback data.

[0075] S207, call the tax bureau's interface to provide feedback data.

[0076] Tax bureau feedback can include: request message encapsulation, data encryption, setting request protocols, response message parsing, and recording feedback results. Feedback management specifically includes: task management, job management, feedback data management, template management, statistics management, and system management. Task management records the execution status of tasks executed daily; job management records the execution status of each job, the number of job records, the number of successes, the execution time, the time consumed, etc.; feedback data management records the feedback status of each piece of data; template management dynamically manages and makes real-time changes to feedback templates for all regions; statistics management compiles the execution status of all tasks daily; and system management manages the cleaning scripts.

[0077] In this embodiment, the feedback interface calls for each tax bureau corresponding to the feedback data are customized on the feedback execution component of each tax bureau, supporting the development and implementation of different versions of tax bureau interfaces. The "Dynamically Configure Feedback Tax Bureau" function specifies the version mapping to the specific execution instance. Each tax bureau's feedback interface call is implemented in separate modules, with completely isolated code. Updates to one tax bureau's interface do not affect the use of other tax bureaus. Each tax bureau's feedback is independent and isolated, with no overlap or influence in the feedback data. For details on data failures at different stages of the feedback process and execution data feedback failure events, please refer to [link / reference needed]. Figure 6 , Figure 6 This is a schematic diagram of feedback anomaly scheduling provided by an embodiment of the present invention. When any process feedback fails, the event updates the status of the feedback data to an abnormal state, triggering a data feedback anomaly scheduling event for re-feedback. 1) Backend triggering: A data feedback anomaly scheduling interface is provided, which triggers a data feedback anomaly scheduling event by specifying the feedback data ID, rescheduling the data for feedback. 2) Backend timed triggering: The backend sets timed triggering rules to trigger the checking of abnormal feedback data awaiting scheduling at a specified time, calling the data re-feedback scheduling event with the feedback data ID, and rescheduling the data feedback.

[0078] For easier understanding, please refer to Figure 7 , Figure 7 This is a schematic diagram of an interface framework corresponding to a data cleaning and processing method provided in an embodiment of the present invention. Figure 7 The process first requires data collection, including system logs, performance data, and business logs. A source data acquisition component acquires the collected data, which is then cleaned using a data cleaning component. A list verification component is then called to verify the list. During data acquisition, cleaning, and verification, feedback data can be monitored, and the acquired source data can be processed through large file fragmentation, feedback quality interception, rate limiting, anti-duplicate verification, rescheduling, template mapping and data parsing, feedback statistics, and post-loan data acquisition. In this embodiment, the list cleaning process utilizes IS scripts to parse and clean post-loan feedback lists for Enterprise Loan Cloud, Quick Loan, and Tax Easy Loan, among others. After data parsing and cleaning, this embodiment routes the data according to the routing configuration strategy, transforming the format of the cleaned data according to the template mapping strategy for data feedback. The backend management in this embodiment can include job management, data management, configuration management, and system management.

[0079] The following describes a data cleaning and processing apparatus provided by an embodiment of the present invention. The data cleaning and processing apparatus described below can be referred to in correspondence with the data cleaning and processing method described above.

[0080] Please refer to the details. Figure 8 , Figure 8 A schematic diagram of a data cleaning and processing apparatus provided in an embodiment of the present invention may include:

[0081] The data to be cleaned acquisition module 100 is used to acquire data to be cleaned in different data formats; wherein, the data to be cleaned is credit data;

[0082] The data parsing module 200 is used to parse the data to be cleaned to obtain cleaned parsed data; wherein, the data parsing is at least one of the following: data type parsing, data content parsing, and data delimiter parsing according to the data format;

[0083] The cleaning and processing module 300 is used to clean and process the data to be cleaned and parsed using cleaning rules to obtain cleaned data; wherein, the cleaning rules include at least one of duplicate data cleaning, missing data cleaning, invalid data cleaning and erroneous data cleaning.

[0084] Furthermore, based on the above embodiments, the data acquisition module 100 to be cleaned may include:

[0085] The data acquisition unit utilizes different methods to acquire the data to be cleaned, which is used to acquire the data to be cleaned in different data formats using secure file transfer protocols, file transfer protocols, cloud storage services, and application programming interfaces.

[0086] Furthermore, based on any of the above embodiments, the data acquisition module 100 to be cleaned may include:

[0087] The data acquisition unit for different formats is used to acquire credit information, loan information, detailed information, summary information, loan balance information, disbursement information, repayment information and loan overdue information in different data formats.

[0088] Furthermore, based on any of the above embodiments, the cleaning and processing module 300 may include:

[0089] The list cleaning unit is used to perform pre-loan authorization list verification, pre-loan user list verification, and hash function verification on the data to be cleaned to obtain the cleaned data.

[0090] Furthermore, based on any of the above embodiments, the data cleaning and processing apparatus may further include:

[0091] The format conversion module is used to process the cleaned data using data mapping rules to obtain target cleaned data in different formats corresponding to the cleaned data; wherein, the data mapping rules are data mapping rules set according to different tax bureau data formats;

[0092] The routing module is used to route the target cleaned data to each tax processor based on the enterprise tax number.

[0093] Furthermore, based on the above embodiments, the above format conversion module may include:

[0094] A multi-rule format conversion module is used to process the cleaned data using feedback interface mapping rules, feedback message format mapping rules, feedback field mapping rules, field format verification rules, and data business rule verification rules.

[0095] Furthermore, based on any of the above embodiments, the data parsing module 200 may include:

[0096] The data parsing unit is used to parse the data to be cleaned using JavaScript technology to obtain the data to be cleaned and parsed.

[0097] It should be noted that the order of the modules and units in the above-mentioned data cleaning and processing device can be changed without affecting the logic.

[0098] The data cleaning and processing apparatus provided in this embodiment of the invention includes: a data acquisition module 100 for acquiring data to be cleaned in different data formats; wherein the data to be cleaned is credit data; a data parsing module 200 for parsing the data to be cleaned to obtain parsed data to be cleaned; wherein the data parsing includes at least one of data type parsing, data content parsing, and data delimiter parsing based on the data format; and a cleaning and processing module 300 for cleaning and processing the parsed data to be cleaned using cleaning rules to obtain cleaned data; wherein the cleaning rules include at least one of duplicate data cleaning, missing data cleaning, invalid data cleaning, and erroneous data cleaning. Therefore, compared to the current inability to clean data in different formats, this invention can clean and process credit data in different formats to obtain the cleaned data required by various enterprises. Furthermore, because secure file transfer protocols, file transfer protocols, cloud storage services, and application programming interfaces (APIs) can be used to obtain the data to be cleaned in different data formats, the credit data cleaned by this invention is more comprehensive, improving the efficiency of credit data processing. Moreover, because it can uniformly clean credit information, loan information, detailed information, summary information, loan balance information, disbursement information, repayment information, and loan overdue information, the comprehensiveness of data cleaning is improved. Furthermore, because the data to be cleaned undergoes pre-loan authorization list verification, pre-loan user list verification, and hash function verification, the resulting cleaned data is more concise and efficient. It is more comprehensive; moreover, by using data mapping rules to process the cleaned data and obtaining target cleaned data in different formats corresponding to the cleaned data, the cleaned data can be fed back to various tax bureaus in a timely manner; furthermore, by using feedback interface mapping rules, feedback message format mapping rules, feedback field mapping rules, field format validation rules, and data business rule validation rules to process the cleaned data, the data obtained by various tax bureaus can be used directly; and since JS script technology is used to parse the data to be cleaned, and JS script technology is a lightweight technology, the overall cleaning efficiency can be improved.

[0099] The following is a description of a data cleaning and processing device provided by an embodiment of the present invention. The data cleaning and processing device described below can be referred to in correspondence with the data cleaning and processing method described above.

[0100] Please refer to Figure 9 , Figure 9 A schematic diagram of a data cleaning and processing device provided in an embodiment of the present invention may include:

[0101] Memory 10 is used to store computer programs;

[0102] The processor 20 is used to execute computer programs to implement the data cleaning and processing method described above.

[0103] The memory 10, processor 20, and communication interface 30 all communicate with each other through the communication bus 40.

[0104] In this embodiment of the invention, the memory 10 is used to store one or more programs. The programs may include program code, which includes computer operation instructions. In this embodiment of the invention, the memory 10 may store programs for implementing the following functions:

[0105] Obtain data to be cleaned in different data formats; among them, the data to be cleaned is credit data.

[0106] The data to be cleaned is parsed to obtain the data to be cleaned and parsed; wherein, the data parsing is at least one of the following: data type parsing, data content parsing, and data delimiter parsing according to the data format.

[0107] The data to be cleaned is processed using cleaning rules to obtain cleaned data; the cleaning rules include at least one of the following: duplicate data cleaning, missing data cleaning, invalid data cleaning, and erroneous data cleaning.

[0108] In one possible implementation, the memory 10 may include a program storage area and a data storage area, wherein the program storage area may store the operating system and applications required for at least one function; and the data storage area may store data created during use.

[0109] Furthermore, memory 10 may include read-only memory and random access memory, providing instructions and data to the processor. A portion of the memory may also include NVRAM. The memory stores operating systems and operating instructions, executable modules, or data structures, or subsets thereof, or extended sets thereof, wherein the operating instructions may include various operating instructions for implementing various operations. The operating system may include various system programs for implementing various basic tasks and handling hardware-based tasks.

[0110] Processor 20 can be a central processing unit (CPU), an application-specific integrated circuit, a digital signal processor, a field-programmable gate array, or other programmable logic device. Processor 20 can be a microprocessor or any conventional processor. Processor 20 can call programs stored in memory 10.

[0111] The communication interface 30 can be an interface for the communication module, used to connect with other devices or systems.

[0112] Of course, it should be noted that, Figure 9 The structure shown does not constitute a limitation on the data cleaning and processing equipment in the embodiments of the present invention. In practical applications, a data cleaning and processing equipment may include more than Figure 9 More or fewer components as shown, or combinations of certain components.

[0113] The following describes the computer-readable storage medium provided in the embodiments of the present invention. The computer-readable storage medium described below can be referred to in correspondence with the data cleaning and processing method described above.

[0114] The present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the data cleaning and processing method described above.

[0115] The computer-readable storage medium may include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0116] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0117] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0118] Finally, it should be noted that in this document, relationships such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0119] The above provides a detailed description of the data cleaning and processing method, apparatus, equipment, and computer-readable storage medium provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A data cleaning and processing method, characterized in that, include: The component is used to acquire data to be cleaned in different data formats; wherein, the data to be cleaned is credit data. The data to be cleaned is parsed to obtain the parsed data to be cleaned; wherein, the data parsing is at least one of the following: data type parsing, data content parsing, and data delimiter parsing according to the data format; the parsing and cleaning logic for different data formats in different scenarios is dynamically performed using components; the parsing logic is customized according to the business scenario; The data to be cleaned and parsed is processed using cleaning rules to obtain cleaned data; wherein, the cleaning rules include at least one of duplicate data cleaning, missing data cleaning, invalid data cleaning, and erroneous data cleaning; the cleaning rules include duplicate data cleaning designed according to the authorized list, missing data cleaning rules designed according to the preset standard data format, and invalid data cleaning and erroneous data cleaning rules designed according to the preset standard data format. The cleaned data is processed using data mapping rules to obtain target cleaned data in different formats corresponding to the cleaned data; wherein, the data mapping rules are data mapping rules set according to different tax bureau data formats; The cleaned target data is routed to various tax processors based on the enterprise's tax number; The step of processing the cleaned data using data mapping rules includes: The cleaned data is processed using feedback interface mapping rules, feedback message format mapping rules, feedback field mapping rules, field format validation rules, and data business rule validation rules.

2. The data cleaning and processing method according to claim 1, characterized in that, The process of acquiring data to be cleaned in different data formats includes: The data to be cleaned is obtained using secure file transfer protocols, file transfer protocols, cloud storage services, and application programming interfaces (APIs) in different data formats.

3. The data cleaning and processing method according to claim 1, characterized in that, The process of acquiring data to be cleaned in different data formats includes: It can obtain credit information, loan information, detailed information, summary information, loan balance information, disbursement information, repayment information, and loan overdue information in different data formats.

4. The data cleaning and processing method according to claim 1, characterized in that, The process of cleaning the data to be cleaned using cleaning rules to obtain cleaned data includes: The data to be cleaned is subjected to pre-loan authorization list verification, pre-loan user list verification, and hash function verification to obtain the cleaned data.

5. The data cleaning and processing method according to claim 1, characterized in that, The step of parsing the data to be cleaned to obtain the parsed data to be cleaned includes: The data to be cleaned is parsed using JavaScript technology to obtain the parsed data to be cleaned.

6. A data cleaning and processing apparatus, characterized in that, The data cleaning and processing method according to any one of claims 1 to 5 includes: The data to be cleaned acquisition module is used to acquire data to be cleaned in different data formats; wherein, the data to be cleaned is credit data; The data parsing module is used to parse the data to be cleaned to obtain cleaned and parsed data; wherein, the data parsing is at least one of the following: data type parsing, data content parsing, and data delimiter parsing according to the data format; The cleaning and processing module is used to clean and process the data to be cleaned and parsed using cleaning rules to obtain cleaned data; wherein, the cleaning rules include at least one of duplicate data cleaning, missing data cleaning, invalid data cleaning, and erroneous data cleaning.

7. A data cleaning and processing device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the data cleaning and processing method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The readable storage medium stores a program that, when executed by a processor, implements the steps of the data cleaning and processing method as described in any one of claims 1 to 5.

Images