Data processing method and device, computer equipment, readable storage medium and program product

Abstract

The invention relates to the field of big data, and particularly discloses a data processing method and device, computer equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: receiving a data processing request after a front end passes auditing; the data processing request carries a to-be-processed data set; performing rule check on each data type in the to-be-processed data set, and after the rule check is passed, generating an intermediate data set according to each data type; performing data verification on the intermediate data set to obtain a target data set; when the timestamp of the target data set is within a preset time period range, acquiring a preset and stored reference data set corresponding to the target data set, and calculating the difference between the target data set and the reference data set; and when the difference meets a preset condition, generating a data processing result according to the target data set. By adopting the method, the processing efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] In modern computer systems, complex data management and processing processes are necessary because the system needs to efficiently process large amounts of different types of data and ensure data accuracy and consistency. These processes help to cope with the diversity and changing needs of data, ensure that data is transmitted and processed correctly between modules, and support automation and flexible adjustments to improve system efficiency and reliability.

[0003] The data management and processing systems in the prior art usually adopt a modular design, distributing functions such as data inspection, rule management, and data set generation in multiple independent modules. These modules interact through preset interfaces to form a complete processing flow. Usually, the data will undergo a preliminary inspection in the system, and then generate data set activation information according to specific rules, and then proceed to subsequent processing. However, this method is highly dependent on manually setting rules and configuring parameters to manage the processing method of data sets, and requires manual intervention to deal with abnormal situations, but the processing efficiency is low. Summary of the invention

[0004] Based on this, it is necessary to provide a data processing method, device, computer equipment, computer-readable storage medium and computer program product that can improve processing efficiency in response to the above technical problems.

[0005] In a first aspect, the present application provides a data processing method, the method comprising:

[0006] Receive a data processing request after the front-end review is passed; the data processing request carries a set of data to be processed;

[0007] Performing rule checking on each data type in the to-be-processed data set, and generating an intermediate data set according to each data type after passing the rule checking;

[0008] Performing data verification on the intermediate data set to obtain a target data set; performing data verification on the intermediate data set to obtain a target data set includes: when the data to be processed is contract data, obtaining a preset contract template; extracting a target field and a resource field from the intermediate data corresponding to each data type according to the preset contract template; verifying the resource field based on the target field to obtain a verification result; when the verification result is passed and the status corresponding to the resource field is acquired, generating the target data set according to the target field;

[0009] When the timestamp of the target data set is within a preset time period, obtaining a preset reference data set corresponding to the target data set, and calculating the difference between the target data set and the reference data set;

[0010] When the difference meets a preset condition, a data processing result is generated according to the target data set.

[0011] In one embodiment, performing rule checking on each data type in the to-be-processed data set and generating an intermediate data set according to each data type after passing the rule checking includes:

[0012] Acquire preset rules corresponding to each of the data types, and perform rule checking on each of the data types according to the preset rules;

[0013] When each of the data types satisfies the preset rule corresponding to the rule check, marking the to-be-processed data corresponding to the data type as target data;

[0014] The target data is integrated according to a preset data structure to obtain the intermediate data set.

[0015] In one embodiment, the obtaining of preset rules corresponding to each of the data types and performing rule checking on each of the data types according to the preset rules includes:

[0016] Read the data tags corresponding to the data types;

[0017] The preset rules corresponding to each of the data types are obtained according to the data tags, and rule checks are performed on each of the data types based on the preset rules.

[0018] In one embodiment, performing data verification on the intermediate data set to obtain the target data set includes:

[0019] When the data to be processed is contract data, obtaining a preset contract template;

[0020] Extracting target fields and resource fields from the intermediate data corresponding to each of the data types according to the preset contract template;

[0021] Verify the resource field based on the target field to obtain a verification result;

[0022] When the verification result is passed and the status corresponding to the resource field is acquired, the target data set is generated according to the target field.

[0023] In one embodiment, performing data verification on the intermediate data set to obtain the target data set includes:

[0024] When the data to be processed is transaction data, data normalization is performed on the intermediate data corresponding to each data type;

[0025] Performing integrity verification on the intermediate data corresponding to the data type after data normalization to obtain the intermediate data corresponding to the target data type;

[0026] Aggregate the intermediate data corresponding to the target data type to obtain the target data set.

[0027] In one embodiment, the preset time period range includes a first sub-preset time period range, a second sub-preset time period range and a third sub-preset time period range; the first sub-preset time period range is smaller than the second sub-preset time period range, and the second sub-preset time period range is smaller than the third sub-preset time period range;

[0028] When the timestamp of the target data set is within a preset time period, obtaining a preset reference data set corresponding to the target data set, and calculating the difference between the target data set and the reference data set, comprises:

[0029] When the timestamp is within the first sub-preset time period, acquiring data message parameters and / or configuration parameters corresponding to the target data set;

[0030] When the timestamp is within the second preset sub-time period, obtaining the reference data set according to the message parameter and / or the configuration parameter;

[0031] When the timestamp is within the third sub-preset time period, the difference between the target data set and the reference data set is calculated.

[0032] In a second aspect, the present application provides a data processing device, the device comprising:

[0033] A receiving module, used to receive a data processing request after the front-end review has been passed; the data processing request carries a set of data to be processed;

[0034] A checking module, used for performing a rule check on each data type in the data set to be processed, and generating an intermediate data set according to each data type after passing the rule check;

[0035] A verification module, used to perform data verification on the intermediate data set to obtain a target data set;

[0036] a difference calculation module, configured to obtain a preset reference data set corresponding to the target data set when the timestamp of the target data set is within a preset time period, and calculate the difference between the target data set and the reference data set;

[0037] The result generating module is used to generate a data processing result according to the target data set when the difference meets a preset condition.

[0038] In one embodiment, the inspection module includes:

[0039] A rule checking unit, used to obtain preset rules corresponding to each of the data types, and perform rule checking on each of the data types according to the preset rules;

[0040] a marking unit, configured to mark the to-be-processed data corresponding to the data type as target data when the data type satisfies the preset rule corresponding to the rule check;

[0041] The first set generating unit is used to integrate the target data according to a preset data structure to generate the intermediate data set.

[0042] In one embodiment, the rule checking unit includes:

[0043] A tag reading subunit, used to read data tags corresponding to each of the data types;

[0044] The rule acquisition and checking subunit is used to acquire the preset rules corresponding to each of the data types according to the data tags, and perform rule checking on each of the data types based on the preset rules.

[0045] In one embodiment, the verification module includes:

[0046] A template acquisition unit, used for acquiring a preset contract template when the data to be processed is contract data;

[0047] A field acquisition unit, used for extracting target fields and resource fields from the intermediate data corresponding to each of the data types according to the preset contract template;

[0048] A verification result unit, used to verify the resource field based on the target field to obtain a verification result;

[0049] The second set generating unit is configured to generate the target data set according to the target field when the verification result is passed and the status corresponding to the resource field is acquired.

[0050] In one embodiment, the verification module includes:

[0051] A normalization unit, used for normalizing the intermediate data corresponding to each data type when the data to be processed is transaction data;

[0052] An integrity verification unit, used to perform integrity verification on the corresponding intermediate data row of the data type after data normalization, and obtain intermediate data corresponding to the target data type;

[0053] The aggregation unit is used to aggregate the intermediate data corresponding to the target data type to obtain the target data set.

[0054] In one embodiment, the preset time period range includes a first sub-preset time period range, a second sub-preset time period range, and a third sub-preset time period range; the first sub-preset time period range is smaller than the second sub-preset time period range, and the second sub-preset time period range is smaller than the third sub-preset time period range, and the result generation module includes:

[0055] The first processing unit is configured to obtain data message parameters and / or configuration parameters corresponding to the target data set when the timestamp is within the first sub-preset time period.

[0056] A second processing unit, configured to obtain the reference data set according to the message parameter and / or the configuration parameter when the timestamp is within the second preset sub-time period;

[0057] The third processing unit is configured to calculate the difference between the target data set and the reference data set when the timestamp is within the third sub-preset time period.

[0058] In a third aspect, the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

[0059] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.

[0060] In a fifth aspect, the present application provides a computer program product, including a computer program, which implements the steps of the above method when executed by a processor.

[0061] The above-mentioned data processing method, device, computer equipment, computer-readable storage medium and computer program product improve the efficiency of data processing through automated steps: first, rule checking automatically filters out data that does not meet the requirements, reducing the workload and error rate of manual screening; second, data verification further ensures the accuracy and consistency of the data, avoiding the tedious process of manual verification one by one; in the difference calculation step, by automatically comparing with reference data, data changes are quickly identified, reducing the time for manual checking of differences; finally, the system automatically determines whether to generate data processing results through preset conditions, can quickly respond to abnormal situations, reduces the need for manual configuration and manual intervention, significantly improves processing efficiency, and optimizes the data management process. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0063] FIG1 is an application environment diagram of a data processing method in one embodiment;

[0064] FIG2 is a schematic flow chart of a data processing method in one embodiment;

[0065] FIG3 is a flow chart of a data processing method according to another embodiment;

[0066] FIG4 is a block diagram of a data processing device in one embodiment;

[0067] FIG. 5 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0068] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application. It should be noted that in the embodiments of the present application, certain software, groups, models and other existing solutions in the industry may be mentioned, which should be considered as exemplary, and their purpose is only to illustrate the feasibility of the implementation of the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0069] The acquisition, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0070] In one embodiment, as shown in FIG. 1, in combination with FIG. 1, FIG. 1 is an application environment diagram of a data processing method in an embodiment. Among them, the front end 102 reviews the data to be processed, and generates a data processing request after the data to be processed is reviewed and passed, and sends the data processing request to the backend system 104, and the backend system 104 receives the data processing request after the front end review passes; the data processing request carries the data set to be processed; the data types in the data set to be processed are checked by rules, and after passing the rule check, an intermediate data set is generated according to each data type; the intermediate data set is verified to obtain the target data set; when the timestamp of the target data set is within the preset time period, the preset stored reference data set corresponding to the target data set is obtained, and the difference between the target data set and the reference data set is calculated; when the difference meets the preset condition, the data processing result is generated according to the target data set. Among them, the front end 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices, and the Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, projection devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The head mounted device may be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The backend system 104 may be an independent physical server, a server cluster or a distributed backend system consisting of multiple physical servers, or a cloud server providing cloud computing services.

[0071] In an exemplary embodiment, as shown in FIG2 , a data processing method is provided, which is described by taking the method applied to the background system 104 in FIG1 as an example, and includes the following steps 202 to 206. Among them:

[0072] Step S202, receiving a data processing request after the front-end review is passed; the data processing request carries a set of data to be processed.

[0073] Among them, a data processing request refers to a data processing task that has been reviewed and approved and submitted by a user or a backend system on the front-end interface. The request includes the data to be processed, that is, the set of data to be processed.

[0074] The front-end system sends the approved request to the back-end data processing system, and the back-end system starts processing the data after receiving the request.

[0075] For example, if the data to be processed is contract data, the front-end system confirms that the contract data contains all necessary fields, such as contract number, information of both parties to the contract, signing date, contract amount, etc. If the key data in the contract, such as amount, date, and contract period, are correct and meet logical requirements, the front-end system will review and approve the data to be processed.

[0076] For example, if the data to be processed is transaction data, the front-end system will perform compliance checks on the transaction amount and time according to the preset rules to ensure that the transaction amount is within a reasonable range and the time does not exceed the time period specified by the business rules. If an abnormal amount or time is found, the front-end system will mark it as an abnormal transaction and prompt manual review.

[0077] Step S204, performing rule check on each data type in the to-be-processed data set, and generating an intermediate data set according to each data type after passing the rule check.

[0078] The data to be processed includes multiple data types. In the above example, the contract number, information about both parties to the contract, the signing date, and the contract amount are all different data types. In this example, rule checking is performed on each data type in the data to be processed. Rule checking is a process of verifying data to ensure that the data complies with preset business rules or standards.

[0079] Optionally, when checking the rules for each data type, the data in the data set to be processed will be checked one by one. If a field in the data does not meet the preset rules, the data entry is considered unqualified. The data is removed from the data set to be processed. This ensures that only data that meets the rules will enter the subsequent processing steps.

[0080] For example, if the data to be processed is contract data, if the contract number, signing date, amount and other fields do not meet the rules, the contract information may be incomplete or erroneous. In this case, the validity of the contract may not be guaranteed, and may even lead to the contract being invalid or not recognized. Therefore, unqualified data will be discovered and raised during the rule checking stage to ensure smooth subsequent processing.

[0081] Optionally, a data set that meets the requirements and is screened out through rule checking is used as an intermediate data set.

[0082] Step S206: perform data verification on the intermediate data set to obtain the target data set.

[0083] In step S204, the data corresponding to each data type in the data to be processed is checked for rules. In this step, it is only to ensure whether the data of each data type meets the corresponding format requirements, such as whether it meets the date rule and the amount rule. However, this is only a check on the data in terms of format, but it cannot guarantee the authenticity and validity of the data. Therefore, in this step, data verification is also performed on the intermediate data set to ensure the authenticity and validity of each data.

[0084] Optionally, the logical correlation between the fields will be checked, for example, the contract signing date must be earlier than the contract effective date, the contract amount should be consistent with the amount in the payment information or within a reasonable error range, etc.

[0085] Step S208: when the timestamp of the target data set is within the preset time period, a preset reference data set corresponding to the target data set is obtained, and the difference between the target data set and the reference data set is calculated.

[0086] Optionally, traverse the target data set and read the timestamp field of each data item. Compare whether the timestamp is within a preset time period, for example, the time period can be the past 30 days, the current quarter, or a specific date range. If the timestamp is within the preset range, obtain a reference data set corresponding to the target data set from the preset storage of the backend system (such as a database, cache, file backend system, etc.) for comparison.

[0087] Optionally, match the data items in the target data set with the data items in the reference data set based on the unique identifier of the data (such as ID or key field). Check whether there are differences in the values ​​of each field (such as changes in amount, different status, etc.). The comparison content may include changes in values, status updates, addition or deletion of fields, etc. Divide the difference information into types such as addition, deletion, and modification, and record the specific difference content. For example, record the field name, original value, new value, and difference description.

[0088] Step S210: when the difference meets the preset condition, a data processing result is generated according to the target data set.

[0089] Optionally, check each difference data to verify the conditions it meets. For example, the difference may include the number of newly added records, modified fields, or deleted records. Determine whether the difference data meets specific preset conditions (such as the data value change exceeds a certain threshold, the state changes, etc.).

[0090] Optionally, corresponding data processing results are generated based on the differences that meet the preset conditions. For example, newly added data in the target data set is marked as "new data" or "to be processed", and a corresponding processing task or log record is generated.

[0091] In the above data processing method, the efficiency of data processing is improved through automated steps: first, rule checking automatically filters out data that does not meet the requirements, reducing the workload and error rate of manual screening; second, data verification further ensures the accuracy and consistency of the data, avoiding the tedious process of manual verification one by one; in the difference calculation step, data changes are quickly identified through automatic comparison with reference data, reducing the time for manual inspection of differences; finally, the system automatically determines whether to generate data processing results through preset conditions, can quickly respond to abnormal situations, reduces the need for manual configuration and manual intervention, significantly improves processing efficiency, and optimizes the data management process.

[0092] In one embodiment, the above-mentioned rule check is performed on each data type in the data set to be processed, and after passing the rule check, an intermediate data set is generated according to each data type, including: obtaining preset rules corresponding to each data type, and performing rule checks on each data type according to the preset rules; when each data type satisfies the preset rules corresponding to the rule check, marking the data to be processed corresponding to the data type as target data; integrating the target data according to the preset data structure to obtain an intermediate data set.

[0093] Optionally, a preset rule corresponding to each data type is obtained, and the data to be processed of each data type is verified according to the obtained preset rule. When a data type satisfies the corresponding preset rule, the data is marked as target data. Finally, according to the preset data structure, all data marked as target data are integrated into a new set, which is the intermediate data set.

[0094] For example, if the data to be processed is transaction data, the preset rules include date rule, amount rule and account type rule. The date rule must be "YYYY-MM-DD" and the date cannot exceed the current date; the amount rule must be a positive number and not exceed 100,000 yuan; the account type rule is a personal account.

[0095] Assume that the flow data is:

[0096] Transaction 1: Date = 2024-10-05, Amount = 5,000 yuan, Account Type = Personal Account

[0097] Transaction 2: Date = 2024-11-01, Amount = -300 yuan, Account Type = Corporate Account

[0098] Transaction 3: Date = 2024-09-30, Amount = 120,000 yuan, Account Type = Personal Account

[0099] Transaction 4: Date = 2024-10-04, Amount = 20,000 yuan, Account Type = Other

[0100] Among them, flow 1 complies with the rules and is marked as target data, while flow 2, flow 3 and flow 4 are marked as unqualified data because they do not comply with the rules.

[0101] Then, only pipeline 1 marked as target data is integrated into the intermediate data set.

[0102] In the above embodiment, valid data is screened through rule checking and integrated into a unified intermediate data set, thereby improving the accuracy of data processing and the simplicity of subsequent operations.

[0103] In one embodiment, the above-mentioned obtaining preset rules corresponding to each data type and performing rule checking on each data type according to the preset rules include: reading the data tag corresponding to each data type; obtaining the preset rules corresponding to each data type according to the data tag, and performing rule checking on each data type based on the preset rules.

[0104] Optionally, each data type usually has a data label associated with it, which is used to identify the attributes of the data. For example, "date", "amount" and "account type" in the running data are data labels.

[0105] Optionally, the preset rules vary according to the data tags. For example, the rule for "date" may be format checking and time range restriction, the rule for "amount" may be value range restriction, and the rule for "account type" may be type matching.

[0106] Optionally, validate each piece of data. For example: Check if the "Date" is in the format and within the allowed time range. Verify if the "Amount" is a positive number and does not exceed the limit. Check if the "Account Type" belongs to the specified category. If a piece of data meets all the preset rules, it is marked as qualified data.

[0107] In one embodiment, targeted data checking is achieved by accurately matching data tags with preset rules, ensuring that the data meets specific requirements and improving the accuracy of data verification.

[0108] In one embodiment, the above-mentioned data verification is performed on the intermediate data set to obtain the target data set, including: when the data to be processed is contract data, obtaining a preset contract template; extracting target fields and resource fields from the intermediate data corresponding to each data type according to the preset contract template; verifying the resource field based on the target field to obtain a verification result; when the verification result is passed and the status corresponding to the resource field is acquired, generating a target data set according to the target field.

[0109] During the data verification process, if the intermediate data set belongs to the contract data type, the backend system first needs to obtain the corresponding preset contract template from the template library. The contract template defines the standard structure and necessary fields of the contract.

[0110] Exemplarily, the backend system selects a suitable contract template from the template library according to the contract type (eg, purchase contract, sales contract), wherein the contract template includes definitions of target fields such as contract number, contract parties, amount, signing date, etc.

[0111] Then, using the contract template, the backend system extracts the key data fields of the contract, namely the target fields, and the related auxiliary data, namely the resource fields, from the intermediate data set.

[0112] For example, the target field may include core information of the contract, such as the contract number, contract amount, signing date, etc. The resource field may include auxiliary information, such as review status, contract attachments, approvers, etc.

[0113] Furthermore, the target field is used to check the resource field to ensure that the status and content of the resource field meet the requirements of the contract. For example, the contract amount determines what the payment status should be, so the resource can be checked for logical dependency through the target field.

[0114] For example, starting from the contract number AAA, the backend system searches for the corresponding approval status. If the approval status of the contract number AAA is "unapproved", the contract data cannot enter the next stage.

[0115] For example, if the contract amount in the target field is RMB 100,000, the backend system will check the resource field "Payment Status". At this time, the payment status of the resource field must be "paid" or "partially paid".

[0116] Finally, only when the verification result is passed and the status of the resource field is "acquired", the target field in the intermediate data set will be extracted and integrated into the target data set.

[0117] Optionally, if all target fields and resource fields pass the verification, they are integrated into the target data set and marked as "valid contract data". Otherwise, the data is rejected from being included in the target data set and the reason for the verification failure is recorded for subsequent processing or manual review.

[0118] In the above embodiment, the preset template and field extraction help to accurately verify the data and automatically generate a target data set that meets the standards.

[0119] In one embodiment, the above-mentioned data verification of the intermediate data set to obtain the target data set includes: when the data to be processed is transaction data, data normalization of the intermediate data corresponding to each data type; integrity verification of the intermediate data corresponding to the data type after data normalization to obtain the intermediate data corresponding to the target data type; and aggregating the intermediate data corresponding to the target data type to obtain the target data set.

[0120] First, data normalization is performed on each data type in the transaction data to be processed, such as transaction amount, transaction time, transaction ID, etc. The purpose of normalization is to ensure that all data conforms to a unified format or standard.

[0121] For example, if the amount fields in the transaction data use different currency units (such as US dollars and euros), the data normalization process will convert all amounts into a preset currency unit, such as RMB.

[0122] Secondly, the integrity of the intermediate data corresponding to the data type after data normalization is verified. In this embodiment, the integrity verification is performed on the consistency of the data fields, that is, to ensure that the logical relationship between different data types is correct, and to verify whether the data in the data set conforms to the overall constraints of the business logic, rather than simply whether the field exists.

[0123] For example, in transaction data, fields such as transaction amount, customer information, and transaction time must not only comply with the rules independently, but also maintain consistency with each other. For example, check whether the transaction type is a refund and confirm that the transaction amount is a negative value; check the transaction time and ensure that the transaction status is completed.

[0124] Finally, the intermediate data corresponding to the target data type is aggregated to obtain the target data set. Aggregation is to merge multiple separate flow records together according to a certain logic in order to generate a unified result that meets specific business needs. It is often used to generate summary reports or for settlement, liquidation and other operations.

[0125] Optionally, for each customer, multiple transactions of the same customer can be classified in each day's transaction flow, for example, separately counting income flow, expenditure flow, transfer flow, etc.

[0126] In the above embodiment, data normalization, integrity verification and aggregation processing are used to ensure that the generated target data set has both a unified format and complete information, thereby meeting the business analysis requirements of transaction data.

[0127] In one embodiment, the preset time period range includes a first sub-preset time period range, a second sub-preset time period range and a third sub-preset time period range; the first sub-preset time period range is smaller than the second sub-preset time period range, and the second sub-preset time period range is smaller than the third sub-preset time period range; when the timestamp of the target data set is within the preset time period range, the preset stored reference data set corresponding to the target data set is obtained, and the difference between the target data set and the reference data set is calculated, including: when the timestamp is within the first sub-preset time period range, the data message parameters and / or configuration parameters corresponding to the target data set are obtained; when the timestamp is within the second sub-preset time period range, the reference data set is obtained according to the message parameters and / or configuration parameters; when the timestamp is within the third sub-preset time period range, the difference between the target data set and the reference data set is calculated.

[0128] Optionally, a preset time period range is first set, which is divided into three sub-preset time periods, and the first sub-preset time period range is smaller than the second sub-preset time period range, and the second sub-preset time period range is smaller than the third sub-preset time period range. This structure allows data processing to be performed step by step in chronological order. When the timestamp of the target data set is within the preset time period range, the background system will obtain the reference data set corresponding to the target data set, and perform corresponding processing according to different time periods.

[0129] Furthermore, in the first sub-preset time period, the background system will obtain message parameters and / or configuration parameters related to the target data set. The purpose of this step is to ensure that there is correct context information when processing data.

[0130] Furthermore, in the second sub-preset time period, the background system will use the message parameters and / or configuration parameters obtained in the previous step to generate a reference data set. This process ensures that the generation of the reference data is consistent with the previous configuration.

[0131] Furthermore, during the third sub-preset time period, the background system calculates the difference between the target data set and the reference data set, aiming to find out the changes or inconsistencies in the data, thereby providing a basis for subsequent data processing and decision-making.

[0132] In the above embodiment, through such a design, the entire process not only helps to ensure the background systematization and consistency of data processing, but also can timely discover the differences between data and improve the accuracy and efficiency of data processing.

[0133] In an exemplary embodiment, in combination with FIG3 , FIG3 is a flowchart of the steps of a data flow method in an embodiment.

[0134] Step 302, receiving the data processing request after the front-end review is passed.

[0135] Receive data processing requests after front-end review. The data set to be processed will contain transaction data related to clearing and payment.

[0136] Step 304: rule checking and intermediate data generation.

[0137] Perform rule checks on each data type in the data set to be processed to ensure that the data complies with preset business rules, for example, whether the contract number exists, whether the signing date is within a reasonable range, and whether the amount is a positive number.

[0138] If the data passes the rule check, it is marked as target data and integrated into an intermediate data set.

[0139] Step 306: perform data verification on the intermediate data set to obtain the target data set.

[0140] When the data to be processed is liquidation data, obtain the preset liquidation template and extract the target field and resource field. Use the target field to check the resource field to ensure that the status and content meet the liquidation requirements, for example, verify whether the liquidation status is "approved" and whether the amount is within a reasonable range.

[0141] When the data to be processed is payment data, the preset payment template is also obtained to extract the target field and resource field. The consistency of the target field and the resource field is verified, for example, whether the payment amount matches the clearing amount and whether the approval status meets expectations.

[0142] Step 308: Data time period check and difference calculation.

[0143] When the timestamp of the target data set is within the preset time period, the following operations are performed:

[0144] First sub-time period range: Get data message parameters and / or configuration parameters corresponding to the target data set. These parameters include detailed information related to clearing and payment, such as expected amount, relevant date, etc.

[0145] Second sub-time period: Generate a reference data set based on the acquired message parameters and / or configuration parameters. The reference data set will be used to compare and verify the accuracy of the target data set.

[0146] The third sub-time period range: calculate the difference between the target data set and the reference data set.

[0147] Step 310: Difference determination and result generation.

[0148] If the amount in the target data set is 1,000 yuan, and the amount in the reference data set is 900 yuan, the difference is 100 yuan. If this difference exceeds the preset tolerance (for example, 10%), it is considered an anomaly.

[0149] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0150] Based on the same inventive concept, the embodiment of the present application also provides a data processing device for implementing the data processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in the one or more data processing device embodiments provided below can refer to the limitations on the data processing method above, and will not be repeated here.

[0151] In an exemplary embodiment, as shown in FIG4 , a data processing device is provided, including: a receiving module 410, a checking module 420, a verification module 430, a difference calculation module 440 and a result generation module 450, wherein:

[0152] The receiving module 410 is used to receive a data processing request after the front-end review is passed; the data processing request carries a set of data to be processed.

[0153] The checking module 420 is used to perform rule checking on each data type in the data set to be processed, and generate an intermediate data set according to each data type after passing the rule checking.

[0154] The verification module 430 is used to perform data verification on the intermediate data set to obtain the target data set.

[0155] The difference calculation module 440 is used to obtain a preset reference data set corresponding to the target data set when the timestamp of the target data set is within a preset time period, and calculate the difference between the target data set and the reference data set.

[0156] The result generation module 450 is used to generate a data processing result according to the target data set when the difference meets a preset condition.

[0157] In one embodiment, the inspection module includes:

[0158] The rule checking unit is used to obtain preset rules corresponding to each data type and perform rule checking on each data type according to the preset rules.

[0159] The marking unit is used to mark the to-be-processed data corresponding to the data type as target data when each data type satisfies the preset rules corresponding to the rule check.

[0160] The first set generating unit is used to integrate the target data according to a preset data structure to generate an intermediate data set.

[0161] In one embodiment, the rule checking unit includes:

[0162] The tag reading subunit is used to read the data tags corresponding to each data type.

[0163] The rule acquisition and checking subunit is used to acquire the preset rules corresponding to each data type according to the data tag, and perform rule checking on each data type based on the preset rules.

[0164] In one embodiment, the verification module includes:

[0165] The template acquisition unit is used to acquire a preset contract template when the data to be processed is contract data.

[0166] The field acquisition unit is used to extract the target field and the resource field from the intermediate data corresponding to each data type according to the preset contract template.

[0167] The verification result unit is used to verify the resource field based on the target field to obtain the verification result.

[0168] The second set generating unit is used to generate a target data set according to the target field when the verification result is passed and the status corresponding to the resource field is acquired.

[0169] In one embodiment, the verification module includes:

[0170] The normalization unit is used to normalize the intermediate data corresponding to each data type when the data to be processed is transaction data.

[0171] The integrity verification unit is used to perform integrity verification on the corresponding intermediate data row of the data type after data normalization to obtain the intermediate data corresponding to the target data type.

[0172] The aggregation unit is used to aggregate the intermediate data corresponding to the target data type to obtain the target data set.

[0173] In one embodiment, the verification module further includes:

[0174] The logic checking unit is used to check the logic consistency between target fields.

[0175] In one embodiment, the preset time period range includes a first sub-preset time period range, a second sub-preset time period range, and a third sub-preset time period range; the first sub-preset time period range is smaller than the second sub-preset time period range, and the second sub-preset time period range is smaller than the third sub-preset time period range, and the result generation module includes:

[0176] The first processing unit is configured to obtain data message parameters and / or configuration parameters corresponding to the target data set when the timestamp is within a first sub-preset time period.

[0177] The second processing unit is configured to obtain a reference data set according to the message parameters and / or configuration parameters when the timestamp is within a second sub-preset time period.

[0178] The third processing unit is used to calculate the difference between the target data set and the reference data set when the timestamp is within a third sub-preset time period.

[0179] Each module in the above data processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each module.

[0180] In an exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be shown in FIG5. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data to be processed. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a data processing method is implemented.

[0181] Those skilled in the art will understand that the structure shown in FIG. 5 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0182] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps of the method in any one of the above embodiments when executing the computer program.

[0183] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method in any one of the above embodiments are implemented.

[0184] In one embodiment, a computer program product is provided, comprising a computer program, which implements the steps of the method in any one of the above embodiments when executed by a processor.

[0185] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0186] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0187] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data processing method, characterized in that: The method comprises: Receive a data processing request after the front-end review is passed; the data processing request carries a set of data to be processed; Performing rule checking on each data type in the to-be-processed data set, and generating an intermediate data set according to each data type after passing the rule checking; Performing data verification on the intermediate data set to obtain a target data set; performing data verification on the intermediate data set to obtain a target data set includes: when the data to be processed is contract data, obtaining a preset contract template; extracting a target field and a resource field from the intermediate data corresponding to each data type according to the preset contract template; verifying the resource field based on the target field to obtain a verification result; when the verification result is passed and the status corresponding to the resource field is acquired, generating the target data set according to the target field; When the timestamp of the target data set is within a preset time period, obtaining a preset reference data set corresponding to the target data set, and calculating the difference between the target data set and the reference data set; When the difference meets a preset condition, a data processing result is generated according to the target data set.

2. The method according to claim 1, characterized in that The performing rule checking on each data type in the to-be-processed data set and generating an intermediate data set according to each data type after passing the rule checking includes: Acquire preset rules corresponding to each of the data types, and perform rule checking on each of the data types according to the preset rules; When each of the data types satisfies the preset rule corresponding to the rule check, marking the to-be-processed data corresponding to the data type as target data; The target data is integrated according to a preset data structure to obtain the intermediate data set.

3. The method according to claim 2, characterized in that The obtaining of preset rules corresponding to each of the data types, and performing rule checking on each of the data types according to the preset rules, includes: Read the data tags corresponding to the data types; The preset rules corresponding to each of the data types are obtained according to the data tags, and rule checks are performed on each of the data types based on the preset rules.

4. The method according to claim 1, characterized in that: After extracting the target field from the intermediate data corresponding to each of the data types according to the preset contract template, the method further includes: The logical consistency between the target fields is checked.

5. The method according to claim 1, characterized in that The step of performing data verification on the intermediate data set to obtain a target data set includes: When the data to be processed is transaction data, data normalization is performed on the intermediate data corresponding to each data type; Performing integrity verification on the intermediate data corresponding to the data type after data normalization to obtain the intermediate data corresponding to the target data type; Aggregate the intermediate data corresponding to the target data type to obtain the target data set.

6. The method according to claim 1, characterized in that The preset time period range includes a first sub-preset time period range, a second sub-preset time period range and a third sub-preset time period range; the first sub-preset time period range is smaller than the second sub-preset time period range, and the second sub-preset time period range is smaller than the third sub-preset time period range; When the timestamp of the target data set is within a preset time period, obtaining a preset reference data set corresponding to the target data set, and calculating the difference between the target data set and the reference data set, comprises: When the timestamp is within the first sub-preset time period, acquiring data message parameters and / or configuration parameters corresponding to the target data set; When the timestamp is within the second preset sub-time period, obtaining the reference data set according to the message parameter and / or the configuration parameter; When the timestamp is within the third sub-preset time period, the difference between the target data set and the reference data set is calculated.

7. A data processing device, characterized in that: The device comprises: A receiving module, used to receive a data processing request after the front-end review has been passed; the data processing request carries a set of data to be processed; A checking module, used for performing a rule check on each data type in the to-be-processed data set, and generating an intermediate data set according to each data type after passing the rule check; A verification module, used to perform data verification on the intermediate data set to obtain a target data set; the verification module includes: a template acquisition unit, used to obtain a preset contract template when the data to be processed is contract data; a field acquisition unit, used to extract a target field and a resource field from the intermediate data corresponding to each data type according to the preset contract template; a verification result unit, used to verify the resource field based on the target field to obtain a verification result; a second set generation unit, used to generate the target data set according to the target field when the verification result is passed and the status corresponding to the resource field is acquired; a difference calculation module, configured to obtain a preset reference data set corresponding to the target data set when the timestamp of the target data set is within a preset time period, and calculate the difference between the target data set and the reference data set; The result generating module is used to generate a data processing result according to the target data set when the difference meets a preset condition.

8. The device according to claim 7, characterized in that The inspection module comprises: A rule checking unit, used to obtain preset rules corresponding to each of the data types, and perform rule checking on each of the data types according to the preset rules; A marking unit, configured to mark the to-be-processed data corresponding to the data type as target data when the data type satisfies the preset rule corresponding to the rule check; The first set generating unit is used to integrate the target data according to a preset data structure to generate the intermediate data set.

9. The device according to claim 8, characterized in that The rule checking unit comprises: A tag reading subunit, used to read data tags corresponding to each of the data types; The rule acquisition and checking subunit is used to acquire the preset rules corresponding to each of the data types according to the data tags, and perform rule checking on each of the data types based on the preset rules.

10. The device according to claim 7, characterized in that The verification module also includes: The logic checking unit is used to check the logic consistency between the target fields.

11. The device according to claim 7, characterized in that The verification module comprises: A normalization unit, used for normalizing the intermediate data corresponding to each data type when the data to be processed is transaction data; An integrity verification unit, used to perform integrity verification on the corresponding intermediate data row of the data type after data normalization, and obtain intermediate data corresponding to the target data type; The aggregation unit is used to aggregate the intermediate data corresponding to the target data type to obtain the target data set.

12. The device according to claim 7, characterized in that in, The preset time period range includes a first sub-preset time period range, a second sub-preset time period range and a third sub-preset time period range; the first sub-preset time period range is smaller than the second sub-preset time period range, and the second sub-preset time period range is smaller than the third sub-preset time period range, and the result generation module includes: A first processing unit, configured to obtain data message parameters and / or configuration parameters corresponding to the target data set when the timestamp is within the first sub-preset time period; A second processing unit, configured to obtain the reference data set according to the message parameter and / or the configuration parameter when the timestamp is within the second preset sub-time period; The third processing unit is configured to calculate the difference between the target data set and the reference data set when the timestamp is within the third sub-preset time period.

13. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

14. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images