Data processing method and device, electronic equipment and computer readable medium

Abstract

The invention discloses a data processing method and device, electronic equipment and a computer readable medium, and relates to the technical field of big data processing.The specific implementation mode comprises the steps that a data processing request is received, and to-be-processed data, a time identifier and a mechanism identifier are obtained; based on the time identifier and the mechanism identifier, slicing the to-be-processed data to obtain sliced data; determining a data processing type, obtaining a database identifier and configuration data, and calling the target framework to generate a query statement corresponding to a database corresponding to the database identifier based on the data processing type, the database identifier and the configuration data; executing the query statement based on the slice data, obtaining query result data, and determining difference data in response to the difference between the query result data and expected data; and generating a correction task based on the query result data, the expected data and the difference data, and calling the thread pool to execute the correction task to obtain correction result data. And the data processing efficiency and accuracy are improved.

Description

Technical Field

[0001] The present application relates to the field of big data processing technology, and in particular to a data processing method, device, electronic device and computer-readable medium. Background Art

[0002] Currently, database management systems usually rely on pre-written query statements to perform data operations. These operations include querying, updating, inserting, and deleting data. However, these systems usually require database administrators or developers to have professional knowledge to write and maintain these query statements. In addition, database management systems often lack flexibility, cannot quickly adapt to changes in business rules, cannot quickly respond to data changes and maintain data consistency and accuracy, and have low efficiency and accuracy in processing corresponding data. Summary of the invention

[0003] In view of this, the embodiments of the present application provide a data processing method, device, electronic device and computer-readable medium, which can solve the existing problems of being unable to quickly respond to data changes and maintain data consistency and accuracy, and low efficiency and accuracy of corresponding data processing.

[0004] To achieve the above objective, according to one aspect of an embodiment of the present application, a data processing method is provided, including:

[0005] Receive data processing requests, obtain data to be processed, time stamp and organization identification;

[0006] Slicing the data to be processed based on the time identifier and the organization identifier to obtain sliced ​​data;

[0007] Determine a data processing type, obtain a database identifier and configuration data, and call a target framework to generate a query statement corresponding to a database corresponding to the database identifier based on the data processing type, the database identifier and the configuration data;

[0008] Execute a query statement based on the slice data to obtain query result data, and in response to a difference between the query result data and the expected data, determine the difference data;

[0009] Based on the query result data, expected data and difference data, a correction task is generated, and the thread pool is called to execute the correction task to obtain the correction result data.

[0010] Optionally, based on the time identifier and the organization identifier, the data to be processed is sliced ​​to obtain sliced ​​data, including:

[0011] Determine the data source based on the organization identifier, and obtain target data corresponding to the data source from the data to be processed;

[0012] Based on the time stamp, the target data is sliced ​​to obtain sliced ​​data.

[0013] Optionally, determine the type of data processing, including:

[0014] In response to the existing query result data, determining that the data processing type is update;

[0015] In response to the absence of query result data, the data processing type is determined to be query.

[0016] Optionally, generating a query statement corresponding to a database corresponding to the database identifier includes:

[0017] According to the data processing type, call the corresponding database statement to generate the template;

[0018] The corresponding placeholders in the database statement generation template are replaced with the database identifier, the threshold value in the configuration data, and the target field name to obtain a query statement corresponding to the database corresponding to the database identifier.

[0019] Optionally, before determining the data processing type, the method further includes:

[0020] In response to the data to be processed being form data, creating a temporary table having the same structure as the form data;

[0021] Insert the data to be processed into a temporary table.

[0022] Optionally, after determining the data processing type, the method further includes:

[0023] According to the data processing type, call the corresponding database statement to generate the template;

[0024] Get the ID of the temporary table, and replace the corresponding placeholders in the database statement generation template with the ID of the temporary table, the threshold value in the configuration data, and the target field name to obtain the query statement corresponding to the temporary table.

[0025] In addition, the present application also provides a data processing device, including:

[0026] A receiving unit, configured to receive a data processing request, and obtain the data to be processed, a time mark, and an organization mark;

[0027] A slicing unit is configured to slice the data to be processed based on the time identifier and the organization identifier to obtain sliced ​​data;

[0028] A query statement generating unit is configured to determine a data processing type, obtain a database identifier and configuration data, and call a target framework to generate a query statement corresponding to a database corresponding to the database identifier based on the data processing type, the database identifier, and the configuration data;

[0029] a difference data determining unit, configured to execute a query statement based on the slice data, obtain query result data, and determine the difference data in response to a difference between the query result data and the expected data;

[0030] The execution unit is configured to generate a correction task based on the query result data, the expected data and the difference data, and call the thread pool to execute the correction task to obtain the correction result data.

[0031] Optionally, the slicing unit is further configured to:

[0032] Determine the data source based on the organization identifier, and obtain target data corresponding to the data source from the data to be processed;

[0033] Based on the time stamp, the target data is sliced ​​to obtain sliced ​​data.

[0034] Optionally, the query statement generating unit is further configured to:

[0035] In response to the existing query result data, determining that the data processing type is update;

[0036] In response to the absence of query result data, the data processing type is determined to be query.

[0037] Optionally, the query statement generating unit is further configured to:

[0038] According to the data processing type, call the corresponding database statement to generate the template;

[0039] The corresponding placeholders in the database statement generation template are replaced with the database identifier, the threshold value in the configuration data, and the target field name to obtain a query statement corresponding to the database corresponding to the database identifier.

[0040] Optionally, the device further includes a data inserting unit configured to:

[0041] In response to the data to be processed being form data, creating a temporary table having the same structure as the form data;

[0042] Insert the data to be processed into a temporary table.

[0043] Optionally, the query statement generating unit is further configured to:

[0044] According to the data processing type, call the corresponding database statement to generate the template;

[0045] Get the ID of the temporary table, and replace the corresponding placeholders in the database statement generation template with the ID of the temporary table, the threshold value in the configuration data, and the target field name to obtain the query statement corresponding to the temporary table.

[0046] In addition, the present application also provides a data processing electronic device, including: one or more processors; a storage device for storing one or more programs, when the one or more programs are executed by one or more processors, the one or more processors implement the data processing method as described above.

[0047] In addition, the present application also provides a computer-readable medium on which a computer program is stored, and when the program is executed by a processor, the data processing method as described above is implemented.

[0048] To achieve the above objective, according to another aspect of the embodiments of the present application, a computer program product is provided.

[0049] A computer program product according to an embodiment of the present application includes a computer program, and when the program is executed by a processor, the data processing method provided by the embodiment of the present application is implemented.

[0050] An embodiment of the above invention has the following advantages or beneficial effects: the application receives a data processing request to obtain the data to be processed, the time mark and the organization mark; slices the data to be processed based on the time mark and the organization mark to obtain sliced ​​data; determines the data processing type, obtains the database mark and configuration data, calls the target framework to generate a query statement corresponding to the database corresponding to the database mark based on the data processing type, the database mark and the configuration data; executes the query statement based on the sliced ​​data to obtain the query result data, and determines the difference data in response to the difference between the query result data and the expected data; generates a correction task based on the query result data, the expected data and the difference data, and calls the thread pool to execute the correction task to obtain the correction result data. Improve data processing efficiency and accuracy.

[0051] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] The accompanying drawings are used to better understand the present application and do not constitute an improper limitation on the present application.

[0053] Figure 1 is a schematic diagram of the main flow of a data processing method according to an embodiment of the present application;

[0054] Figure 2 is a schematic diagram of the main flow of a data processing method according to an embodiment of the present application;

[0055] Figure 3 It is a main flow chart of a data processing method according to an embodiment of the present application;

[0056] Figure 4is a schematic diagram of main units of a data processing device according to an embodiment of the present application;

[0057] Figure 5 is an exemplary system architecture diagram to which the embodiments of the present application can be applied;

[0058] Figure 6 It is a structural diagram of a computer system of a terminal device or server suitable for implementing an embodiment of the present application. DETAILED DESCRIPTION

[0059] The following is a description of the exemplary embodiments of the present application in conjunction with the accompanying drawings, including various details of the embodiments of the present application to facilitate understanding, and they should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present application. Similarly, for the sake of clarity and conciseness, the description of known functions and structures is omitted in the following description. It should be noted that the acquisition, transmission, storage, use, and processing of data in the technical solution of the present application are in compliance with the relevant provisions of national laws and regulations. It should be noted that in the embodiments of the present application, certain software, components, models, and other existing solutions in the industry may be mentioned, which should be considered as exemplary, and their purpose is only to illustrate the feasibility of the implementation of the technical solution of the present application, but it does not mean that the applicant has or must have used the solution. In the technical solution of the present application, the collection, analysis, use, transmission, storage, and other aspects of the user's personal information involved are in compliance with the provisions of relevant laws and regulations, and are used for legal and reasonable purposes, and are not shared, leaked, or sold outside of these legal uses, and are subject to supervision and management by regulatory authorities. Take necessary measures to prevent illegal access to such personal information, ensure that persons with access to personal information comply with relevant laws and regulations, and ensure the security of personal information. Once such personal information is no longer needed, risks should be minimized by limiting or even prohibiting data collection and / or deleting data.

[0060] When used, including in certain related applications, user privacy is protected by de-identifying data, such as by removing specific identifiers when used, controlling the amount or specificity of data stored, controlling how data is stored, and / or other methods of de-identification.

[0061] Figure 1 is a schematic diagram of the main flow of a data processing method according to an embodiment of the present application, such as Figure 1 As shown, the data processing method mainly includes the following steps S101 to S105.

[0062] Step S101, receiving a data processing request, obtaining data to be processed, a time mark and an organization mark.

[0063] In this embodiment, the execution subject of the data processing method (for example, it can be a server) can receive the data processing request through a wired connection or a wireless connection. The data processing request can be a request to automatically detect and report data that does not comply with the configuration rules, thereby helping to maintain the consistency and accuracy of the data. The embodiment of the present application does not specifically limit the content of the data processing request. After receiving the data processing request, the execution subject can obtain the data to be processed, the time mark and the organization mark corresponding to the request. Among them, the time mark can be a time point or time period of interest configured by the user, such as July 2024, and the organization mark can be the code or name of the organization of interest configured by the user, such as organization YY.

[0064] Step S102, slicing the data to be processed based on the time identifier and the organization identifier to obtain sliced ​​data.

[0065] When there are multiple different organizations corresponding to the organization identifier, the executing entity can filter out the target data corresponding to the time identifier from the data to be processed, and slice the target data according to the organization identifier to quickly and accurately obtain each slice data, thereby ensuring the efficiency and accuracy of data processing.

[0066] Step S103, determining the data processing type, obtaining the database identifier and configuration data, and calling the target framework to generate a query statement corresponding to the database corresponding to the database identifier based on the data processing type, the database identifier and the configuration data.

[0067] Specifically, determining the data processing type includes: in response to existing query result data, determining the data processing type as update; in response to no query result data, determining the data processing type as query.

[0068] The data processing type may include update and query. The data processing type may be determined based on the existing query result data to ensure the accuracy of the data processing type determination.

[0069] Determine the definition data corresponding to each placeholder in the target framework, and perform semantic similarity matching between the data processing type, database identifier and configuration data and the definition data of each placeholder to determine the placeholders corresponding to the data processing type, database identifier and configuration data, and then replace the statements of the corresponding placeholders to obtain the query statement corresponding to the database corresponding to the database identifier, thereby realizing personalized and accurate generation of query statements corresponding to the database and ensuring the accuracy of data processing.

[0070] Step S104, executing a query statement based on the slice data to obtain query result data, and in response to a difference between the query result data and the expected data, determining the difference data.

[0071] Executing query statements in the slice data corresponding to the obtained data to be processed to accurately query and obtain the difference data that does not conform to the configuration rules can improve the data query speed and further improve the data processing speed.

[0072] Step S105, generating a correction task based on the query result data, expected data and difference data, and calling a thread pool to execute the correction task to obtain correction result data.

[0073] Call the task generation thread to fill the query result data, expected data and difference data into the correction task generation template to generate the corresponding correction task. Put the correction task into the asynchronous task pool, call the configured thread pool to obtain and execute the correction task from the asynchronous task pool to quickly and accurately obtain the correction result data.

[0074] This embodiment receives a data processing request, obtains the data to be processed, the time mark and the organization mark; slices the data to be processed based on the time mark and the organization mark to obtain sliced ​​data; determines the data processing type, obtains the database mark and configuration data, calls the target framework to generate a query statement corresponding to the database corresponding to the database mark based on the data processing type, the database mark and the configuration data; executes the query statement based on the sliced ​​data to obtain the query result data, and determines the difference data in response to the difference between the query result data and the expected data; generates a correction task based on the query result data, the expected data and the difference data, and calls the thread pool to execute the correction task to obtain the correction result data. Improve the efficiency and accuracy of data processing.

[0075] Figure 2 is a schematic diagram of the main flow of a data processing method according to an embodiment of the present application, such as Figure 2 As shown, the data processing method mainly includes the following steps S201 to S206.

[0076] Step S201, receiving a data processing request, obtaining the data to be processed, a time mark and an organization mark.

[0077] The data to be processed may be the user's financial transaction data, and the time identifier may be a continuous time point or time period, or may be a discontinuous time point or time period. The embodiment of the present application does not specifically limit the time identifier. The organization identifier may be the number or name of the organization that the user needs to analyze.

[0078] Step S202, determining the data source based on the organization identifier, and obtaining target data corresponding to the data source from the data to be processed.

[0079] Multiple organization identifiers can correspond to one data source, and the target data corresponding to the determined data source can be obtained from the data to be processed. By determining the data source corresponding to the organization identifier, the number of data acquisition times can be reduced, data acquisition time can be saved, time waste caused by repeated access to the same data source can be avoided, and data processing efficiency and accuracy can be improved.

[0080] Step S203, slicing the target data based on the time identifier to obtain sliced ​​data.

[0081] The target data is sliced ​​according to the continuous or discontinuous time points and time periods corresponding to the time identifiers, so as to accurately and efficiently obtain the sliced ​​data that the user is interested in, reduce the time waste caused by processing unimportant data, and improve data processing efficiency.

[0082] Step S204, determine the data processing type, obtain the database identifier and configuration data, and call the target framework to generate a query statement corresponding to the database corresponding to the database identifier based on the data processing type, the database identifier and the configuration data.

[0083] Specifically, before determining the data processing type, the method further includes: in response to the data to be processed being form data, creating a temporary table having the same structure as the form data; and inserting the data to be processed into the temporary table.

[0084] For example, in an embodiment of the present application, in addition to checking the existing data in the database, it also supports reusing user configuration data to check single form data (i.e., form data, i.e., data not in the database). The main steps are as follows: use the h2 database to create a temporary table with the same structure as the form data (h2 database, a lightweight Java database management system, emphasizes its fast speed, memory mode and embedded / server mode flexibility); insert the form data into the temporary table; generate the check SQL related to the temporary table according to the user's configuration (i.e., the query statement corresponding to the temporary table); execute these SQLs (i.e., the query statement corresponding to the temporary table) in the background to determine which rules have not passed and return the failed rules.

[0085] Specifically, after determining the data processing type, the method also includes: calling the corresponding database statement generation template according to the data processing type; obtaining the identifier of the temporary table, and replacing the corresponding placeholder in the database statement generation template with the identifier of the temporary table, the threshold in the configuration data, and the target field name to obtain a query statement corresponding to the temporary table.

[0086] The data processing type may include update and query. After the execution subject determines the data processing type, the corresponding database statement generation template may be called according to the data processing type. For example, when the data processing type is query, the database statement generation template is called to obtain the identifier of the temporary table, and the corresponding placeholder in the database statement generation template is replaced with the identifier of the temporary table, the threshold value in the configuration data, and the target field name to obtain the query statement corresponding to the temporary table.

[0087] Step S205, executing a query statement based on the slice data to obtain query result data, and in response to a difference between the query result data and the expected data, determining the difference data.

[0088] Expected data may be user-configured data, such as "a single transaction amount of the user does not exceed RMB 50,000". If the query result data is "a single transaction amount of the user is RMB 60,000", it is determined that there is a difference between the query result data and the expected data, and the difference data may be "a single transaction amount of the user exceeds RMB 10,000".

[0089] Step S206, generating a correction task based on the query result data, expected data and difference data, and calling a thread pool to execute the correction task to obtain correction result data.

[0090] The thread pool is called to adjust the transaction parameters based on the correction task generated by the query result data, the expected data and the difference data so that the query result data is consistent with the expected data, and the correction result data is obtained.

[0091] Figure 3 is a schematic diagram of the main flow of a data processing method according to an embodiment of the present application, such as Figure 3 As shown, the data processing method mainly includes the following steps S301 to S307.

[0092] Step S301, receiving a data processing request, obtaining the data to be processed, a time mark and an organization mark.

[0093] Step S302, slicing the data to be processed based on the time identifier and the organization identifier to obtain sliced ​​data.

[0094] When there is only one different organization corresponding to the organization identifier, the corresponding target data can be first obtained from the data to be processed according to the organization identifier, and then the target data can be sliced ​​according to the time identifier to obtain the sliced ​​data accurately and quickly.

[0095] Step S303, determine the data processing type, and obtain the database identifier and configuration data.

[0096] The execution subject can obtain the request data corresponding to the data processing request, and the request data can carry the data processing type identifier, and the data processing type is determined according to the data processing type identifier in the request data. The data to be processed is obtained according to the link carried in the request data. The database identifier is used to characterize the target database required for the data query involved in the data processing. The configuration data may include data such as the query date and condition threshold.

[0097] Step S304: according to the data processing type, call the corresponding database statement generation template.

[0098] The execution subject can determine the statement keyword corresponding to the data processing type, such as select, and match the statement keyword with the words in each database statement generation template, thereby accurately determining the database statement generation template corresponding to the data processing type.

[0099] Step S305: replace the corresponding placeholders in the database statement generation template with the database identifier, the threshold value in the configuration data, and the target field name, so as to obtain a query statement corresponding to the database corresponding to the database identifier.

[0100] The placeholders in the statement generation template are consistent with the data format of the corresponding data. The execution entity can determine the data format corresponding to the database identifier, the threshold in the configuration data, and the target field name, such as binary data, symbolic data, time series data, spatial data, etc., and determine the placeholders corresponding to the database identifier, the threshold in the configuration data, and the target field name according to the data format, and replace the corresponding placeholders in the database statement generation template with the database identifier, the threshold in the configuration data, and the target field name, so as to accurately and quickly obtain the query statement corresponding to the database corresponding to the database identifier.

[0101] Step S306, executing the query statement based on the slice data, obtaining the query result data, and determining the difference data in response to the difference between the query result data and the expected data.

[0102] The execution entity can arrange the slice data in time sequence, and then execute the query statement in the arranged slice data to obtain the query result data. In response to the difference between the query result data and the expected data, the difference data is determined, thereby ensuring the orderliness of the query result data and facilitating user understanding and reference.

[0103] Step S307, based on the query result data, expected data and difference data, a correction task is generated, and a thread pool is called to execute the correction task to obtain correction result data.

[0104] The thread pool is called to correct and adjust the transaction parameters based on the correction task generated based on the query result data, expected data and difference data so that the query result data is consistent with the expected data, and the corrected result data is obtained. The corrected result data is the data processing result data when the query result data is consistent with the expected data.

[0105] The embodiments of the present application can realize the automatic generation of SQL statements based on simple configuration. Users can configure SQL statements by defining a series of rules and parameters, and automatically generate SQL statements according to the configuration data. These SQL statements can be used to perform database query, update, insert and delete operations. It can also automatically detect and report data that does not comply with the configuration rules, thereby helping to maintain the consistency and accuracy of the data. Allow users to define data operation requirements by configuring rules and parameters. Based on these configurations, SQL statements suitable for a variety of database queries and update operations can be automatically generated without manual writing and execution. The system supports time and organization data slicing, and automatically detects and persists data that does not comply with the configuration rules. It helps to maintain the accuracy and consistency of data and reduce human errors. Based on the quick verification of a single piece of data in the h2 database, data that has not been stored can be verified without additional configuration.

[0106] Specifically, rule configuration and parameter definition: users define data operation requirements by configuring a series of rules and parameters. These rules can include business logic, data validation, data conversion, etc. For example, users can define a rule such as "the transaction amount of a user shall not exceed 50,000 yuan per transaction". Parameters can be static values ​​(such as fixed dates, constants) or dynamic values ​​(such as values ​​obtained from other data sources). Supports batch import through Excel templates to provide convenience for the generation of massive rules. Based on the user's rule configuration, the querydsl framework is used to generate SQL statements suitable for various database operations. These operations include: query: generate SELECT statements based on the query conditions defined by the user; update: update the records in the result table when the data that does not meet the rule configuration is queried according to the rules. As business rules change, the system can flexibly adjust the generated SQL statements. For example, if the business rules change, the system can automatically update the relevant SQL statements without manual intervention. This helps to maintain the consistency of data operations and reduces the workload of manual maintenance. Verification data slicing and verification result generation: Support automatic slicing and verification of data based on the login user time and institution dimensions. For example, if XX branch only cares about the data of a certain data source in July 2024, the user can configure the relevant time and institution parameters, and the system automatically slices the configuration parameters, thereby reducing the amount of verification data and improving verification efficiency. The generated result data will maximize the retention of the information of the original data in the original table, so that users can locate the original problem data and correct it. In addition to verifying the existing data in the database, it also supports reusing user configuration data to verify single form data (not data in the database). The main steps are as follows: Use h2 database to create a temporary table with the same structure as the form data (h2 database, a lightweight Java database management system, emphasizes its fast speed, memory mode and embedded / server mode flexibility); insert the form data into the temporary table; generate the verification sql related to the temporary table (that is, the query statement corresponding to the temporary table) according to the user's configuration; execute these sql (that is, the query statement corresponding to the temporary table) in the background to determine which rules have not passed and return the failed rules. Using the querydsl framework to generate SQL solutions greatly reduces development costs, and there is no need to repeatedly develop and adapt to different types of databases. Automatic data slicing makes the verification data more targeted and accurate, reduces the amount of original data, and improves the verification speed, which is crucial for data in a single table of more than 10 million. The introduction of h2 for form data verification can meet the verification rules for data outside the database without repeatedly configuring additional rules, enriching the application scenarios of verification.

[0107] Figure 4 Schematic diagram of the main units of the data processing device according to an embodiment of the present application. Figure 4As shown, the data processing device 400 includes a receiving unit 401, a slicing unit 402, a query statement generating unit 403, a difference data determining unit 404 and an executing unit 405.

[0108] The receiving unit 401 is configured to receive a data processing request and obtain the data to be processed, a time mark and an organization mark.

[0109] The slicing unit 402 is configured to slice the data to be processed based on the time identifier and the organization identifier to obtain sliced ​​data.

[0110] The query statement generating unit 403 is configured to determine the data processing type, obtain the database identifier and configuration data, and call the target framework to generate a query statement corresponding to the database corresponding to the database identifier based on the data processing type, the database identifier and the configuration data.

[0111] The difference data determining unit 404 is configured to execute a query statement based on the slice data, obtain query result data, and determine the difference data in response to a difference between the query result data and the expected data.

[0112] The execution unit 405 is configured to generate a correction task based on the query result data, the expected data and the difference data, and call the thread pool to execute the correction task to obtain the correction result data.

[0113] In some embodiments, the slicing unit 402 is further configured to: determine the data source based on the organization identifier, and obtain target data corresponding to the data source from the data to be processed; and slice the target data based on the time identifier to obtain sliced ​​data.

[0114] In some embodiments, the query statement generating unit 403 is further configured to: in response to the existence of query result data, determine the data processing type as update; in response to the absence of query result data, determine the data processing type as query.

[0115] In some embodiments, the query statement generation unit 403 is further configured to: call the corresponding database statement generation template according to the data processing type; replace the corresponding placeholders in the database statement generation template with the database identifier, the threshold value and the target field name in the configuration data to obtain the query statement corresponding to the database corresponding to the database identifier.

[0116] In some embodiments, the data processing device further includes Figure 4 The data insertion unit not shown in the figure is configured to: in response to the data to be processed being form data, create a temporary table with the same structure as the form data; and insert the data to be processed into the temporary table.

[0117] In some embodiments, the query statement generation unit 403 is further configured to: call the corresponding database statement generation template according to the data processing type; obtain the identifier of the temporary table, and replace the corresponding placeholder in the database statement generation template with the identifier of the temporary table, the threshold in the configuration data, and the target field name to obtain the query statement corresponding to the temporary table.

[0118] It should be noted that the data processing method and the data processing device of the present application have a corresponding relationship in terms of specific implementation contents, so the repeated contents will not be described again.

[0119] Figure 5 An exemplary system architecture 500 to which the data processing method or data processing device according to the embodiment of the present application can be applied is shown.

[0120] like Figure 5 As shown, system architecture 500 may include terminal devices 501, 502, 503, a network 504 and a server 505. Network 504 is used to provide a medium for communication links between terminal devices 501, 502, 503 and server 505. Network 504 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0121] Users can use terminal devices 501, 502, 503 to interact with server 505 through network 504 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 501, 502, 503, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only for example).

[0122] The terminal devices 501 , 502 , and 503 may be various electronic devices having a data processing screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0123] Server 505 can be a server that provides various services, such as a background management server that supports data processing requests submitted by users using terminal devices 501, 502, and 503 (only as an example). The background management server can receive data processing requests, obtain data to be processed, time identifiers, and organization identifiers; slice the data to be processed based on the time identifier and organization identifier to obtain sliced ​​data; determine the data processing type, obtain the database identifier and configuration data, call the target framework to generate a query statement corresponding to the database corresponding to the database identifier based on the data processing type, database identifier, and configuration data; execute the query statement based on the sliced ​​data, obtain the query result data, and determine the difference data in response to the difference between the query result data and the expected data; generate a correction task based on the query result data, the expected data, and the difference data, and call the thread pool to execute the correction task to obtain the correction result data. Improve data processing efficiency and accuracy.

[0124] It should be noted that the data processing method provided in the embodiment of the present application is generally executed by the server 505 , and accordingly, the data processing device is generally set in the server 505 .

[0125] It should be understood that Figure 5 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0126] Reference below Figure 6 , which shows a schematic diagram of the structure of a computer system 600 of a terminal device suitable for implementing an embodiment of the present application. Figure 6 The terminal device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0127] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the computer system 600 are also stored. The CPU 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0128] The following components are connected to the I / O interface 605: an input section 606 including a keyboard, a mouse, etc.; an output section 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, a modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage section 608 as needed.

[0129] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the central processing unit (CPU) 601, the above-mentioned functions defined in the system of the present application are executed.

[0130] It should be noted that the computer-readable medium shown in the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. Computer-readable storage media may include, for example, but are not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0131] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the above-mentioned module, program segment or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0132] The units involved in the embodiments described in the present application may be implemented by software or by hardware. The units described may also be set in a processor, for example, may be described as: a processor includes a receiving unit, a slicing unit, a query statement generating unit, a difference data determining unit, and an execution unit. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0133] As another aspect, the present application also provides a computer-readable medium, which may be included in the device described in the above embodiment; or it may exist independently and not be assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device receives a data processing request, obtains the data to be processed, the time mark and the mechanism mark; slices the data to be processed based on the time mark and the mechanism mark to obtain sliced ​​data; determines the data processing type, obtains the database mark and configuration data, and calls the target framework to generate a query statement corresponding to the database corresponding to the database mark based on the data processing type, the database mark and the configuration data; executes the query statement based on the sliced ​​data, obtains the query result data, and determines the difference data in response to the difference between the query result data and the expected data; generates a correction task based on the query result data, the expected data and the difference data, and calls the thread pool to execute the correction task to obtain the correction result data.

[0134] The computer program product of the present application includes a computer program, and the computer program implements the data processing method in the embodiment of the present application when executed by a processor.

[0135] According to the technical solution of the embodiment of the present application, the data processing efficiency and accuracy are improved.

[0136] The above specific implementations do not constitute a limitation on the protection scope of this application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of this application should be included in the protection scope of this application.

Claims

1. A data processing method, characterized in that: include: Receive data processing requests, obtain data to be processed, time stamp and organization identification; Slicing the data to be processed based on the time identifier and the organization identifier to obtain sliced ​​data; Determine a data processing type, obtain a database identifier and configuration data, and call a target framework to generate a query statement corresponding to a database corresponding to the database identifier based on the data processing type, the database identifier and the configuration data; Execute the query statement based on the slice data to obtain query result data, and determine difference data in response to a difference between the query result data and expected data; Based on the query result data, the expected data and the difference data, a correction task is generated, and a thread pool is called to execute the correction task to obtain the correction result data.

2. The method according to claim 1, characterized in that The slicing of the data to be processed based on the time identifier and the organization identifier to obtain sliced ​​data includes: Determine a data source based on the organization identifier, and obtain target data corresponding to the data source from the data to be processed; Based on the time mark, the target data is sliced ​​to obtain sliced ​​data.

3. The method according to claim 1, characterized in that Determining the data processing type includes: In response to the existing query result data, determining that the data processing type is update; In response to the absence of query result data, the data processing type is determined to be query.

4. The method according to claim 1, characterized in that: The step of generating a query statement corresponding to the database corresponding to the database identifier includes: According to the data processing type, calling the corresponding database statement generation template; The corresponding placeholders in the database statement generation template are replaced with the database identifier, the threshold value in the configuration data, and the target field name to obtain a query statement corresponding to the database corresponding to the database identifier.

5. The method according to claim 1, characterized in that Before determining the data processing type, the method further includes: In response to the data to be processed being form data, creating a temporary table having the same structure as the form data; Insert the data to be processed into the temporary table.

6. The method according to claim 5, characterized in that After determining the data processing type, the method further includes: According to the data processing type, calling the corresponding database statement generation template; The identifier of the temporary table is obtained, and the corresponding placeholders in the database statement generation template are replaced with the identifier of the temporary table, the threshold value in the configuration data, and the target field name to obtain a query statement corresponding to the temporary table.

7. A data processing device, characterized in that: include: A receiving unit, configured to receive a data processing request, and obtain the data to be processed, a time mark, and an organization mark; a slicing unit, configured to slice the data to be processed based on the time identifier and the organization identifier to obtain sliced ​​data; A query statement generating unit is configured to determine a data processing type, obtain a database identifier and configuration data, and call a target framework to generate a query statement corresponding to a database corresponding to the database identifier based on the data processing type, the database identifier and the configuration data; a difference data determining unit, configured to execute the query statement based on the slice data, obtain query result data, and determine the difference data in response to a difference between the query result data and the expected data; The execution unit is configured to generate a correction task based on the query result data, the expected data and the difference data, and call a thread pool to execute the correction task to obtain the correction result data.

8. The device according to claim 7, characterized in that The slicing unit is further configured to: Determine a data source based on the organization identifier, and obtain target data corresponding to the data source from the data to be processed; Based on the time mark, the target data is sliced ​​to obtain sliced ​​data.

9. The device according to claim 7, characterized in that The query statement generating unit is further configured to: In response to the existing query result data, determining that the data processing type is update; In response to the absence of query result data, the data processing type is determined to be query.

10. The device according to claim 7, characterized in that The query statement generating unit is further configured to: According to the data processing type, calling the corresponding database statement generation template; The corresponding placeholders in the database statement generation template are replaced with the database identifier, the threshold value in the configuration data, and the target field name to obtain a query statement corresponding to the database corresponding to the database identifier.

11. The device according to claim 7, characterized in that The device further comprises a data inserting unit configured to: In response to the data to be processed being form data, creating a temporary table having the same structure as the form data; Insert the data to be processed into the temporary table.

12. The device according to claim 11, characterized in that The query statement generating unit is further configured to: According to the data processing type, calling the corresponding database statement generation template; The identifier of the temporary table is obtained, and the corresponding placeholders in the database statement generation template are replaced with the identifier of the temporary table, the threshold value in the configuration data, and the target field name to obtain a query statement corresponding to the temporary table.

13. A data processing electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

14. A computer readable medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images