A data processing method, apparatus, device, storage medium, and program product
By comparing the end-of-day processing files with the target original data in the database, the validity of the files was determined and the missing data was generated in reverse, thus solving the problem of database data loss caused by date switching and ensuring the accuracy and security of the simulation test.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- INDUSTRIAL AND COMMERCIAL BANK OF CHINA
- Filing Date
- 2025-08-28
- Publication Date
- 2026-07-31
AI Technical Summary
After the date switch, the discontinuous dates in the system cause the upstream system to generate empty or erroneous files, resulting in data loss or incomplete information in the database, which affects the verification of the business functions of the downstream system under simulation testing.
By obtaining the validity of the end-of-day processing file, the validity of the file is determined by comparing the target original data stored in the database with the end-of-day processing file. In case of anomalies, missing data is generated in reverse based on the global data model and data dictionary, the file is updated, and then fully loaded and stored.
This ensured the integrity of the database data during the simulation test, prevented data loss, and guaranteed the security and accuracy of the downstream system business function verification.
Smart Images
Figure CN122492336A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of big data, and in particular to a data processing method, apparatus, device, storage medium, and program product. Background Technology
[0002] In business scenario simulation testing, system functions are often verified after a date change. Since the testing environment may switch to a future date, it can lead to discontinuous dates.
[0003] When the upstream system generates data files, it generates them in batches at the end of the day according to the date. If the date is not changed continuously, the upstream system will generate empty files or error files and load the empty files or error files into the fully loaded database. This will result in the loss of data or incomplete information stored in the database, which will affect the verification of the business functions of the downstream system under simulation test. Summary of the Invention
[0004] This invention provides a data processing method, apparatus, device, storage medium, and program product, relating to the field of big data and applicable to the fintech field. The method enables the reverse generation and updating of files with invalidity after determining their validity, followed by full loading and storage, ensuring the security and accuracy of downstream system business function verification during simulation testing.
[0005] According to one aspect of the present invention, a data processing method is provided, the method comprising:
[0006] Obtain the end-of-day processing file sent by the business processing unit, and determine the validity of the end-of-day processing file based on the file primary key in the end-of-day processing file and the target original data stored in the database.
[0007] When the validity of the end-of-day processing file is abnormal, the global data model is determined based on the field information of the end-of-day processing file and the historical business association table stored in the database.
[0008] Based on the data dictionary and global data model, update the end-of-day processing files with abnormal validity, and update the historical business association table based on the updated end-of-day processing files.
[0009] The data processing method provided in this invention, after receiving the end-of-day processing file sent by the business processing unit, compares the target original data stored in the database with the end-of-day processing file to determine the validity of the end-of-day processing file. This allows for a preliminary judgment of the end-of-day processing file, preventing the database from directly storing abnormal files and overwriting the existing data in the end-of-day processing file due to "skipping days," thus avoiding the loss of original data in the database. If the validity of the end-of-day processing file is determined to be abnormal, a global data model can be determined based on the field information of the end-of-day processing file and the historical business association table. Based on the global data model and the data dictionary, missing data entries in the end-of-day processing file are generated in reverse to update the end-of-day processing file. Finally, the updated end-of-day processing file is used to fully load and store multiple historical business association tables in the database. In other words, the above steps solve the problem that the current issue of directly loading problematic files from the upstream system into the database in full due to date switching, resulting in data loss or incomplete information stored in the database, is resolved by reversing the generation and updating of files with abnormal validity after determining the validity of the downloaded files, and then loading and storing them in full. This ensures the security and accuracy of verifying the business functions of the downstream system under simulation testing.
[0010] According to another aspect of the present invention, a data processing apparatus is provided, the apparatus comprising:
[0011] The file loading module is used to obtain the end-of-day processing file sent by the business processing unit, and determine the validity of the end-of-day processing file based on the file primary key in the end-of-day processing file and the target original data stored in the database.
[0012] The data generation module is used to determine a global data model based on the field information of the end-of-day processing file and the historical business association table stored in the database when the validity of the end-of-day processing file is abnormal; update the end-of-day processing file with abnormal validity based on the data dictionary and the global data model; and update the historical business association table based on the updated end-of-day processing file.
[0013] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:
[0014] At least one processor; and
[0015] A memory that is communicatively connected to at least one processor; wherein,
[0016] The memory stores a computer program that can be executed by at least one processor, such that the at least one processor is able to perform the data processing method of any embodiment of the present invention.
[0017] According to another aspect of the present invention, a computer-readable storage medium is provided that stores computer instructions for causing a processor to execute and implement the data processing method of any embodiment of the present invention.
[0018] According to another aspect of the present invention, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the data processing method of any embodiment of the present invention.
[0019] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description
[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0021] Figure 1 A flowchart illustrating a data processing method provided in an embodiment of the present invention;
[0022] Figure 2 Another flowchart illustrating the data processing method provided in an embodiment of the present invention;
[0023] Figure 3 This is a schematic diagram of the structure of the data processing device provided in an embodiment of the present invention;
[0024] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation
[0025] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.
[0026] It should be noted that the terms "original," "target," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.
[0027] Figure 1 This is a flowchart illustrating a data processing method provided in an embodiment of the present invention. This embodiment is applicable to simulation tests of date switching, ensuring that even if dates are not consecutive, data stored in the database will not be lost or incomplete. The method can be executed by a data processing device, which can be implemented in hardware and / or software. This data processing device can be configured in an electronic device, such as a computer or terminal device. In this embodiment, the electronic device containing the data processing device can serve as a downstream system of the upstream system. Figure 1 As shown, the method includes:
[0028] S101. Obtain the end-of-day processing file sent by the business processing unit, and determine the validity of the end-of-day processing file based on the file primary key in the end-of-day processing file and the target original data stored in the database.
[0029] In this embodiment, the business processing unit is the upstream system, which can optionally be a corporate e-banking system. The end-of-day processing file is the processing file automatically executed before the end of each day, performing operations such as summarizing, clearing, and verifying the day's business data. The file primary key is one or more fields used to uniquely identify each record in the end-of-day processing file (e.g., uniquely identifying a record in the file). The database is the library mentioned in this embodiment used to load and store the files transmitted from the upstream system. The target raw data is the actual data obtained on the same date as the end-of-day processing file obtained in the current simulation test. Validity is used to determine whether the end-of-day processing file is valid or abnormal; abnormal files include empty files or erroneous files.
[0030] Specifically, in business scenario simulation testing, during end-of-day batch processing on an actual date, the system date of the business processing unit can be switched. The business processing unit will then generate an end-of-day processing file corresponding to the switched date and download the end-of-day processing file to the data processing device. At this time, the data processing device can compare the primary key in the end-of-day processing file with the target original data stored in the database that has the same date as the end-of-day processing file, thus determining the validity of the end-of-day processing file.
[0031] For example, assuming the current actual date is August 22nd, a simulation test is conducted at the end of August 22nd, switching the system date of the business processing unit to August 25th. This causes the business processing unit to generate end-of-day processing files for each day before August 25th, i.e., August 24th and earlier. In existing technology, because the date is directly switched to the 25th, meaning the 23rd and 24th haven't actually occurred, the business processing unit cannot execute the complete end-of-day batch processing flow. Therefore, the business processing unit can only generate empty or erroneous files. Furthermore, after the business processing unit downloads empty or erroneous files to the fully loaded database, it overwrites the original correct data and its corresponding data relationship tables. Based on this, in this embodiment, when the end-of-day processing file sent by the business processing unit after the date switch is received, the difference between the target original data stored in the database and the end-of-day processing file can be compared first, and the validity of the end-of-day processing file can be determined based on this difference.
[0032] Furthermore, the storage of files transmitted by the daily business processing units in the database is divided into multiple business association tables based on the different types or categories of data in the files (taking banks as an example, business association tables include tables such as the online banking interbank name and code table, the online banking interbank code table, the amount limit parameter table, and the interbank name and code table, etc.). Each entry in the transmitted file is stored in the corresponding business association table according to its data type or category. Therefore, after switching dates and obtaining the end-of-day processing file transmitted by the business processing unit, the validity of the end-of-day processing file can be determined by comparing each entry in the end-of-day processing file with the target original data in these aforementioned business association tables.
[0033] In this embodiment, after receiving the end-of-day processing file sent by the business processing unit, the target original data stored in the database is compared with the end-of-day processing file to determine the validity of the end-of-day processing file. This can achieve a preliminary judgment on the end-of-day processing file and prevent the database from directly storing abnormal files when the end-of-day processing file is abnormal due to "skipping days", thus avoiding the problem of losing the original data in the database.
[0034] S102. When the validity of the end-of-day processing file is abnormal, determine the global data model based on the field information of the end-of-day processing file and the historical business association table stored in the database.
[0035] The field information refers to the fields of each entry in the end-of-day processing file, or the filename, file path, file header, or even scheduling instruction information of the end-of-day processing file. The historical business association table, as mentioned in this embodiment, is the updated business association table obtained after the database stores the historical files transmitted by the business processing unit. The global data model is used to represent the relationships between all data in the database and the rules that the data must follow.
[0036] Specifically, if the validity of the end-of-day processing file is determined to be abnormal, it can be determined that the end-of-day processing file is empty or an erroneous file, such as missing some entries. At this time, based on the field information of the end-of-day processing file and the historical business relationship tables stored in the database, the target business relationship tables and target data required for the current valid end-of-day processing file can be determined first. Then, the target data and target business relationship tables are integrated to generate a global data model to guide the subsequent generation of valid end-of-day processing files.
[0037] S103. Based on the data dictionary and global data model, update the end-of-day processing files with abnormal validity, and update the historical business association table based on the updated end-of-day processing files.
[0038] The data dictionary is used to describe and explain the source and rules of business data, that is, to store the metadata of business data.
[0039] Specifically, after generating the global data model, the primary key can be extracted from the data dictionary. Based on the global data model, data for missing entries is generated according to field types and business logic requirements. This missing data is then added to the end-of-day processing file, which is considered invalid, thus updating the end-of-day processing file. Furthermore, based on the data type of each entry in the updated end-of-day processing file, each entry is updated to its corresponding historical business relationship table, achieving a full database load and data storage.
[0040] Optionally, once the end-of-day processing file is valid, or after updating the historical business association table, a loading completion instruction can be sent to the business processing unit so that the business processing unit can provide data services to the user.
[0041] In this embodiment, if the validity of the end-of-day processing file is determined to be abnormal, a global data model can be determined based on the field information of the end-of-day processing file and the historical business association table. Then, based on the global data model and the data dictionary, missing data entries in the end-of-day processing file are generated in reverse to update the file. Finally, the updated end-of-day processing file is used to fully load and store multiple historical business association tables in the database. In other words, the above steps solve the problem of data loss or incomplete information in the database caused by directly loading problematic files from the upstream system into the database when the date is changed. This approach achieves the following: after determining the validity of the downloaded file, files with abnormal validity are reverse-generated and updated before being fully loaded and stored, ensuring the security and accuracy of downstream system business function verification during simulation testing.
[0042] It is worth noting that the terms "upstream system" and "downstream system" mentioned in this embodiment are relative concepts and not intended to define a specific system. The information collected in this embodiment is authorized by the user or fully authorized by all parties. The collection, storage, use, processing, transmission, provision, disclosure, and application of the relevant data all comply with the relevant laws, regulations, and standards of the relevant countries and regions, and necessary confidentiality measures have been taken. The data does not violate public order and good morals, and corresponding operation entry points are provided for users to choose to authorize or refuse.
[0043] The data processing method provided in this invention, after receiving the end-of-day processing file sent by the business processing unit, compares the target original data stored in the database with the end-of-day processing file to determine the validity of the end-of-day processing file. This allows for a preliminary judgment of the end-of-day processing file, preventing the database from directly storing abnormal files and overwriting the existing data in the end-of-day processing file due to "skipping days," thus avoiding the loss of original data in the database. If the validity of the end-of-day processing file is determined to be abnormal, a global data model can be determined based on the field information of the end-of-day processing file and the historical business association table. Based on the global data model and the data dictionary, missing data entries in the end-of-day processing file are generated in reverse to update the end-of-day processing file. Finally, the updated end-of-day processing file is used to fully load and store multiple historical business association tables in the database. In other words, the above steps solve the problem that the current issue of directly loading problematic files from the upstream system into the database in full due to date switching, resulting in data loss or incomplete information stored in the database, is resolved by reversing the generation and updating of files with abnormal validity after determining the validity of the downloaded files, and then loading and storing them in full. This ensures the security and accuracy of verifying the business functions of the downstream system under simulation testing.
[0044] Figure 2This is another flowchart illustrating the data processing method provided in this embodiment of the invention. Based on the above embodiments and other examples, this embodiment mainly details the steps of determining the global data model and updating the end-of-day processing file. Figure 2 As shown, the method includes:
[0045] S201. Obtain the end-of-day processing file sent by the business processing unit.
[0046] Specifically, during the date-switching environment test, if the system date of the business processing unit is switched and end-of-day batch processing is performed, the business processing unit will generate an end-of-day processing file and download it to the data processing device. At this point, the end-of-day processing file is the batch file used for testing in the date-switching environment test.
[0047] S202. Using the preset identification information in the end-of-day processing file as the primary key, the original data stored in the database containing the same field as the preset identification information is used as the target original data.
[0048] In this context, the primary key is a unique identifier used to represent each row of records in the data table. In this embodiment, the original data is data from the historical business association table in the database. The preset identifier information can be determined by the user based on the file type of the end-of-day processing file downloaded during this environment test. For example, if the end-of-day processing file type is the end-of-day situation of different branches of a bank, then the preset identifier information can be the branch name or branch number.
[0049] Specifically, after the upstream system downloads the end-of-day processing file, the database determines which business association table it should correspond to based on the field names in the downloaded file. Therefore, the field names in the downloaded file, i.e., the end-of-day processing file, can be used as the primary key. For example, if the field in the end-of-day processing file is a row name or row number, then the row name or row number can be used as the primary key. The row name and row number table associated with the end-of-day processing file in the database can be selected as a reference. The historical row name and row number table currently stored in the database (because the database is fully loaded, it will only store the table obtained in the previous actual situation or the table obtained in the previous simulation test) can be used as the target business association table. The original data in the table that matches the row name or row number in the end-of-day processing file can be selected as the target original data.
[0050] S203. Determine the quantity threshold based on the number of original entries in the target raw data, and determine the quantity difference based on the number of entries in the end-of-day processing file and the number of original entries.
[0051] The quantity threshold is a user-defined threshold used to limit the number of entries. The original entry count is used to characterize the number of entries in the target raw data.
[0052] Specifically, the quantity threshold is determined based on the number of original entries in the target raw data, including:
[0053] (i) Determine the optional limits based on the processing date of the end-of-day processing documents.
[0054] The processing date refers to the date corresponding to the end of the day after this date cutoff, under the simulation test.
[0055] Specifically, since the simulation test on specific days aims to predict the system's capabilities under special conditions, such as holidays or peak shopping days like "Double Eleven," the system's data processing capacity can be assessed based on the adaptability of the processing dates. This provides a more reasonable threshold for determining the validity of the end-of-day processing files. Optionally, the optional limit is generally 5%, which can be adjusted based on the amount of data processed on the chosen date.
[0056] (ii) Determine the quantity threshold based on the original number of entries and the optional limit.
[0057] Specifically, the quantity threshold can be determined by multiplying the original number of entries by the optional limit.
[0058] Additionally, the quantity difference can be determined based on the number of entries at the end of the day and the original number of entries in the end-of-day processing file. For example, the quantity difference can be obtained by subtracting the original number of entries from the number of entries at the end of the day and taking the absolute value.
[0059] In this embodiment, the optional limits are determined based on the processing date of the end-of-day processing file. The optional limits can be dynamically adjusted according to the specific data volume of the cut-off date required for this simulation test, ensuring that the optional limits are close to the actual date situation. This provides an accurate and environmentally adaptable limit guarantee for the subsequent determination of quantity limits.
[0060] S204. If the difference in quantity is greater than the quantity threshold, the validity of the end-of-day processed files is determined to be abnormal.
[0061] Specifically, the quantity difference is compared with the quantity threshold. If the quantity difference is greater than the quantity threshold, it indicates that multiple entries are missing in the end-of-day processing file or that the end-of-day processing file is empty. Therefore, it can be determined that the validity of the end-of-day processing file is abnormal.
[0062] S205. When the validity of the end-of-day processing file is abnormal, determine the historical business data that is consistent with the field information of the end-of-day processing file based on the field information of the end-of-day processing file and the historical business association table, and use the historical business data as the target data.
[0063] The field information refers to the field names, which are the names of each column in the table.
[0064] Specifically, when the validity of the end-of-day processing file is determined to be abnormal, each historical business association table can be traversed based on the field names of each group in the end-of-day processing file to determine the historical business data in each historical business association table that contains the same field names as each group's field names, and the historical business data can be used as the target data.
[0065] S206. Use the historical business association table where the target data is located as the target business association table.
[0066] Specifically, the historical business association table containing the target data can also be used as the target business association table. In this embodiment, the target data is the data that is downloaded from the upstream system and stored in the downstream system after end-of-day processing under normal circumstances. At this time, the target data is the data that should have been included in the end-of-day processing file downloaded from the upstream system during this environmental test.
[0067] Optionally, after determining the target data and the target business association table, the data can be processed, such as data cleaning, transformation, and formatting, to provide data that meets the file requirements for subsequent regeneration or updating of the end-of-day processing file.
[0068] In this embodiment, by traversing each historical business association table based on field information, the target data and the target business association table are obtained, and the field definition of the final end-of-day processing file that needs to be generated or updated can be obtained.
[0069] S207. Determine the global table based on the structured query statement and the target data and the target business association table.
[0070] Structured query statements are used to filter and combine the target data required from the target business association table.
[0071] Specifically, based on structured query statements, target data is filtered out from at least one target business association table, and then the target business association tables containing this target data are combined according to the required combination method using join operations to obtain the global table.
[0072] S208. Perform reverse generation processing on the global table to obtain the global data model.
[0073] The reverse generation process is used to automatically read and visualize the global table at the metadata level.
[0074] Specifically, the global table is reverse-engineered to obtain a global data model, including:
[0075] (a) Perform structuring and visualization processing on the global table to obtain the global graph.
[0076] The global graph is used to represent the data in the global table, the attributes of the data, and the relationships between different data.
[0077] Specifically, the global table is structured and visualized. For example, the metadata of the global table (such as tables, fields, keys, and constraints) is parsed, and the parsed information is used to draw an entity relationship diagram to obtain the global diagram.
[0078] (ii) Perform reverse generation processing on the global graph to obtain the global data model.
[0079] The global data model is used to display information such as the tablespace, indexes, partitions, and data types of the global table.
[0080] Specifically, by performing reverse generation on the global graph, a structured document (such as a list of tables, field definitions, relationship descriptions, and business rules) can be obtained. Finally, the structured document is presented in the form of a physical data model, which yields the global data model.
[0081] In this embodiment, a global graph is first obtained based on a global table, and a global data model is then obtained based on the global graph. This provides information such as the relationships between target data in the target business association table, and provides a framework and rules for regenerating each entry in the end-of-day processing file.
[0082] S209. Determine the field type based on the global data model and the end-of-day processing file with abnormal validity.
[0083] In this embodiment, the field type is used to determine whether the data is technical or business-related. For example, field types can be divided into technical field types and business field types; technical field types are used to define the nature and format of the data stored in the file, such as numeric, character, date and time types, etc.; business field types are based on technical types, and also include specific business meanings and rules, such as amount, date, name, status flag, percentage, etc.
[0084] Specifically, based on the global data model, all target business related tables can be traversed, and data that is in the target business related tables but not in the end-of-day processing file can be regarded as missing data, and the field type corresponding to the missing data can be determined.
[0085] S210. Generate missing entries based on field type, the target business association table corresponding to the field type, and the data dictionary.
[0086] Specifically, the primary key can be extracted from the data dictionary. Based on the data model, each field is traversed. During the traversal, the corresponding target business association table is determined according to the field type, and each field in the target business association table corresponding to the field type is obtained as the table structure; for example, the field name, data type, default value, and business logic constraints are obtained. At this point, missing entries can be generated based on the list of primary keys for missing data and the data dictionary. For example, when traversing each missing primary key, if the missing primary key field has a business logic constraint, a value can be directly assigned based on the primary key and the target business association table to generate a missing entry; if the missing primary key field has no business logic constraint, the default value defined in the data dictionary is directly used as the data in the generated missing entry.
[0087] In this embodiment, by utilizing a global data model, a data dictionary, and the types of fields to be filled, the possible data structures of the target business association table and the data dictionary are learned to generate missing entries. This approach ensures the business compliance of the generated values based on the data dictionary and the correctness of the association logic of the generated values based on the data model. It achieves data that highly simulates reality and conforms to all business rules, reduces the manual cost of filling in missing data, and improves the accuracy of generating missing data.
[0088] S211. Update the end-of-day processing file with invalidation based on missing entries, and determine the data in the updated end-of-day processing file to be filled into the corresponding historical business association table.
[0089] Specifically, following the steps described above, missing entries in the abnormal end-of-day processing files can be generated one by one. Therefore, after obtaining all entries or generating each entry, the missing entries can be updated in the end-of-day processing files with invalidity. Furthermore, after the end-of-day processing files are updated, the data in the updated files can be used to fill the corresponding historical business association tables. For example, based on the filename or fields of the end-of-day processing file, it can be determined which historical business association table the data belongs to, and the original data in that historical business association table is deleted. The data from the updated end-of-day processing file is then filled into that historical business association table, thus updating the historical business association table—that is, achieving a full load of data.
[0090] S212. If the quantity difference is less than or equal to the quantity threshold, the validity of the end-of-day processed document is determined to be valid.
[0091] Specifically, the quantity difference is compared with the quantity threshold. If the quantity difference is less than or equal to the quantity threshold, it means that the validity of the end-of-day processing file is normal, and S213 can be executed directly.
[0092] S213. Send a loading completion instruction to the service processing unit so that the service processing unit can provide data services to the user.
[0093] In this embodiment, S213 can be executed after S211 or S212 has been executed.
[0094] Specifically, after confirming the validity of the end-of-day processing file or updating the historical business association table, a loading completion command can be sent to the business processing unit to inform it that the simulation test is complete. At this point, the business processing unit can then normally provide users with relevant business data services, such as online business services.
[0095] This embodiment addresses the issue of incomplete business data when processing batch data with discontinuous dates, where empty or abnormal files are generated instead of incremental data loading, leading to the overwriting of business relationship tables by empty files and incomplete business data information. By automatically learning the table structure through the field types and data types of the business data, combined with a data dictionary and historical business relationship tables, it generates data table entries to complete the missing data information. This technical solution significantly improves the correctness of data loading in simulated testing environments, reduces data loss caused by full loading, and minimizes the impact on subsequent verification of business logic. Furthermore, it replaces the time spent on manual data completion, providing a preferred method for correctly loading data in simulated testing environments.
[0096] Figure 3 This is a schematic diagram of the structure of a data processing apparatus provided in an embodiment of the present invention. Figure 3 As shown, the device includes:
[0097] The file loading module 301 is used to obtain the end-of-day processing file sent by the business processing unit, and determine the validity of the end-of-day processing file based on the file primary key in the end-of-day processing file and the target original data stored in the database.
[0098] The data generation module 302 is used to determine the global data model based on the field information of the end-of-day processing file and the historical business association table stored in the database when the validity of the end-of-day processing file is abnormal; update the end-of-day processing file with abnormal validity based on the data dictionary and the global data model; and update the historical business association table based on the updated end-of-day processing file.
[0099] Optionally, the validity of the end-of-day processing file is determined based on the file primary key in the end-of-day processing file and the target raw data stored in the database. Specifically, the file loading module 301 is used for:
[0100] Using the preset identifier information in the end-of-day processing file as the primary key, the original data stored in the database containing the same field as the preset identifier information is used as the target original data. A quantity threshold is determined based on the number of original entries in the target original data, and a quantity difference is determined based on the number of end-of-day entries in the end-of-day processing file and the number of original entries. If the quantity difference is less than or equal to the quantity threshold, the end-of-day processing file is determined to be valid; if the quantity difference is greater than the quantity threshold, the end-of-day processing file is determined to be invalid.
[0101] Optionally, a quantity threshold is determined based on the number of original entries in the target raw data. Specifically, the file loading module 301 is used for:
[0102] The optional limits are determined based on the processing date of the end-of-day processing documents; the quantity threshold is determined based on the original number of entries and the optional limits.
[0103] Optionally, based on the field information of the end-of-day processing file and the historical business relationship table stored in the database, a global data model is determined. The data generation module 302 is specifically used for:
[0104] Based on the field information of the end-of-day processing file and the historical business association table, historical business data that is consistent with the field information of the end-of-day processing file is identified and used as target data; the historical business association table containing the target data is used as the target business association table; based on the structured query statement and according to the target data and the target business association table, a global table is determined; the global table is then reverse-engineered to obtain the global data model.
[0105] Optionally, the global table is reverse-engineered to obtain a global data model. The data generation module 302 is specifically used for:
[0106] The global table is structured and visualized to obtain a global graph; the global graph is then reverse-engineered to obtain a global data model.
[0107] Optionally, based on the data dictionary and the global data model, update the end-of-day processing files with abnormal validity, and update the historical business relationship table based on the updated end-of-day processing files. The data generation module 302 is specifically used for:
[0108] The field types are determined based on the global data model and the end-of-day processing files with invalidity. Missing entries are generated based on the field types, the target business association tables corresponding to the field types, and the data dictionary. The end-of-day processing files with invalidity are updated based on the missing entries, and the data in the end-of-day processing files is filled into the corresponding historical business association tables based on the updated end-of-day processing files.
[0109] Optionally, the device further includes a transmitting module, which is specifically used for:
[0110] After the end-of-day processing file is deemed valid or the historical business association table is updated, a load completion instruction is sent to the business processing unit so that the business processing unit can provide data services to the user.
[0111] The data processing apparatus provided in the embodiments of the present invention can execute the data processing method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the method.
[0112] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.
[0113] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 may also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.
[0114] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.
[0115] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as data processing methods.
[0116] In some embodiments, the data processing method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the data processing method by any other suitable means (e.g., by means of firmware).
[0117] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.
[0118] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.
[0119] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.
[0120] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).
[0121] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.
[0122] A computing system can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.
[0123] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the data processing method provided in any embodiment of this invention.
[0124] In implementing the computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages or a combination thereof. Programming languages include object-oriented programming languages such as Java, Smalltalk, and C++, as well as conventional procedural programming languages such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0125] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.
[0126] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.
Claims
1. A data processing method, characterized by, The method includes: Obtain the end-of-day processing file sent by the business processing unit, and determine the validity of the end-of-day processing file based on the file primary key in the end-of-day processing file and the target original data stored in the database; When the validity of the end-of-day processing file is abnormal, the global data model is determined based on the field information of the end-of-day processing file and the historical business association table stored in the database. Based on the data dictionary and the global data model, update the end-of-day processing files with abnormal validity, and update the historical business association table based on the updated end-of-day processing files.
2. The method of claim 1, wherein, The step of determining the validity of the end-of-day processing file based on the file primary key in the end-of-day processing file and the target raw data stored in the database includes: Using the preset identifier information in the end-of-day processing file as the primary key, the original data stored in the database containing the same field as the preset identifier information is used as the target original data; A quantity threshold is determined based on the number of original entries in the target raw data, and a quantity difference is determined based on the number of end-of-day entries in the end-of-day processed file and the number of original entries. If the quantity difference is less than or equal to the quantity threshold, then the validity of the end-of-day processing file is determined to be valid; If the difference in quantity is greater than the quantity threshold, then the validity of the end-of-day processing file is determined to be abnormal.
3. The method of claim 2, wherein, The step of determining the quantity threshold based on the number of original entries in the target raw data includes: The optional limits are determined based on the processing date of the end-of-day processing file; The quantity threshold is determined based on the original number of entries and the optional limit.
4. The method of claim 2, wherein, The step of determining the global data model based on the field information of the end-of-day processing file and the historical business association table stored in the database includes: Based on the field information of the end-of-day processing file and the historical business association table, determine the historical business data that is consistent with the field information of the end-of-day processing file, and use the historical business data as the target data; Use the historical business association table where the target data is located as the target business association table; The global table is determined based on the structured query statement and the target data and the target business association table. The global table is reverse-engineered to obtain the global data model.
5. The method of claim 4, wherein, The process of reverse-generating the global table to obtain the global data model includes: The global table is structured and visualized to obtain a global graph; The global graph is reverse-engineered to obtain the global data model.
6. The method of claim 4, wherein, The step of updating end-of-day processing files with abnormal validity based on the data dictionary and the global data model, and updating the historical business association table based on the updated end-of-day processing files, includes: The field types are determined based on the global data model and the end-of-day processing files with abnormal validity. Based on the field type, the target business association table corresponding to the field type, and the data dictionary, generate missing entries; Update the end-of-day processing file with invalidity based on the missing entries, and determine the data in the updated end-of-day processing file to be filled into the corresponding historical business association table.
7. The method of claim 1, wherein, The method further includes: After the end-of-day processing file is deemed valid or the historical business association table is updated, a loading completion instruction is sent to the business processing unit so that the business processing unit can provide data services to the user.
8. A data processing apparatus, characterized by, The device includes: The file loading module is used to obtain the end-of-day processing file sent by the business processing unit, and determine the validity of the end-of-day processing file based on the file primary key in the end-of-day processing file and the target original data stored in the database. The data generation module is used to determine a global data model based on the field information of the end-of-day processing file and the historical business association table stored in the database when the validity of the end-of-day processing file is abnormal; update the end-of-day processing file with abnormal validity based on the data dictionary and the global data model; and update the historical business association table based on the updated end-of-day processing file.
9. An electronic device, comprising: include: One or more processors; Memory, used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the data processing method as described in any one of claims 1 to 7.
10. A readable storage medium, having stored thereon a computer program, characterized in that, When the program is executed by the processor, it implements the data processing method as described in any one of claims 1 to 7.
11. A computer program product, characterised in that, It includes a computer program that, when executed by a processor, implements the data processing method as described in any one of claims 1 to 7.