Data exporting method and device, equipment and medium

Abstract

The invention relates to the technical field of computers, in particular to a data exporting method and device, equipment and a medium, and aims to improve the stability and accuracy of data exporting. The method comprises the steps of obtaining to-be-exported target data from an original data table based on a preset export condition, and constructing a read-only copy table; reading first paging data from the auxiliary book table according to a preset paging rule, and distributing a paging identifier; according to the sequence of the paging identifiers, sequentially writing the first paging data into a first export file; after the first export file is exported, checking each piece of first paging data in the secondary table with corresponding data in the original data table, and taking at least one piece of first paging data with inconsistent data as abnormal paging data; and on the basis of the paging identifier of the abnormal paging data, obtaining corresponding second paging data from the original data table again, replacing the corresponding first paging data in the first export file with the second paging data, and generating a second export file.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a data export method, apparatus, device, and medium. Background Technology

[0002] In existing data export scenarios, exporting data from large-scale tables typically requires completing the reading of a large amount of data and file generation operations within a short period of time. Some systems still use a method of querying and exporting all data at once. This method can easily lead to excessive application memory usage and even memory overflow when dealing with large amounts of data, affecting the stable operation of the system.

[0003] On the other hand, in real-world business environments, database data is typically in a state of continuous updating. During the data export process, the original data tables may undergo addition, deletion, or modification operations, leading to inconsistencies between the exported results and the real-time data status in the database. This can result in issues such as data omissions, duplications, or inaccuracies, affecting the reliability of the exported data and subsequent business processing.

[0004] Therefore, how to reduce system resource consumption during the export process while ensuring data export efficiency, and effectively address the inconsistency risks caused by data changes during the export process, so as to improve the stability and accuracy of data export, has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0005] This application provides a data export method, apparatus, device, and medium to improve the stability and accuracy of data export.

[0006] Firstly, this application provides a data export method, the method comprising: Based on preset export conditions, the target data to be exported is obtained from the original data table, and a read-only copy table is constructed based on the target data; The first page data is read from the copy table according to the preset pagination rules, and a corresponding pagination identifier is assigned to each first page data; the first page data is written to the first exported file in the order of the pagination identifiers. After the first exported file is exported, the first page data in the copy table is compared with the corresponding data in the original data table to obtain each verification result; if there is at least one first page data with inconsistent data in each verification result, the at least one first page data is regarded as abnormal page data. For each abnormal pagination data, based on the pagination identifier of the abnormal pagination data, the corresponding second pagination data is retrieved from the original data table again, and the second pagination data is used to replace the corresponding first pagination data in the first exported file to generate a second exported file.

[0007] Through the above implementation methods, this application constructs a read-only copy table based on preset export conditions at the beginning of the data export stage. This ensures that the data set upon which the export process depends remains stable throughout the entire export cycle, avoiding inconsistencies in pagination results caused by data additions, modifications, or deletions due to direct access to the original data table during pagination export, thus preventing missing or duplicate pagination data. Simultaneously, by sequentially reading pagination data from the read-only copy table according to pagination rules and writing it to the export file, loading all data at once is avoided, effectively reducing the consumption of system memory and database resources during the export process and improving the stability of long-term, large-volume data export. After export is complete, by comparing the pagination data in the copy table with the corresponding data in the original data table, abnormal pagination data where the original data has changed during the export can be identified, thereby discovering potential inconsistencies without affecting the continuity of the export process. Furthermore, only for the identified abnormal pagination data, the corresponding data is retrieved from the original data table again and the export file is replaced, avoiding duplicate export of all data. This ensures export efficiency while improving the consistency and accuracy of the export results with the original data.

[0008] Therefore, the data export method of this application can reduce system resource consumption while ensuring data export efficiency, and effectively deal with inconsistencies caused by data changes during the export process, thereby improving the stability and reliability of data export.

[0009] In one possible embodiment, the method includes: For each first page of data in the replica table, perform the following operations: According to the preset feature generation rules, perform aggregation operations on at least one data field in the first paginated data to generate corresponding pagination feature values; construct corresponding pagination feature representations based on the pagination feature values. According to the same feature generation rules as the first pagination data, the original pagination feature representation generated for the target data is obtained; the target data refers to data in the original data table that has the same pagination identifier range as the first pagination data. The pagination feature representation is compared with the original pagination feature representation to obtain the verification result corresponding to the first pagination data.

[0010] Through the above implementation methods, this application performs aggregation operations on at least one data field in the paginated data and generates pagination feature values, abstracting the paginated data into a feature representation that can characterize its overall state. This avoids comparing each data record within a page one by one during the verification process. Simultaneously, by generating original pagination feature representations for data with the same pagination identifier range in the original data table according to the same feature generation rules and comparing the two, it is possible to quickly determine whether the overall paginated data has changed at the page level, thereby effectively reducing the computational complexity and system resource consumption of the verification process. Furthermore, only when the pagination feature representations are inconsistent are the corresponding paginated data determined to be abnormal, giving subsequent data compensation processing a clear objective and avoiding unnecessary data processing operations. This ensures the accuracy of data consistency verification while improving verification efficiency and system stability in large-scale data export scenarios.

[0011] In one possible embodiment, the method includes: Select at least one data field from the first paginated data; Perform field-level aggregation operations on multiple data records corresponding to at least one data field in the first pagination data to obtain the field aggregation result of the at least one data field; The field aggregation results of the at least one data field are subjected to feature mapping processing to generate pagination feature values ​​corresponding to the first pagination data.

[0012] Through the above implementation method, this application selects at least one data field from the paginated data and performs field-level aggregation operations on multiple data records corresponding to that field. This compresses multiple data records within a page into a representative field aggregation result. Then, by performing feature mapping processing on the field aggregation result, pagination feature values ​​are generated, thereby abstracting the overall state of an entire page of data into a single feature representation. Based on this pagination feature value, the overall consistency of the paginated data can be verified, avoiding the need to compare each data record within a page one by one during the verification process. This effectively reduces the computational complexity and system resource consumption of the verification process. At the same time, it can still quickly identify whether the paginated data has changed in scenarios with large amounts of data and long export times, improving data verification efficiency and ensuring the accuracy and stability of the data export results.

[0013] In one possible embodiment, the method includes: Based on the order of the pagination identifiers, the corresponding first pagination data is read sequentially; After reading each first page of data, the currently read first page of data is written as a write unit to the first exported file; After completing the writing of the first page of data currently being read, continue reading and writing the next page of data until all page data has been written.

[0014] Through the above implementation method, this application reads page data sequentially according to the page identifier order, and writes each page data as an independent writing unit sequentially to the export file. This allows the data export process to proceed step by step at the page level, avoiding reading and writing large amounts of data at once, thereby reducing the instantaneous occupation of system memory and resources during the export process. At the same time, the next page data is processed only after the current page data is written, giving the export process clear processing boundaries. This is beneficial for maintaining the continuity and controllability of the export process in scenarios with large amounts of data and long export times, thereby improving the stability of data export and reducing the impact on system operation.

[0015] In one possible embodiment, the method includes: Determine if the first exported file exists; If the first exported file does not exist, the first exported file is created, and the currently read first page data is written into the first exported file; If the first exported file already exists, the currently read first page data is appended to the first exported file.

[0016] Through the above implementation methods, this application determines the existence status of the export file before writing the paginated data to the export file, and creates a new export file if the export file does not exist, and writes the current paginated data in an append-only manner if the export file already exists. This ensures that the paginated data can be written to the same export file continuously and orderly, avoiding the loss of exported data due to repeated file creation or overwriting. At the same time, this writing method allows the data export process to continuously expand the export file as the paginated data is processed step by step, which is beneficial for maintaining the integrity and consistency of the export results in scenarios with large amounts of data and long-term export, thereby improving the reliability and stability of the data export process.

[0017] In one possible embodiment, the method includes: According to the preset data organization rules, the multiple data records in the currently read first page data are arranged sequentially; Write the sorted data records into the first exported file in sequence, and write delimiters between adjacent data records.

[0018] Through the above implementation method, when writing paginated data to the export file, this application first arranges multiple data records within the pagination in sequence according to preset data organization rules, and then writes them to the export file in sequence, with delimiters written between adjacent data records. This makes the storage structure of paginated data in the export file clear and the boundaries well-defined, facilitating subsequent processing of the export file. At the same time, this data organization method ensures the consistency of the order of data records within the pagination, avoiding the impact on the readability and usability of the exported data due to disordered writing order, thereby improving the data standardization and overall maintainability of the exported file.

[0019] In one possible embodiment, the method includes: In the second pagination data, determine the changed data corresponding to the abnormal pagination data; Based on the changed data, perform a partial replacement process on the data in the first exported file that corresponds to the abnormal pagination data; After the local replacement process is completed, the second exported file is generated.

[0020] Through the above implementation methods, after identifying abnormal pagination data, this application further identifies the actual changed data content within the abnormal pagination and performs local replacement processing on the corresponding data in the first exported file only for the changed data, thereby avoiding rewriting the entire pagination data or the entire exported file. This processing method can significantly reduce the data rewriting scope and processing overhead while ensuring the accuracy of the export results, making the data correction process more accurate and efficient. It is particularly suitable for scenarios where only a small amount of data changes during the export process, thereby improving the processing efficiency and reliability of export results in large-scale data export and verification scenarios.

[0021] Secondly, this application provides a data export system, the system comprising: a data storage module, a copy construction module, a pagination export module, a change determination module, and a change processing module; The data storage module is used to store the original data table; The replica construction module is used to obtain the target data to be exported from the original data table based on preset export conditions, and construct a read-only replica table based on the target data; The pagination export module is used to read first page data from the copy table according to preset pagination rules, and assign a corresponding pagination identifier to each first page data; and write each first page data into the first export file in the order of the pagination identifiers. The change determination module is used to compare each first page data in the copy table with the corresponding data in the original data table after the pagination export module has exported the first export file, and obtain each verification result; if there is at least one first page data with inconsistent data in each verification result, then the at least one first page data is regarded as abnormal pagination data. The change processing module is used to, for each abnormal pagination data, retrieve the corresponding second pagination data from the original data table based on the pagination identifier of the abnormal pagination data, and replace the corresponding first pagination data in the first exported file with the second pagination data to generate a second exported file.

[0022] Thirdly, this application provides a data export apparatus, the apparatus comprising: The replica construction unit obtains the target data to be exported from the original data table based on preset export conditions, and constructs a read-only replica table based on the target data; The first export unit reads first page data from the copy table according to a preset paging rule, and assigns a corresponding paging identifier to each first page data; and writes each first page data into the first export file in the order of the paging identifiers. After the first exported file is exported, the verification unit verifies each first page data in the copy table with the corresponding data in the original data table to obtain each verification result; if there is at least one first page data with inconsistent data in each verification result, then the at least one first page data is regarded as abnormal page data. The second export unit, for each abnormal pagination data, retrieves the corresponding second pagination data from the original data table based on the pagination identifier of the abnormal pagination data, and replaces the corresponding first pagination data in the first export file with the second pagination data to generate a second export file.

[0023] In one possible embodiment, the calibration unit is used for: For each first page of data in the replica table, perform the following operations: According to the preset feature generation rules, perform aggregation operations on at least one data field in the first paginated data to generate corresponding pagination feature values; construct corresponding pagination feature representations based on the pagination feature values. According to the same feature generation rules as the first pagination data, the original pagination feature representation generated for the target data is obtained; the target data refers to data in the original data table that has the same pagination identifier range as the first pagination data. The pagination feature representation is compared with the original pagination feature representation to obtain the verification result corresponding to the first pagination data.

[0024] In one possible embodiment, the calibration unit is used for: Select at least one data field from the first paginated data; Perform field-level aggregation operations on multiple data records corresponding to at least one data field in the first pagination data to obtain the field aggregation result of the at least one data field; The field aggregation results of the at least one data field are subjected to feature mapping processing to generate pagination feature values ​​corresponding to the first pagination data.

[0025] In one possible embodiment, the calibration unit is used for: Based on the order of the pagination identifiers, the corresponding first pagination data is read sequentially; After reading each first page of data, the currently read first page of data is written as a write unit to the first exported file; After completing the writing of the first page of data currently being read, continue reading and writing the next page of data until all page data has been written.

[0026] In one possible embodiment, the first derivation unit is used for: Determine if the first exported file exists; If the first exported file does not exist, the first exported file is created, and the currently read first page data is written into the first exported file; If the first exported file already exists, the currently read first page data is appended to the first exported file.

[0027] In one possible embodiment, the first derivation unit is used for: According to the preset data organization rules, the multiple data records in the currently read first page data are arranged sequentially; Write the sorted data records into the first exported file in sequence, and write delimiters between adjacent data records.

[0028] In one possible embodiment, the second derivation unit is used for: In the second pagination data, determine the changed data corresponding to the abnormal pagination data; Based on the changed data, perform a partial replacement process on the data in the first exported file that corresponds to the abnormal pagination data; After the local replacement process is completed, the second exported file is generated.

[0029] Fourthly, this application provides an electronic device, comprising: Memory, used to store program instructions; A processor is configured to invoke program instructions stored in the memory and execute the steps of the method described in any one of the first aspects according to the obtained program instructions.

[0030] Fifthly, this application provides a computer-readable storage medium storing a computer program, the computer program including program instructions that, when executed by a computer, cause the computer to perform the method described in any one of the first aspects.

[0031] In a sixth aspect, this application provides a computer program product comprising: computer program code, which, when run on a computer, causes the computer to perform the method described in any one of the first aspects. Attached Figure Description

[0032] Figure 1 This is a schematic diagram illustrating an application scenario of an embodiment of this application; Figure 2 A flowchart illustrating an implementation of a data export method provided in this application embodiment; Figure 3 A general schematic diagram of a data export method provided in an embodiment of this application; Figure 4 A structural diagram of a data export system provided in an embodiment of this application; Figure 5 A structural diagram of a data export device provided in an embodiment of this application; Figure 6 This is a structural diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. Unless otherwise specified, the embodiments and features in the embodiments of this application can be arbitrarily combined with each other. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than that shown here.

[0034] The terms "first" and "second" in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the term "comprising" and any variations thereof are intended to cover non-exclusive protection. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices. The term "multiple" in this application can mean at least two, for example, two, three, or more, and the embodiments of this application do not impose limitations.

[0035] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These embodiments should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description. It should be noted that in the embodiments of this application, certain existing industry solutions such as software, components, and models may be mentioned. These should be considered exemplary, intended only to illustrate the feasibility of implementing the technical solutions of this application, and do not imply that the applicant has already used or necessarily used such solutions.

[0036] The acquisition, transmission, storage, and use of data in this application all comply with the requirements of relevant national laws and regulations.

[0037] Before introducing the data export method provided in the embodiments of this application, for ease of understanding, the technical background of the embodiments of this application will be described in detail below.

[0038] In existing data export scenarios, especially in e-commerce and financial settlement systems, the end-of-day phase typically requires the centralized export of a large amount of business data generated that day for subsequent reconciliation, accounting, or auditing. Taking transaction-based systems as an example, end-of-day reconciliation often requires obtaining all order, payment, and refund transaction data for the day and generating corresponding reconciliation files to ensure the accuracy and completeness of business data.

[0039] In related technologies, the method of generating reconciliation files by querying and exporting all data at once is still used. This means that data is not paginated during export; instead, all data meeting the conditions is read directly from the database and written to the file. When exporting large amounts of data, this method can easily cause the application to consume a large amount of memory resources in a short period, increasing system memory pressure and even causing memory overflow and other abnormal issues if the available memory is exceeded, thus affecting the normal execution of the data export task and the stable operation of the system. At the same time, a large-scale query at once will also put a significant load on the database server, increasing the consumption of CPU, memory, and I / O resources, thereby affecting the responsiveness of other business operations.

[0040] To reduce resource consumption, some technologies employ paginated queries to export data in batches. This involves repeatedly reading portions of data from the database according to preset pagination rules and gradually writing them to the exported file. While this method alleviates the performance pressure of a single query to some extent, it still has shortcomings in real-world business environments. During paginated export, because each page of data is retrieved from the database through multiple independent queries, if the original data table undergoes addition, deletion, or modification operations between paginated queries, it can cause data offsets or changes in the order of subsequent paginated queries. This can result in some data being exported repeatedly or skipped, or the exported data content being inconsistent with the actual data state in the final database, leading to data errors.

[0041] Therefore, how to reduce system resource consumption during the export process while ensuring data export efficiency, and effectively address the inconsistency risks caused by data changes during the export process, so as to improve the stability and accuracy of data export, has become a technical problem that urgently needs to be solved in this field.

[0042] In view of this, this application proposes a data export method. The beneficial effects of this application are that, while ensuring the efficiency of large-scale data export, it effectively reduces system resource consumption and can cope with the inconsistency risks caused by changes in the original data during the data export process, thereby improving the stability and accuracy of the data export results.

[0043] First, by constructing a read-only copy table from the original data table based on preset export conditions, the data export process is executed based on a relatively stable data set, avoiding the impact of real-time data changes by directly accessing the original data table during the export process, which helps to improve the stability of the export process.

[0044] Secondly, by reading data from the replica table in pages and writing each page to the export file in page order, a large amount of data is avoided from being loaded at once, thereby reducing memory usage and database access pressure. This is suitable for continuous export operations in scenarios with large amounts of data.

[0045] Furthermore, after the initial pagination export is completed, the pagination data in the copy table is compared with the corresponding data in the original data table, and any inconsistent or abnormal pagination data is located. This allows the system to accurately identify problematic pagination caused by data changes during the export process, thus avoiding the need to re-export all data indiscriminately.

[0046] Finally, by retrieving the corresponding data from the original data table only for abnormal pagination data and performing partial replacement processing on the exported file, the accuracy of the exported results is ensured while reducing the scope of data rewriting and processing overhead, thereby improving the overall efficiency of the data export and correction process.

[0047] The preferred embodiments of this application will be described in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit this application. Furthermore, the embodiments and features in the embodiments of this application can be combined with each other without conflict.

[0048] like Figure 1 The diagram shown is an application scenario illustration of an embodiment of this application. The application scenario diagram includes two terminal devices 110 and one server 120.

[0049] In this embodiment, the terminal device 110 includes, but is not limited to, mobile phones, tablets, laptops, desktop computers, e-book readers, smart voice interaction devices, smart home appliances, and in-vehicle terminals. The terminal device may have a client installed for data management or data export. This client can be software (e.g., a browser, data management software, business management system client, etc.), or a webpage, mini-program, etc. The server 120 is the backend server corresponding to the software, webpage, or mini-program, or a data processing server specifically used for performing data export processing; this application does not impose specific limitations. The server 120 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0050] It should be noted that the data export method in each embodiment of this application can be executed by an electronic device, which can be a terminal device 110 or a server 120. That is, the method can be executed by the terminal device 110 or the server 120 alone, or by both the terminal device 110 and the server 120. For example, when executed by both the terminal device 110 and the server 120, the terminal device 110 can initiate a data export request and transmit preset export conditions to the server 120. After receiving the data export request, the server 120 obtains the target data to be exported from the original data table based on the export conditions and constructs a read-only copy table. Subsequently, the server 120 performs pagination processing on the data in the copy table according to preset pagination rules and writes each page of data into the export file in sequence. After completing the initial export, the server 120 compares the pagination data in the copy table with the corresponding data in the original data table, and when abnormal pagination data is detected, it retrieves the corresponding pagination data from the original data table again and updates the export file. The terminal device 110 can receive the export results or export status information returned by the server 120 and display or manage the data export process.

[0051] In one alternative implementation, the terminal device 110 and the server 120 can communicate via a communication network.

[0052] In one alternative implementation, the communication network is a wired network or a wireless network.

[0053] It should be noted that, Figure 1 The examples shown are merely illustrative; in reality, the number of terminal devices and servers is unlimited and is not specifically limited in the embodiments of this application.

[0054] In the embodiments of this application, when there are multiple servers, the multiple servers can form a blockchain, and the server is a node on the blockchain; as disclosed in the data export method of this application, the exported files, verification results, etc. involved can be stored on the blockchain to improve the traceability and credibility of the relevant data.

[0055] Furthermore, the embodiments of this application can be applied to various scenarios, including not only data export scenarios such as resource transfer data export, log data export, and reconciliation data generation, but also scenarios such as cloud technology, artificial intelligence, smart transportation, and assisted driving.

[0056] The data export method provided by the exemplary embodiments of this application will be described below with reference to the accompanying drawings and the application scenarios described above. It should be noted that the application scenarios described above are only shown to facilitate understanding of the spirit and principles of this application, and the embodiments of this application are not limited in any way in this respect.

[0057] See Figure 2 The diagram shown is an implementation flowchart of a data export method provided in this application embodiment, applied to a server. The specific implementation flow of this method is as follows: steps S21~S24: S21: Based on preset export conditions, obtain the target data to be exported from the original data table, and construct a read-only copy table based on the target data.

[0058] In this embodiment of the application, in order to obtain the target data set for subsequent pagination processing before data export, after receiving the data export request, the target data to be exported is obtained from the original data table based on the preset export conditions, and the target data is written into an independent data storage structure to construct a read-only copy table for this data export.

[0059] The preset export conditions are used to limit the range of data to be exported, and may include time range conditions, business type conditions, or business status conditions. For example, in an end-of-day reconciliation scenario, preset export conditions can be used to limit the target data to order data, payment data, or refund data generated or updated on the current day.

[0060] In one optional implementation, preset export conditions are used to limit the data range of this data export. The preset export conditions include at least one of time range conditions, business type conditions, business status conditions, and data identification conditions, or a combination of the above conditions.

[0061] In one implementation, preset export conditions are generated by the terminal device after receiving user input and sent to the terminal device; based on the preset export conditions, target data that meets the conditions is filtered from the original data table.

[0062] In another implementation, the preset export conditions are automatically generated by an intelligent decision-making module located on the side. The intelligent decision-making module is used to acquire operational information related to the data export task and determine the preset export conditions for this data export based on the operational information.

[0063] The operational information includes at least one or more of the following: historical data export records, historical data verification results, data change frequency in the original data table, data volume of the data to be exported, system resource usage status, or current business operation status.

[0064] In one optional implementation, the intelligent decision-making module determines the time range conditions, business type conditions, or business status conditions for this data export based on operational information, so as to control the target data scale for this data export.

[0065] For example, when the frequency of data changes in the original data table is detected to be higher than a preset threshold within a certain time period, the intelligent decision-making module can narrow down the time range conditions; or, when the system resource utilization rate is detected to be high, the intelligent decision-making module can adjust the business type conditions and select only data from certain business types as target data.

[0066] During the process of acquiring the target data to be exported and constructing a read-only copy table based on the target data, a temporary table can be created as a read-only copy table to store the target data that meets the preset export conditions. The data structure of the temporary table is consistent with the original data table and includes at least an identifier field to uniquely identify each data record, so that each data record can be uniquely identified in the copy table. For example, when the original data table is an order data table, a temporary order data table can be created as a read-only copy table to store the order data to be exported on that day.

[0067] In one optional implementation, the process of constructing the read-only replica table is performed using a batch import method. Optionally, target data that meets preset export conditions and has not yet been written to the replica table can be read multiple times from the original data table, and the read target data can be written to the replica table in batches. The amount of data written in each batch can be preset to a fixed quantity, for example, no more than a preset number of data records can be written each time.

[0068] For example, in one alternative implementation, order data whose update time falls within a specified time range can be filtered from the original order data table based on the order update time field. The number of data records written to the copy table is limited to a preset number each time the data is written, while data records that already exist in the copy table are excluded, thereby enabling the batch import of the target data.

[0069] During the batch import process described above, the number of data records imported in each batch can be determined after the import is completed. When the number of data records imported in this batch is zero, it is determined that there is no target data that has not yet been written to the replica table, thus completing the construction of the read-only replica table.

[0070] Using the above method, a replica table can be built without reading all the target data at once, providing a basic data source for subsequent paginated reading and data export.

[0071] S22: Read the first page data from the copy table according to the preset paging rules, and assign a corresponding paging identifier to each first page data; write each first page data into the first exported file in the order of the paging identifiers.

[0072] In this embodiment, after the read-only replica table is constructed, page data is read from the replica table according to a preset paging rule, and the data obtained from each page read is used as the first page data. The first page data refers to the set of data with a preset data volume range that is read at once from the replica table according to the paging rule.

[0073] The preset pagination rules can be used to limit the number of data records contained in each page and the order of the paginated data. For example, the pagination rules can limit each first page to contain no more than a preset number of data records (e.g., 1000), and arrange the paginated data in order based on the identifier field of the data records.

[0074] Considering that reading and writing all the copy data at once can easily lead to excessive system resource consumption in large-scale data export scenarios, this application proposes a data export method based on page identifier order to write the export file page by page, so as to reduce the scale of single data processing and improve the stability of the export process.

[0075] Optionally, based on the order of the page identifier, the corresponding first page data is read sequentially; after reading each first page data, the currently read first page data is written as a write unit to the first exported file; after completing the writing of the currently read first page data, the next first page data is read and written, until all first page data is written.

[0076] In one optional implementation, data can be read from the replica table using a paginated query. Each paginated query returns a preset number of data records, which are then sorted according to the order of the identifier field. For the first paginated read, the paginated query can be performed directly based on the replica table. For subsequent paginated reads, the next paginated data with an identifier field value greater than the last data record in the previous paginated data can be read, thus ensuring the sequential continuity of the paginated data.

[0077] In this embodiment, the writing unit corresponds to the first paginated data set obtained from a single pagination query. Optionally, after completing a pagination query, multiple data records in the first paginated data set are converted into data content in a preset format and temporarily stored in a data set, for example, in a list structure. Each element in the data set corresponds to one data record, serving as the smallest processing unit for subsequent writing to the exported file.

[0078] During the paginated export process, after the entire write unit corresponding to the data set is written to the first export file, the data set and memory resources corresponding to the paginated data are released, and the reading, conversion and writing operations of the next paginated data are continued. When the number of data records returned by a paginated read is zero, it is determined that all data in the replica table has been exported, thus completing the generation of the first export file.

[0079] Through the above implementation methods, this application processes each page of data as an independent writing unit, thereby decoupling pagination query, memory temporary storage, and file writing. This avoids the problem of excessive memory consumption caused by loading all data at once, and ensures the order and stability of the export process, thus improving the reliability of system operation and data export efficiency in large-scale data export scenarios.

[0080] In this application embodiment, considering that there may be situations where the exported file has not yet been created or needs to be continuously written on the basis of an existing exported file during the paginated export process, this application further proposes a write control method based on the file existence status judgment during the above-mentioned paginated write process to ensure the continuity and security of the paginated write process.

[0081] Optionally, the process of writing the currently read first page of data as a write unit to the first exported file includes: Check if the first exported file exists; if the first exported file does not exist, create the first exported file and write the currently read first page data into the first exported file; if the first exported file already exists, append the currently read first page data to the first exported file.

[0082] In this embodiment, the writing unit is used to carry the first page data obtained from a single pagination query. Optionally, after completing a pagination query, multiple data records in the first page data are converted into data content in a preset format and stored in a data set. The data set is used to temporarily store all data records corresponding to the current page so that they can be written as a whole to the first exported file.

[0083] In one optional implementation, before performing the write operation, the storage path of the exported file is first determined, and a file write object for writing is created based on the storage path. Subsequently, multiple data records in the write unit are sequentially written to the first exported file using an append-only writing method, without overwriting the already written data. After the write unit is completed, the corresponding write object is closed and the related resources are released.

[0084] Through the above implementation methods, this application can realize on-demand creation and append writing control of exported files during the paginated export process, avoiding the risk of data loss due to repeated creation or overwriting of files. At the same time, in conjunction with the processing method of the paginated writing unit, it effectively improves the robustness and recoverability of the export process, and further enhances the stability and practicality in large-scale data export scenarios.

[0085] Considering that the order and organization of different data records in the file directly affect the readability of the exported file and the efficiency of subsequent processing during the paginated writing process, this application proposes a paginated writing control method based on preset data organization rules to ensure the structural consistency and order stability of paginated data in the exported file.

[0086] Optionally, the process of writing the currently read first page data to the first exported file includes: arranging multiple data records in the currently read first page data in sequence according to preset data organization rules; writing the arranged data records to the first exported file in sequence, and writing delimiters between adjacent data records.

[0087] In this embodiment, the data organization rules are used to define the writing order of each data record in the paginated data in the exported file and the separation method between records. Optionally, when writing the current paginated data as a writing unit to the exported file, multiple data records in the writing unit are traversed sequentially and written to the exported file one by one in a preset order. At the same time, after each data record is written, a separator identifier used to distinguish adjacent records is written, thereby forming a data content with a clear structure.

[0088] Through the above implementation methods, this application can ensure that the paginated data is arranged in a stable and clear order in the exported file, avoid parsing errors caused by disordered record order or unclear boundaries, and improve the availability and reliability of the exported file in subsequent reconciliation, verification or parsing processing.

[0089] Considering that the paginated writing process involves operations such as file creation, writing, and resource release, if the relevant resources are not released in time or abnormal situations are not handled properly, it may lead to problems such as file handle occupation, writing failure, or export interruption. Therefore, this application further proposes a write processing method with resource management and abnormal control in the above-mentioned paginated writing process.

[0090] In one alternative implementation, when performing a paging data write operation, a write object for performing file writing is created before the write begins, and the system resources associated with the write object are automatically released after the data writing of the current write unit is completed, so as to avoid the long-term occupation of resources from affecting system stability.

[0091] In another optional implementation, when performing paging write operations, abnormal situations that may occur during the write process are captured and processed. When a file write abnormality or resource access abnormality is detected, the corresponding abnormal information is recorded, and the current write operation is terminated or retry is performed according to the abnormality, so as to ensure the controllability and security of the data export process.

[0092] Through the above implementation methods, this application can achieve automatic management of system resources and effective control of abnormal situations during the paginated writing and exporting of files, avoiding data export failures caused by unreleased resources or writing errors, thereby further improving the stability and robustness of the data exporting process.

[0093] Considering that the probability of data changes during the export process varies for different paginated data in actual business scenarios, this application further proposes an export status management method based on pagination identifiers.

[0094] Optionally, during the first export process of pagination, after the writing of each first page of data is completed, the corresponding export status information is recorded for that page identifier; the export status information is used to indicate whether the corresponding page data has been exported.

[0095] By maintaining pagination-level export status information, if an abnormal interruption occurs during the export process, the reading and writing operations of subsequent pagination data can continue from the pagination identifier that was most recently exported, thereby avoiding the repeated export of completed pagination data and improving the fault tolerance and execution efficiency of the export process.

[0096] In this way, the first exported file has a page-level execution status record during the export process, which enables the data export process to resume after abnormal interruption, reduces redundant calculations and invalid writes, and improves the execution efficiency and fault tolerance of the export task.

[0097] S23: After the first exported file is exported, the data of each first page in the copy table is compared with the corresponding data in the original data table to obtain the verification results; if there is at least one first page data that is inconsistent in the verification results, then at least one first page data is regarded as abnormal page data.

[0098] In this embodiment of the application, after the generation of the first exported file is completed, the consistency processing stage of the paginated data is entered, which is used to deal with the problem that the original data table may change during the data export process, resulting in the export result being inconsistent with the actual data state.

[0099] Specifically, based on the pagination identifier assigned to each first page of data during the pagination export process, pagination-level consistency processing is performed on the first page of data in the copy table and the corresponding data in the original data table. Corresponding data refers to the set of data in the original data table that shares the same pagination identifier range as the first page of data.

[0100] Considering that the data in the original data table may have been added, deleted or modified after the pagination export is completed, directly checking all data records in the exported file one by one would not only be computationally expensive, but also difficult to meet efficiency requirements in large-scale data scenarios. Therefore, this application also proposes a data consistency check method based on pagination feature representation, which can reduce the system resource consumption during the check process while ensuring the accuracy of the check.

[0101] Optionally, the process of comparing each first page of data in the replica table with the corresponding data in the original data table to obtain each verification result includes: For each first page of data in the replica table, perform the following operations: According to the preset feature generation rules, perform aggregation operations on at least one data field in the first pagination data to generate corresponding pagination feature values; construct corresponding pagination feature representations based on the pagination feature values; obtain the original pagination feature representations generated for the target data according to the same feature generation rules as the first pagination data; the target data refers to data in the original data table that has the same pagination identifier range as the first pagination data; compare the pagination feature representations with the original pagination feature representations to obtain the verification results corresponding to the first pagination data.

[0102] First, according to preset feature generation rules, aggregation operations are performed on at least one data field in the first paginated data to generate corresponding pagination feature values. Data fields can be business fields, status fields, or key fields reflecting changes in data content. Aggregation operations are used to summarize the values ​​of multiple data records in the paginated data on their data fields to form pagination feature values ​​that characterize the overall features of the paginated data.

[0103] After obtaining the pagination feature values, a corresponding pagination feature representation is constructed based on these values. The pagination feature representation is used to describe the first page of data from an overall perspective, avoiding the need to compare each data record in the paginated data one by one.

[0104] Subsequently, following the same feature generation rules as the first page data, corresponding original pagination feature representations are generated for the target data in the original data table that have the same pagination identifier range as the first page data. Target data refers to the set of data in the original data table that corresponds to the first page data in terms of the pagination identifier range.

[0105] After obtaining the pagination feature representation and the original pagination feature representation, the two are compared to obtain the verification result corresponding to the first pagination data. When the comparison result indicates that the two are consistent, it is determined that the corresponding first pagination data has not undergone any data changes that would affect the export result during the export process; when the comparison result indicates that the two are inconsistent, it is determined that the corresponding first pagination data has a risk of data changes during the export process, and the first pagination data is identified as abnormal pagination data.

[0106] By using the pagination-level feature verification method described above, it is possible to quickly determine whether pagination data has changed before and after export without comparing data records one by one, thereby limiting the scope of subsequent data processing to pagination data that is at risk of change.

[0107] Through the above implementation methods, this application performs consistency verification between paginated data and the original data table based on pagination feature representation, raising the data verification granularity from the data record level to the pagination level. This significantly reduces the computational complexity and system resource consumption during the verification process while ensuring the accuracy of the verification results. Simultaneously, this method can quickly locate the range of paginated data at risk of data change, avoiding repeated verification or processing operations on already confirmed consistent paginated data. This improves the efficiency of data consistency processing and the overall system stability in large-scale data export scenarios.

[0108] Considering that subtle changes in a single data record (such as modifications to field values) in paginated data may not be directly reflected as changes in the number of records, making it difficult to identify whether paginated data has become abnormal in a timely manner, this application also proposes a pagination feature generation scheme based on paginated data field aggregation and feature mapping, which is used to abstractly represent the overall content features of paginated data.

[0109] Optionally, the process of performing aggregation operations on at least one data field in the first paged data to generate corresponding paged feature values ​​includes: selecting at least one data field from the first paged data; performing field-level aggregation operations on multiple data records corresponding to at least one data field in the first paged data to obtain a field aggregation result of at least one data field; and performing feature mapping processing on the field aggregation result of at least one data field to generate paged feature values ​​corresponding to the first paged data.

[0110] In this embodiment of the application, the process of performing aggregation operations on at least one data field in the first pagination data to generate corresponding pagination feature values ​​includes the following steps.

[0111] First, select at least one data field from the first page of data. The data field can be selected based on the business scenario; for example, it can be a monetary field, a status field, a time field, or other business fields that reflect data changes. The selection of a data field is not limited to a single field; it can also be a combination of multiple fields.

[0112] Subsequently, for at least one data field selected in the first pagination data, a field-level aggregation operation is performed on the corresponding multiple data records to obtain a field aggregation result for at least one data field. Field-level aggregation operations are used to summarize the values ​​of each data record in the pagination data for that field, forming a field aggregation result that reflects the overall distribution or changes of that field.

[0113] After obtaining the field aggregation results, feature mapping processing is performed on the field aggregation results of at least one data field to generate pagination feature values ​​corresponding to the first paginated data. Feature mapping processing is used to map the field aggregation results into a feature representation in a unified format, so that features can be compared between different paginated data in subsequent steps.

[0114] In this embodiment of the application, feature mapping processing is used to convert the field aggregation results obtained from field-level aggregation operations into unified pagination feature values ​​that can be used for pagination consistency verification.

[0115] Specifically, after obtaining the field aggregation result of at least one data field, feature mapping processing is performed on the field aggregation result. Feature mapping processing includes, but is not limited to, one or more combinations of the following processing methods: First, the field aggregation results are standardized to eliminate the impact of differences in the value range, order of magnitude, or data distribution of different fields on subsequent validation. Standardization may include numerical normalization, value range compression, precision standardization, or format standardization of the field aggregation results, making the aggregation results of different paginated data on the same field comparable.

[0116] After standardization, the field aggregation results are mapped to feature elements in a preset format. Feature elements can be used to represent the overall state of paginated data in the corresponding field, such as reflecting the value distribution characteristics, summary characteristics, or change characteristics of the field in the current paginated data.

[0117] When only one field is selected, the feature element corresponding to that field can be directly used as the pagination feature value. When multiple fields are selected, the feature elements corresponding to multiple fields can be combined and mapped according to the preset field order or field weight rules to generate a pagination feature value that represents the overall feature of the paginated data.

[0118] The combined mapping is not a simple concatenation, but is used to ensure that the feature elements corresponding to different fields in the pagination feature values ​​have a definite positional relationship and mapping rules, so that different pagination data have a consistent feature structure when generating pagination feature values.

[0119] Through the above feature mapping process, the field aggregation results formed by multiple fields and multiple data records in the paginated data can be uniformly mapped to a pagination feature value, which is used for subsequent consistency verification with the pagination feature value generated by the corresponding pagination data in the original data table.

[0120] For example, in a business scenario where order data is exported, the first page of data contains multiple order records, and each order record includes at least an order amount field.

[0121] You can select the order amount field as the feature generation field and perform field-level aggregation operations on the order amount of all order records in the paginated data, such as calculating the sum, mean, or distribution of order amounts within a preset range in the paginated data.

[0122] After obtaining the field aggregation result corresponding to the order amount field, perform feature mapping processing on the field aggregation result, such as mapping the field aggregation result to a fixed-precision numerical feature element, and using the feature element as the pagination feature value corresponding to the pagination data.

[0123] When the order amount of any order record within the corresponding pagination range in the original data table changes, even if the order record's identifier remains unchanged, the field aggregation result will change accordingly, resulting in a change in the generated pagination feature value. By comparing the pagination feature value with the original pagination feature value, it can be determined that there is an abnormal change in the corresponding pagination data.

[0124] The above method can compress the values ​​of multiple data records in multiple fields in paginated data into a single pagination feature value, which can then be used to represent the overall characteristics of the paginated data.

[0125] In one optional implementation, in actual business scenarios, only a small number of data records in the paginated data may change, and the changes may take various forms, such as minor adjustments to field values, changes in amount, or status changes. To further improve the accuracy and adaptability of abnormal pagination location, this application proposes an abnormal pagination location scheme.

[0126] In one alternative implementation, based on generating pagination feature values ​​for the first pagination data, a feature histogram of the pagination data is further constructed to describe the distribution of field values ​​in the pagination data.

[0127] Specifically, according to the preset interval division rules, the value range of at least one data field in the first page of data is segmented and statistically analyzed. For example, the amount field is divided into several amount intervals, the status field is divided into different status categories, and the number or proportion of data records in each interval is counted to form a feature histogram of the paged data.

[0128] Based on the same interval division rules, data with the same pagination identifier range as the first pagination data are obtained from the original data table, and corresponding original pagination feature histograms are generated. Subsequently, the pagination feature histograms are compared and analyzed with the original pagination feature histograms. When there is a difference in the statistical results corresponding to at least one interval, the first pagination data is determined to be abnormal pagination data.

[0129] For example, in an order reconciliation scenario, a certain page originally contained 800 small orders and 200 medium orders. However, after exporting, the corresponding page range in the original data table becomes 790 small orders and 210 medium orders. Although the total number of orders has not changed, the distribution of the corresponding interval in the feature histogram has changed, which can be used to identify the page as an abnormal page.

[0130] Through the above implementation methods, this application can identify abnormal pagination by changing the value distribution within paginated data. Even if the number of paginated data records remains unchanged, it can effectively detect changes in field values, thereby improving the sensitivity of anomaly location.

[0131] In another alternative implementation, this application proposes an anomaly pagination location scheme based on pagination data fingerprint digest to reduce the computational complexity of verification.

[0132] Optionally, after obtaining the first page of data, the values ​​of multiple fields from multiple data records in the page of data are concatenated in a predetermined order according to a preset data serialization rule to form a page of data sequence. Subsequently, a page of data fingerprint digest is generated based on the page of data sequence to uniquely represent the overall content characteristics of the page of data in the current export state.

[0133] Based on the same serialization rules, an original pagination data fingerprint digest is generated for data in the original data table that has the same pagination identifier range as the first pagination data, and the two are compared. When the pagination data fingerprint digest does not match the original pagination data fingerprint digest, the first pagination data is determined to be abnormal pagination data.

[0134] For example, in a payment transaction export scenario, the order number, payment amount, and payment status fields are concatenated in a fixed order to generate a paginated data sequence, and a summary value is calculated for this sequence. When the amount of an order is modified, even if the order's position in the pagination does not change, the summary value of its corresponding pagination will change, thus triggering an abnormal pagination detection.

[0135] Through the above implementation methods, this application can achieve rapid verification of the overall consistency of paginated data without comparing each field individually, and is suitable for export scenarios with large data scale and high verification performance requirements.

[0136] In another alternative implementation, considering that a single verification method may have the risk of missing detection in some extreme scenarios, this application further proposes an abnormal pagination location scheme based on joint verification of multiple pagination features.

[0137] Optionally, for the same first page of data, multiple different types of pagination features can be generated simultaneously, including but not limited to pagination feature values, pagination feature histograms, and pagination data fingerprint summaries. Each of these pagination features is then compared and analyzed with the original pagination features generated within the corresponding pagination identifier range in the original data table.

[0138] When the comparison results of at least one pagination feature indicate inconsistencies, the corresponding first pagination data can be determined to be abnormal pagination data.

[0139] For example, during the order export process, if only the status field of one order changes within a certain page, this change may result in a small change in the page feature value, but it will be reflected in the page feature histogram or fingerprint summary, thus being successfully identified through the joint verification mechanism.

[0140] Through the above implementation methods, this application can improve the reliability of abnormal pagination identification and reduce the probability of missed detection by cross-validation of multi-dimensional features.

[0141] In another alternative implementation, this application further introduces historical state evolution analysis of pagination data to achieve more intelligent abnormal pagination location.

[0142] Optionally, in multiple data export tasks, the pagination feature value or pagination feature representation corresponding to each pagination identifier is recorded, and a historical state sequence of pagination data is constructed. Based on the historical state sequence, the range or trend of change of each pagination data under normal conditions is determined.

[0143] In the current export task, when the pagination characteristics corresponding to a certain pagination data exceed its historical state range or deviate from the preset change trend, the pagination data is identified as abnormal pagination data.

[0144] For example, if the total amount of a certain page remains within a stable range in multiple consecutive exports, but shows a significant offset in the current export task, it can be determined that the page is abnormal based on the historical evolution results.

[0145] Through the above implementation methods, this application can identify unexpected sudden data changes, improve the intelligence level of abnormal pagination location, and is particularly suitable for long-running batch data export systems.

[0146] Through the pagination feature generation and verification process described above, at least one abnormal pagination data with inconsistencies can be identified. In practical applications, multiple verification methods can be combined to perform multiple rounds of filtering on the pagination data to further improve the accuracy and reliability of abnormal pagination identification.

[0147] Optionally, for cases where new or deleted data records exist in the paginated data, a quick determination can be made based on changes in the number of data records within the pagination identifier range. When a discrepancy is detected between the number of records in the corresponding paginated data in the replica table and the original data table, the paginated data can be directly identified as abnormal paginated data without further complex feature comparison processing.

[0148] For paginated data where there are no additions or deletions, but field values ​​may have been modified, consistency checks can be further performed based on the aforementioned pagination feature values, pagination feature representations, or field aggregation results. Since field-level aggregation operations and feature mappings can reflect the overall content characteristics of paginated data in a statistical sense, even if the data record identifiers in the paginated data remain unchanged, as long as the value of any data field changes, the corresponding pagination feature value will also change, thus enabling accurate identification of abnormal paginated data with modification behavior.

[0149] In one optional implementation, multi-level anomaly filtering can be performed on paginated data in the following order: record quantity verification, pagination feature verification, and field-level feature verification. When any verification step indicates that there is an anomaly in the paginated data, the subsequent verification operations can be terminated in advance and the corresponding paginated data can be marked as abnormal paginated data, thereby reducing the overall verification overhead while ensuring the accuracy of identification.

[0150] After identifying the abnormal pagination data, there is no need to reprocess all exported data. Instead, subsequent data correction operations are performed only on the pagination data that has been identified as abnormal, thereby achieving partial updates and compensation of the exported results.

[0151] S24: For each abnormal pagination data, based on the pagination identifier of the abnormal pagination data, retrieve the corresponding second pagination data from the original data table, and replace the corresponding first pagination data in the first exported file with the second pagination data to generate the second exported file.

[0152] The second page data refers to the set of data in the original data table that is reread according to the same page identifier range and page rules as the abnormal page data, reflecting the latest data status. The second page data corresponds one-to-one with the corresponding first page data in terms of page range, but its data content is based on the state of the original data table at the time of rereading.

[0153] In this embodiment of the application, after the abnormal pagination data is identified, the export operation is not re-executed for all exported data. Instead, only for the pagination data that is determined to be abnormal, the corresponding data range is accurately located based on its pagination identifier, and the second pagination data within that range is retrieved from the original data table to compensate and update the export results.

[0154] Considering that in some business scenarios, the number of data records that have actually changed in the abnormal pagination data is usually much smaller than the total amount of data in the pagination data, if the exported file is updated by replacing the entire page directly, it may still bring unnecessary file rewriting overhead. Therefore, this application also proposes a partial replacement export update scheme based on the identification of changed data.

[0155] Optionally, the process of replacing the corresponding first page data in the first exported file with the second page data to generate the second exported file includes: determining the changed data corresponding to the abnormal page data in the second page data; performing partial replacement processing on the data in the first exported file corresponding to the abnormal page data based on the changed data; and generating the second exported file after completing the partial replacement processing.

[0156] In one alternative implementation, the changed data corresponding to the abnormal pagination data is first determined in the second pagination data. Changed data refers to data records that differ from the first pagination data in at least the number of data records or the values ​​of data fields.

[0157] Optionally, the data in the first page and the data in the second page can be matched and compared based on the unique identifier field of the data record to determine the newly added data record, the deleted data record, and the data record whose field value has changed.

[0158] After identifying the changed data, a partial replacement process is performed on the data content corresponding to the abnormal pagination data in the first exported file based on the changed data. Specifically, the data segment corresponding to the abnormal pagination data can be located in the first exported file, and only the data records corresponding to the changed data within it are replaced, inserted, or deleted, while the unchanged data records in the same page remain unchanged.

[0159] After completing the partial replacement process, a second export file is generated based on the updated export content. The second export file is used to replace the original first export file as the final data export result.

[0160] In one optional embodiment, if only the amount field of a single order record in the abnormal pagination data is modified, the text content corresponding to that order record can be located and replaced in the exported file without rewriting other order records in that pagination; if there are additions or deletions of order records in the abnormal pagination data, the corresponding insertion or deletion operations can be performed in the exported file to ensure that the data content in the exported file is consistent with the original data table.

[0161] Through the above implementation methods, this application can perform fine-grained update processing only on the data content that has actually changed after identifying abnormal pagination data, without having to rewrite the entire page of data or all exported files. This significantly reduces I / O overhead and system resource consumption during the export compensation process while ensuring the consistency and accuracy of the exported data.

[0162] By combining pagination-level anomaly localization with record-level partial replacement, this application not only improves the processing efficiency in large-scale data export scenarios, but also enhances the export process's adaptability to data changes, enabling the stable generation of highly consistent export files even in business environments where data is continuously updated.

[0163] See Figure 3 As shown, this is a general schematic diagram of a data export method provided in an embodiment of this application. Figure 3 As shown in this embodiment, a copy table for this data export is first constructed based on the original data table. The copy table is generated by replicating the target data from the original data table, providing a stable data source during the export process and thus preventing updates to the original data table during the export from affecting the export results.

[0164] After the replica table is built, the data in the replica table is paginated according to preset pagination rules to form multiple sets of first-page data, for example... Figure 3 The diagram illustrates page 1, page 2, page 3, etc. Subsequently, the page data is imported sequentially according to page order and written to the first exported file to complete the initial data export process.

[0165] After the first exported file is generated, a comparison operation is performed between the copy table and the updated original data table to generate corresponding verification results. These verification results determine whether any data inconsistencies exist due to additions, deletions, or modifications made to the original data table during the export process, and further identify any abnormal pagination data with inconsistencies, such as... Figure 3The diagram illustrates abnormal pagination data 1, abnormal pagination data 3, etc.

[0166] For identified abnormal pagination data, based on the corresponding pagination identifier, the second pagination data corresponding to the abnormal pagination data is retrieved from the updated original data table, such as second pagination data 1 and second pagination data 3 as shown in the figure. Subsequently, the second pagination data is used to replace the corresponding abnormal pagination data in the first exported file, thereby performing compensation update processing on the first exported file and finally generating the second exported file.

[0167] Considering that the above data export, verification, and compensation update process involves multiple functional steps, and that each processing stage has obvious independence and synergy in terms of timing and function, this application further proposes a data export system based on a modular architecture. The above method can be completed collaboratively by multiple functional modules, thereby improving the clarity, scalability, and operational stability of the system implementation.

[0168] See Figure 4 As shown, it is a structural diagram of a data export system provided in an embodiment of this application. The data export system 400 includes: a data storage module 401, a copy construction module 402, a pagination export module 403, a change determination module 404, and a change processing module 405.

[0169] Data storage module 401 is used to store the original data table; The replica construction module 402 is used to obtain the target data to be exported from the original data table based on preset export conditions, and construct a read-only replica table based on the target data; The pagination export module 403 is used to read the first pagination data from the copy table according to the preset pagination rules, and assign a corresponding pagination identifier to each first pagination data; and write each first pagination data into the first export file in the order of the pagination identifiers. The change determination module 404 is used to compare each first page data in the copy table with the corresponding data in the original data table after the pagination export module 403 has exported the first export file, and obtain each verification result; if there is at least one first page data with inconsistent data in each verification result, then at least one first page data is regarded as abnormal pagination data. The change processing module 405 is used to retrieve the corresponding second page data from the original data table for each abnormal page data based on the page identifier of the abnormal page data, and replace the corresponding first page data in the first exported file with the second page data to generate a second exported file.

[0170] It should be noted that the above division of functional modules is only for illustrating the logical structure of the data export system of this application, and does not constitute a limitation on the number of modules or the specific implementation method. The specific processing flow, execution steps and optional implementation methods corresponding to each module can be understood with reference to the relevant descriptions in the foregoing method embodiments. Their technical principles and implementation effects are consistent with those in the above method embodiments, and will not be repeated here.

[0171] Through the modular system structure described above, this application can decouple the functions of copy construction, paginated export, change verification, and data compensation update in the data export process. The modules collaborate with each other through clear functional boundaries, which not only facilitates system implementation and maintenance, but also enables the stable and efficient generation of more consistent export files in business scenarios with large-scale data export and continuous data changes.

[0172] Through the above-described system implementation method, this application effectively reduces system resource consumption while ensuring data export efficiency, and significantly improves the adaptability and accuracy of the exported results to changes in the original data, further enhancing the practicality and reliability of the data export system in complex business environments.

[0173] Based on the same inventive concept, embodiments of this application provide a data export device, please refer to... Figure 5 The data export device 500 includes: The replica construction unit 501 obtains the target data to be exported from the original data table based on preset export conditions, and constructs a read-only replica table based on the target data; The first export unit 502 reads the first page data from the copy table according to the preset paging rules, and assigns a corresponding paging identifier to each first page data; according to the order of the paging identifiers, the first page data is written to the first export file in sequence. After the first exported file is exported, the verification unit 503 verifies each first page data in the copy table with the corresponding data in the original data table to obtain each verification result; if there is at least one first page data with inconsistent data in each verification result, then at least one first page data is regarded as abnormal page data. The second export unit 504, for each abnormal pagination data, retrieves the corresponding second pagination data from the original data table based on the pagination identifier of the abnormal pagination data, and replaces the corresponding first pagination data in the first export file with the second pagination data to generate the second export file.

[0174] In one possible embodiment, the verification unit 503 is used for: For each first page of data in the replica table, perform the following operations: According to the preset feature generation rules, perform aggregation operations on at least one data field in the first pagination data to generate corresponding pagination feature values; construct corresponding pagination feature representations based on the pagination feature values. According to the same feature generation rules as the first page data, obtain the original pagination feature representation generated for the target data; the target data refers to the data in the original data table that has the same pagination identifier range as the first page data. The pagination feature representation is compared with the original pagination feature representation to obtain the verification result corresponding to the first pagination data.

[0175] In one possible embodiment, the verification unit 503 is used for: Select at least one data field from the first page of data; Perform field-level aggregation operations on multiple data records corresponding to at least one data field in the first pagination data to obtain the field aggregation result of at least one data field; Perform feature mapping processing on the field aggregation results of at least one data field to generate the pagination feature value corresponding to the first pagination data.

[0176] In one possible embodiment, the verification unit 503 is used for: Based on the order of the pagination identifiers, the corresponding first page data is read sequentially; After reading each first page of data, the currently read first page of data is written as a write unit to the first exported file; After completing the writing of the first page of data, continue reading and writing the next first page of data until all first page data has been written.

[0177] In one possible embodiment, the first derived unit 502 is used for: Check if the first exported file exists; If the first export file does not exist, create the first export file and write the currently read first page data into the first export file; If the first exported file already exists, append the currently read first page data to the first exported file.

[0178] In one possible embodiment, the first derivation unit 502 is used for: According to the preset data organization rules, the multiple data records in the first page of data currently being read are arranged sequentially; Write the sorted data records into the first exported file in sequence, and write delimiters between adjacent data records.

[0179] In one possible embodiment, the second derivation unit 504 is used for: In the second page of data, identify the changed data corresponding to the abnormal page data; Based on the changed data, perform partial replacement processing on the data in the first exported file that corresponds to the abnormal pagination data; After completing the local replacement process, a second exported file is generated.

[0180] Based on the same inventive concept, embodiments of this application provide an electronic device. Please refer to... Figure 6 As shown, the electronic device includes at least one processor 601 and a memory 602 connected to the at least one processor. In this embodiment, the specific connection medium between the processor 601 and the memory 602 is not limited. Figure 6 Taking the connection between processor 601 and memory 602 via bus 600 as an example, bus 600 in... Figure 6 The connections between other components are indicated by thick lines and are for illustrative purposes only, not as limiting information. The 600 bus can be divided into address bus, data bus, control bus, etc., for ease of representation. Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0181] In this embodiment of the application, the memory 602 stores instructions that can be executed by at least one processor 601. By executing the instructions stored in the memory 602, at least one processor 601 can perform the steps included in the aforementioned data export method.

[0182] The processor 601 serves as the control center of the electronic device. It connects to various parts of the device via various interfaces and lines, and performs overall monitoring by running or executing instructions stored in the memory 602 and accessing data stored in the memory 602, thus controlling the various functions and processing data of the electronic device. Optionally, the processor 601 may include one or more processing units. The processor 601 may integrate an application processor and a modem processor. The application processor primarily handles the operating system and applications, while the modem processor primarily handles wireless communication. It is understood that the modem processor may not be integrated into the processor 601. In some embodiments, the processor 601 and the memory 602 may be implemented on the same chip; in other embodiments, they may be implemented on separate chips.

[0183] Processor 601 can be a general-purpose processor, such as a central processing unit (CPU), digital signal processor, application-specific integrated circuit, field-programmable gate array or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, capable of implementing or executing the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the data export method disclosed in the embodiments of this application can be directly manifested as execution by a hardware processor, or execution by a combination of hardware and software modules within the processor.

[0184] Memory 602, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules. Memory 602 may include at least one type of storage medium, such as flash memory, hard disk, multimedia card, card-type memory, random access memory (RAM), static random access memory (SRAM), programmable read-only memory (PROM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), magnetic storage, magnetic disk, optical disk, etc. Memory 602 can be any other medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer, but is not limited thereto. In the embodiments of this application, memory 602 may also be a circuit or any other device capable of implementing storage functions for storing program instructions and / or data.

[0185] By designing and programming the processor 601, the code corresponding to the data export method described in the foregoing embodiments can be embedded into the chip, so that the chip can execute the steps of the aforementioned data export method when running. How to design and program the processor 601 is a well-known technique to those skilled in the art, and will not be described in detail here.

[0186] Based on the same inventive concept, embodiments of this application also provide a computationally readable storage medium storing computer instructions that, when executed on a computer, cause the computer to perform the steps of the aforementioned data export method.

[0187] In some possible implementations, various aspects of the data export method provided in this application may also be implemented in the form of a program product, which includes program code that, when the program product is run on an electronic device, causes the detection device to perform the steps in the data export method according to the various exemplary embodiments of this application described above.

[0188] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0189] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0190] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0191] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0192] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A data export method, characterized in that, The method includes: Based on preset export conditions, the target data to be exported is obtained from the original data table, and a read-only copy table is constructed based on the target data; The first page data is read from the copy table according to the preset pagination rules, and a corresponding pagination identifier is assigned to each first page data; the first page data is written to the first exported file in the order of the pagination identifiers. After the first exported file is exported, the first page data in the copy table is compared with the corresponding data in the original data table to obtain each verification result; if there is at least one first page data with inconsistent data in each verification result, the at least one first page data is regarded as abnormal page data. For each abnormal pagination data, based on the pagination identifier of the abnormal pagination data, the corresponding second pagination data is retrieved from the original data table again, and the second pagination data is used to replace the corresponding first pagination data in the first exported file to generate a second exported file.

2. The method as described in claim 1, characterized in that, The step of comparing each first page data in the copy table with the corresponding data in the original data table to obtain each verification result includes: For each first page of data in the replica table, perform the following operations: According to the preset feature generation rules, perform aggregation operations on at least one data field in the first paginated data to generate corresponding pagination feature values; construct corresponding pagination feature representations based on the pagination feature values. According to the same feature generation rules as the first pagination data, the original pagination feature representation generated for the target data is obtained; the target data refers to data in the original data table that has the same pagination identifier range as the first pagination data. The pagination feature representation is compared with the original pagination feature representation to obtain the verification result corresponding to the first pagination data.

3. The method as described in claim 2, characterized in that, The step of performing aggregation operations on at least one data field in the first paginated data to generate corresponding pagination feature values ​​includes: Select at least one data field from the first paginated data; Perform field-level aggregation operations on multiple data records corresponding to at least one data field in the first pagination data to obtain the field aggregation result of the at least one data field; The field aggregation results of the at least one data field are subjected to feature mapping processing to generate pagination feature values ​​corresponding to the first pagination data.

4. The method as described in claim 1, characterized in that, The step of writing each first page of data sequentially into the first exported file according to the pagination identifier includes: Based on the order of the pagination identifiers, the corresponding first pagination data is read sequentially; After reading each first page of data, the currently read first page of data is written as a write unit to the first exported file; After completing the writing of the first page of data currently being read, continue reading and writing the next page of data until all page data has been written.

5. The method as described in claim 4, characterized in that, The step of writing the currently read first page data as a write unit to the first exported file includes: Determine if the first exported file exists; If the first exported file does not exist, the first exported file is created, and the currently read first page data is written into the first exported file; If the first exported file already exists, the currently read first page data is appended to the first exported file.

6. The method as described in claim 5, characterized in that, The step of writing the currently read first page data into the first exported file includes: According to the preset data organization rules, the multiple data records in the currently read first page data are arranged sequentially; Write the sorted data records into the first exported file in sequence, and write delimiters between adjacent data records.

7. The method as described in claim 1, characterized in that, The step of replacing the corresponding first pagination data in the first exported file with the second pagination data to generate the second exported file includes: In the second pagination data, determine the changed data corresponding to the abnormal pagination data; Based on the changed data, perform a partial replacement process on the data in the first exported file that corresponds to the abnormal pagination data; After the local replacement process is completed, the second exported file is generated.

8. A data export system, characterized in that, The system includes: a data storage module, a copy construction module, a pagination export module, a change determination module, and a change processing module; The data storage module is used to store the original data table; The replica construction module is used to obtain the target data to be exported from the original data table based on preset export conditions, and construct a read-only replica table based on the target data; The pagination export module is used to read first page data from the copy table according to preset pagination rules, and assign a corresponding pagination identifier to each first page data; and write each first page data into the first export file in the order of the pagination identifiers. The change determination module is used to compare each first page data in the copy table with the corresponding data in the original data table after the pagination export module has exported the first export file, and obtain each verification result; if there is at least one first page data with inconsistent data in each verification result, then the at least one first page data is regarded as abnormal pagination data. The change processing module is used to, for each abnormal pagination data, retrieve the corresponding second pagination data from the original data table based on the pagination identifier of the abnormal pagination data, and replace the corresponding first pagination data in the first exported file with the second pagination data to generate a second exported file.

9. A data export device, characterized in that, The device includes: The replica construction unit obtains the target data to be exported from the original data table based on preset export conditions, and constructs a read-only replica table based on the target data; The first export unit reads first page data from the copy table according to a preset paging rule, and assigns a corresponding paging identifier to each first page data; and writes each first page data into the first export file in the order of the paging identifiers. After the first exported file is exported, the verification unit verifies each first page data in the copy table with the corresponding data in the original data table to obtain each verification result; if there is at least one first page data with inconsistent data in each verification result, then the at least one first page data is regarded as abnormal page data. The second export unit, for each abnormal pagination data, retrieves the corresponding second pagination data from the original data table based on the pagination identifier of the abnormal pagination data, and replaces the corresponding first pagination data in the first export file with the second pagination data to generate a second export file.

10. An electronic device, characterized in that, include: Memory, used to store program instructions; A processor is configured to invoke program instructions stored in the memory and execute the steps of the method according to any one of claims 1-7.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, the computer program including program instructions that, when executed by a computer, cause the computer to perform the method as described in any one of claims 1-7.

12. A computer program product, characterized in that, The computer program product includes: computer program code, which, when run on a computer, causes the computer to perform the method described in any one of claims 1-7.

Images