Data processing method, processing device and electronic equipment

Abstract

The invention relates to the technical field of data processing, in particular to a data processing method and device and electronic equipment. The data processing method comprises the steps of performing slicing processing on original import data based on a self-adaptive slicing rule to generate a plurality of batch files; based on a producer-consumer thread model comprising a producer thread and a consumer thread, executing the import and verification of the batch files in parallel; wherein the producer thread imports batch data into a temporary verification table and generates a verification task; the consumer thread calls a preset verification storage process to execute verification on the verification task, and inserts batch data into a target table from the temporary verification table after verification is passed; and recording the processing state of each batch of data in real time so as to support data breakpoint continuation after abnormal interruption.

Description

Technical Field

[0001] This application belongs to the field of vehicle control technology, specifically relating to a data processing method, processing device, and electronic device. Background Technology

[0002] In the current process of enterprise digital transformation, MySQL, as a mainstream database type, often faces scenarios involving the high-frequency import of massive amounts of data (such as e-commerce user behavior data and financial transaction record data import). This places extremely high demands on import efficiency, fault tolerance, and data consistency. However, MySQL's massive data import technology has significant drawbacks: on the one hand, it adopts a serial mode of "full import followed by full verification," resulting in long total import and verification times and low data import efficiency. When anomalies are detected, a full rollback and re-import are required, leading to a significant increase in time costs and the potential for duplicate imports and primary key conflicts. On the other hand, in pursuit of speed, related technologies often skip critical verifications such as foreign keys, formatting, and business rules, resulting in the import of "orphan data" and dirty data, severely increasing business risks. Furthermore, the verification logic is often placed at the application layer, and its decoupling from the import process leads to poor scalability and an inability to flexibly adapt to the verification needs of multiple business scenarios. Summary of the Invention

[0003] This application aims to address at least one of the technical problems existing in the prior art. To this end, this application proposes a data processing method, processing device, and electronic device that enables a MySQL database to achieve high efficiency, fault tolerance, consistency, and flexible scalability with a low performance loss design.

[0004] In a first aspect, embodiments of this application provide a data processing method, comprising: slicing the original imported data based on adaptive slicing rules to generate multiple batch files; performing the import and verification of the batch files in parallel based on a producer-consumer thread model including producer threads and consumer threads; wherein, the producer thread imports the batch data into a temporary verification table and generates a verification task; the consumer thread calls a preset verification stored procedure to perform verification on the verification task, and inserts the batch data from the temporary verification table into the target table after the verification passes; and recording the processing status of each batch of data in real time to support data interruption resumption after abnormal interruption.

[0005] In some embodiments, the adaptive slicing rules are dynamically determined based on the field characteristics of the original imported data and the hardware performance parameters of the target server.

[0006] In some embodiments, the original imported data is sliced ​​based on adaptive slicing rules, including: obtaining the hardware performance parameters of the target server; dynamically determining the target number of rows for a single batch processing based on the hardware performance parameters and the field characteristics of the original imported data, so that the estimated time for a single batch processing meets the preset duration constraint; and splitting the original imported data according to the target number of rows to generate multiple independent batch files.

[0007] In some embodiments, dynamically determining the target number of rows for a single batch of processing includes: in a target environment, determining a basic row count constraint for processing data within a preset time constraint; determining a first scaling factor for the basic row count constraint based on whether the original imported data contains a large object field, and performing a first scaling process on the basic row count constraint based on the first scaling factor; determining a second scaling factor for the basic row count constraint based on hardware performance parameters, and performing a second scaling process on the basic row count constraint after processing by the first scaling factor based on the second scaling factor, so as to obtain the target number of rows.

[0008] In some embodiments, based on the producer-consumer thread model, the import and verification operations of batch files are executed in parallel, including: the producer thread adds unprocessed batch files to a blocking queue according to the batch list file and in the order of batch identifiers; loads batch data from the blocking queue into a temporary verification table using a target loading command, and closes the non-primary key index of the temporary verification table after the import is completed; and imports the verification task containing the batch identifier, temporary verification table name, and verification type into the task queue.

[0009] In some embodiments, based on the producer-consumer thread model, the batch file import and verification operations are executed in parallel, and the consumer thread obtains a verification task from the task queue, calls the verification stored procedure in the database to perform the verification, inserts the batch data from the temporary verification table into the target table after the verification is passed, and rebuilds the relevant indexes.

[0010] In some embodiments, a stored procedure for validation within the database is invoked to perform validation, including: validating the format, length, and non-null constraints of data fields to perform a first-level validation; validating the uniqueness of the primary key and the validity of foreign key associations of the data to perform a second-level validation; and validating the correctness of the business logic of the data according to configurable business rule expressions to perform a third-level validation.

[0011] In some embodiments, the processing status of each batch of data is recorded in real time, including: creating a record in the breakpoint status management table for each batch of data; wherein the record includes at least the batch number, file path, processing status, start time, and completion time; updating the processing status during processing; wherein the processing status includes waiting for processing, importing, verifying, success, and failure; when an abnormal interruption occurs, the system restarts and queries the breakpoint status management table, skips the batch data with a success status, and performs a recovery operation on the batch data with a status of importing, verifying, or failure.

[0012] Secondly, embodiments of this application provide a data processing apparatus, including: a slicing processing module configured to slice the original imported data based on adaptive slicing rules to generate multiple batch files; a verification module configured to perform batch file import and verification in parallel based on a producer-consumer thread model including producer threads and consumer threads; wherein, the producer thread imports batch data into a temporary verification table and generates verification tasks; the consumer thread calls a preset verification stored procedure to perform verification on the verification tasks, and inserts the data from the temporary verification table into the target table after the verification passes; and a continuation module configured to record the processing status of each batch of data in real time to support data breakpoint continuation after abnormal interruption.

[0013] Thirdly, embodiments of this application provide an electronic device, including: a processor and a memory, the memory storing a computer program, and the processor executing the program to implement the steps of the data processing method as described in the first aspect.

[0014] The technical solution provided in this application slices the data in the system according to preset slicing rules, and then sends the processed batch files to the producer thread and consumer thread in sequence. In the consumer thread, the processed batch files are verified according to a preset verification stored procedure. The use of sliced ​​batch files can effectively reduce the import time of large amounts of data, and the parallel processing mode of the producer thread and consumer thread can further improve the data processing efficiency. At the same time, the verification in the consumer process can also remove unnecessary data, avoid data quality risks, and verify different business logic scenarios during the verification process, effectively improving the scalability and practicality of data import processing.

[0015] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description

[0016] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1A flowchart illustrating the data processing method provided in the embodiments of this application; Figure 2 A flowchart for data slicing processing provided in this application embodiment; Figure 3 A flowchart for dynamically determining the target number of rows in a single batch processing provided in this application embodiment; Figure 4 A flowchart for executing verification by calling a verification stored procedure in the database, provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of the data processing apparatus provided in the embodiments of this application; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0017] Reference numerals: 500 - Data processing device; 510 - Slicing processing module; 520 - Verification module; 530 - Continuation module; 610 - Processor; 620 - Memory; 630 - Input / output interface; 640 - Communication interface; 650 - Bus. Detailed Implementation

[0018] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While some embodiments of this application are shown in the drawings, it should be understood that this application can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this application. It should be understood that the drawings and embodiments of this application are for illustrative purposes only and are not intended to limit the scope of protection of this application.

[0019] It should be understood that the steps described in the method embodiments of this application may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this application is not limited in this respect.

[0020] The data processing method, processing apparatus, and electronic device provided in the embodiments of this application are described below with reference to the accompanying drawings.

[0021] refer to Figure 1 This is a flowchart of a data processing method provided in an embodiment of this application.

[0022] Step S101: Based on adaptive slicing rules, the original imported data is sliced ​​to generate multiple batch files.

[0023] Specifically, the data processing method provided in this application embodiment has an adaptive slicing rule pre-configured in the system. When the system receives the original imported data, it will slice the original imported data according to the adaptive slicing rule to obtain multiple batch files containing different data. The original imported data formats supported by the system include CSV, TXT, Excel, etc., and are encoded as UTF-8 / GBK.

[0024] As an optional implementation, the adaptive slicing rule is dynamically determined based on the field characteristics of the original imported data and the hardware performance parameters of the target server.

[0025] Specifically, in this embodiment, the adaptive slicing rule is dynamically determined based on the field characteristics of the original imported data and the hardware performance parameters of the target server, so as to achieve optimal slicing of any original imported data. The field characteristics of the original imported data are used to determine the complexity of processing a single row of data, and the hardware performance parameters of the target server are used to determine the system throughput and the upper limit of processing speed.

[0026] refer to Figure 2 This is a flowchart of data slicing processing provided in an embodiment of this application.

[0027] Step S201: Obtain the hardware performance parameters of the target server.

[0028] Specifically, the hardware performance parameters of the target server are obtained, including but not limited to CPU parameters, memory parameters, and storage system performance parameters. Among them, the number of CPU cores determines the number of threads that the server can execute, memory determines the amount of data that the target server can store at the same time (such as the number of indexes or the amount of temporary table data), and storage system performance (such as disk IOPS) determines the data read and write speed.

[0029] Step S202: Based on hardware performance parameters and field characteristics of the original imported data, dynamically determine the target number of rows for a single batch processing so that the estimated time for a single batch processing meets the preset duration constraint.

[0030] Specifically, the field characteristics of the original imported data include large object fields and structured fields. Large object fields include, but are not limited to, TEXT / BLOB fields that represent log content or images encoded in Base64. Data containing large object fields usually takes a long time to read and write. Structured fields include integer (int), short character (varchar), and date / time fields, which take less time to read and write.

[0031] Based on the obtained hardware performance parameters of the target server, and combined with the field characteristics of the original imported data, real-time analysis is performed to dynamically calculate the optimal number of target rows per batch for slicing the original imported data, so that the estimated processing time of the target rows per batch meets the preset time constraint, wherein the preset time constraint is preferably 20 to 30 seconds in this embodiment of the application.

[0032] refer to Figure 3 This is a flowchart for dynamically determining the target number of rows for a single batch processing, provided in an embodiment of this application.

[0033] Step S301: Under the target environment, determine the basic row number constraint for processing data within the preset time constraint.

[0034] Specifically, when dynamically determining the target number of rows to be processed in a single batch, the target environment in which the system operates imposes a constraint on the basic number of rows of data to be processed within the preset time limit. The target environment represents the systematic variables that affect the system's processing performance when slicing the original imported data, including the hardware performance of the target server, MySQL parameter configuration, target table structure, target table indexes, and other loads running on the database server at the same time (i.e., the number of concurrent tasks running in the system).

[0035] In this embodiment, a binary search combined with curve fitting is preferred to determine the basic row number constraint. Considering the target environmental conditions, a sufficiently large row number search range is first initialized. The lower limit time needs to be much less than 20 seconds, and the upper limit time needs to be much greater than 30 seconds to ensure complete data coverage. Multiple rounds of iterative testing (e.g., 5-6 rounds) are then performed. In each iteration, the median value of the current search range is taken as the current basic row number constraint. Furthermore, the processing time for data import and verification under the current basic row number constraint is calculated, and the calculated processing time is compared with the preset duration constraint to dynamically adjust the boundary of the basic row number constraint: if the processing time is less than 20 seconds, it indicates the system still has potential, and the lower limit of the search range needs to be increased to test a larger basic row number constraint; if the processing time is greater than 30 seconds, it indicates the system is overloaded, and the upper limit of the search range needs to be decreased to test a smaller basic row number constraint. Through iteration, the search range is rapidly converged, resulting in a set of discrete scatter data points around the preset duration constraint. Each scatter data point contains the corresponding basic row number constraint and its corresponding computational processing time.

[0036] Furthermore, a quadratic polynomial fitting process is performed on the discrete scatter data to obtain the maximum number of rows that the hardware environment can support. Using the obtained discrete scatter data as samples, a quadratic polynomial curve fitting is performed using the least squares method to obtain a continuous mathematical model with the number of rows as the independent variable and the processing time as the dependent variable. This mathematical model can reveal the variation law of processing time with the basic row number constraint under the current target environment. Based on this mathematical model, the optimal basic row number constraint that satisfies the preset duration constraint under the current target environment can be calculated by inputting the duration constraint. This achieves high-precision acquisition of the basic row number constraint corresponding to the preset duration constraint without the need for extensive testing.

[0037] It should be noted that, under normal circumstances, the original imported data used to determine the basic row count constraint is structured field data, which can eliminate the interference of data complexity and accurately obtain the basic data processing capabilities of the target environment system.

[0038] Step S302: Determine the first scaling factor for the basic row number constraint based on whether the original imported data contains a large object field, and perform the first scaling process on the basic row number constraint based on the first scaling factor.

[0039] Specifically, after identifying whether the original imported data contains large object fields, the system needs to determine the corresponding first scaling factor based on the number of large object fields. The determination of the first scaling factor is obtained by testing with test sets containing different data characteristics. For original imported data containing large object fields, its processing requires more memory copying and disk read / write, and its data processing time is longer. The system can allocate a first scaling factor significantly less than 1.0, such as 0.25 to 0.4, to perform the first scaling process on the basic row number constraint in order to actively reduce its row number constraint, thereby offsetting the additional time overhead brought by the large object fields.

[0040] Step S303: Based on the hardware performance parameters, determine the second scaling factor for the basic row number constraint, and perform a second scaling process on the basic row number constraint after processing by the first scaling factor based on the second scaling factor to obtain the target row number.

[0041] Specifically, the system can also determine a second scaling factor for the basic row count constraint based on hardware performance parameters. These hardware performance parameters can be pre-configured by the user or obtained automatically by running diagnostic scripts. Under different hardware performance parameter conditions, the basic row count constraint processed by the first scaling factor is scaled a second time according to the second scaling factor to obtain the target row count.

[0042] In this embodiment of the application, when the number of central processing unit cores is greater than or equal to 8, it can be considered that the basic row count constraint can be improved by 10% accordingly. When the disk IOPS is greater than or equal to 10,000 IOPS, it can be considered that the basic row count constraint can be improved by 15% accordingly. For example, when the hardware performance parameters are that the central processing unit has 8 cores and the disk IOPS is 12,000 IOPS, the second scaling factor can be 1.265.

[0043] As an optional implementation, in a multi-concurrency environment, the system can further scale and confirm the basic row count constraint. By simulating typical concurrent scenarios under actual operation (such as multi-threaded operation scenarios), the actual processing time and resource (such as CPU, disk I / O) utilization of the system under this concurrency pressure can be observed. If the processing time in the concurrent environment exceeds the duration constraint under the current target environment condition, a concurrency scaling factor less than 1 (e.g., 0.85) can be recorded to reduce the basic row count constraint for calculation in the single-threaded environment.

[0044] As an optional implementation, a basic constraint lookup table is generated based on the basic row count constraint, the first scaling factor, the second scaling factor, and the duration constraint.

[0045] Specifically, by combining the aforementioned basic row count constraints, first scaling factor, second scaling factor, and duration constraints, the estimated value of the "basic row count constraint" for each specific scenario is calculated. These data and the corresponding target environment are then filled into structured data tables, generating the "basic constraint lookup table." Based on this table, the system performance of the target environment can be accurately grasped. When the target environment changes, the system can quickly respond to the upper limit of the current slice's row count through a combination of "table lookup" and "interpolation," with a response time within 50ms. This abandons the mechanical mode of "fixed row count slicing" in traditional technology and provides fundamental technical support for the subsequent system to import and verify data under optimal performance.

[0046] Step S203: Split the original imported data according to the target number of rows to generate multiple independent batch files.

[0047] Specifically, the original imported data is sliced ​​and split according to the obtained target number of rows, thereby generating multiple independent batch files. These independent batch files are then named. In this embodiment, the file naming format is "data_batch_[batch number]_[starting row number]_[ending row number]_[data type].csv" (e.g., data_batch_001_1_100000_struct.csv, data_batch_002_100001_103000_large.csv). Furthermore, the original data field names are retained at the beginning of each batch file, and a batch checksum (e.g., batch_check_sum=SHA-1(batch data content)) is added to the end, ultimately forming a batch list file in JSON format. The batch checksum is used for subsequent data integrity verification.

[0048] In this embodiment of the application, the batch manifest file records metadata information for all batch files, and the metadata words and their corresponding contents are shown in Table 1: Table 1

[0049] Step S102: Based on the producer-consumer thread model, which includes producer threads and consumer threads, the import and verification of batch files are performed in parallel. The producer thread imports the batch data into a temporary verification table and generates a verification task. The consumer thread calls a preset verification stored procedure to perform verification on the verification task and inserts the batch data from the temporary verification table into the target table after the verification is passed.

[0050] Specifically, after generating multiple batch files, the system further sends the multiple batch files to a producer-consumer thread model that includes a producer thread and a consumer thread. In the producer thread, the multiple batch files are read, the read data is imported into a temporary verification table, and a verification task is generated. Further, the consumer thread performs verification on the verification task by calling a preset verification stored procedure. After the verification is passed, the currently processed batch data is inserted from the temporary verification table into the target table to record the data verification result.

[0051] As an optional embodiment, based on the producer-consumer thread model, the batch file import and verification operations are executed in parallel, including: the producer thread adds unprocessed batch files to the blocking queue according to the batch list file and in the order of batch identifiers; from the blocking queue, the batch data is loaded into the temporary verification table through the target loading command, and the non-primary key index of the temporary verification table is closed after the import is completed; and the verification task containing the batch identifier, the temporary verification table name and the verification type is imported into the task queue.

[0052] Specifically, when the system performs batch file import and verification operations, it first adds unprocessed batch files to a blocking queue according to the batch list file and in the order of batch identifiers. The producer thread can then load the batch data into a temporary verification table using the target loading command based on the batch files in the blocking queue. The temporary verification table is configured to disable non-primary key indexes to reduce the time spent maintaining the primary key index when generating the data verification table, thereby improving data processing efficiency. In this embodiment, the temporary verification table name is preferably configured as "tmp_import_batch_[batch_id]". Further, in the producer thread, the batch representation of the batch files, the temporary verification table name, and the verification type are encapsulated and imported into the task queue, awaiting the consumer thread to trigger the verification process.

[0053] It should be noted that the temporary verification table exists independently and is used to create an independent processing environment for the data in the batch files in the producer thread. This allows the producer thread and the consumer thread to run simultaneously without interfering with each other, and the simultaneous operation of the producer thread and the consumer thread can effectively improve the data processing efficiency.

[0054] It should be noted that the non-primary key index of the temporary check table can also be stored in other temporary tables or partitioned tables to achieve the same technical effect as closing the non-primary key index. At the same time, storing the non-primary key index in other temporary tables or partitioned tables can enable the rapid reconstruction of the non-primary key index later, further reducing the data processing time.

[0055] As an optional embodiment, based on the producer-consumer thread model, the batch file import and verification operations are executed in parallel, and the consumer thread obtains the verification task from the task queue, calls the verification stored procedure in the database to perform the verification, inserts the batch data from the temporary verification table into the target table after the verification is passed, and rebuilds the relevant indexes.

[0056] Specifically, the consumer thread can call the verification stored procedure in the database to perform verification on the verification tasks in the task queue. Furthermore, it can perform a batch generation operation on the verified data, so that the verified batch data is inserted into the target table from the temporary verification table. This effectively reduces the contention for resources in the system. At the same time, the batch data is transferred within the system, which can effectively avoid frequent data transmission between the application layer and the data layer, thereby reducing network latency.

[0057] After the batch generation operation is completed, the system further performs a reconstruction operation on the non-primary key indexes. Only the index partitions involved in the current verified batch data are reconstructed, avoiding unnecessary resource consumption and reducing the time spent on index maintenance. After the non-primary key index reconstruction is complete, the system deletes the temporary verification table to release system storage resources and then retrieves new verification tasks from the task queue.

[0058] It should be noted that when the producer thread and consumer thread are running, if there are no tasks to be verified in the current system, the consumer thread is controlled to be disabled to avoid the central processing unit from idling and consuming computing resources; if the number of tasks to be verified in the current task queue reaches a preset threshold, the producer thread is controlled to be disabled to avoid too many tasks to be verified, which may lead to memory overflow and data loss.

[0059] It should be noted that the number of producer threads and consumer threads can be adjusted according to the number of CPUs. For example, with an 8-core CPU, in order to avoid resource contention caused by too many threads, two producer threads and two consumer threads can be deployed to the system, which can further improve the efficiency of data processing. In a multi-threaded scenario, multiple verification environments compete fairly and without hotspots.

[0060] It should be noted that in multi-threaded application scenarios, the number of threads to be woken up depends on the threshold number of currently runnable tasks to be verified. When the threshold number of tasks to be verified is greater than twice the number of CPU cores, only one producer thread and one consumer thread need to be woken up to avoid waking up too many verification scenarios and causing a decrease in data throughput. When the threshold number of tasks to be verified is less than or equal to the number of CPU cores or when a large amount of data floods into the system in a short period of time, all producer and consumer threads need to be woken up to work, so as to ensure the resource utilization and data processing throughput of the system at critical pressure moments, avoid task queue backlog and overall process blockage due to too few threads, and thus maintain the continuous high concurrency processing capability of the data import pipeline.

[0061] refer to Figure 4 This is a flowchart of a verification stored procedure in the database being called to perform verification, as provided in an embodiment of this application.

[0062] Step S401: Verify the format, length, and non-null constraints of the data fields to perform the first level of verification.

[0063] Specifically, the first level of validation checks the format, length, and non-null constraints of the data fields in the validation task. When validating the format and length of data fields, it is necessary to determine whether the data of any data type in the validation task meets the current data type requirement. For example, the mobile field defined as varchar(11) needs to meet the requirement of "length is 11 and contains only numbers", and the field defined as order_time needs to meet the requirement of "format is YYYY-MM-DD HH:MM:SS and is not a future time". When validating the non-null constraints of data fields, it is necessary to determine whether there are null values ​​in the data of fields defined as non-null in the validation task (such as order_id and user_id).

[0064] In the first-level verification, if a row of data in the task to be verified fails due to format, length, or non-null constraints, the corresponding row number and the reason for the verification failure are recorded, and the row of data is marked as abnormal data and stored in the abnormal data table. Further, the remaining data in the current task to be verified continues to be verified without interrupting the current task to be verified, and finally the first verification result is generated. This avoids interrupting the verification of the entire batch due to a single data error, and allows the system to complete the inspection of all data at once. If there are multiple errors in the same batch of data to be verified, a complete abnormal data table can be generated, while reducing the overhead of repeated start-up and stop, thereby shortening the total processing time of the target data.

[0065] Step S402: Verify the uniqueness of the primary key and the validity of the foreign key association of the data to perform the second-level verification.

[0066] Specifically, the second level of verification involves checking the uniqueness of the primary key and the validity of foreign key associations. When performing primary key uniqueness verification, the number of unique primary key values ​​in the first verification result is first obtained. This number is then compared with the expected number. If the obtained number is less than the expected number, it indicates that duplicate data exists in the current first verification result. Further, the specific rows where each duplicate primary key value appears are located and processed. Typically, the processing method is to retain the first occurrence of the data and mark the remaining duplicate data as abnormal. Alternatively, other processing strategies can be configured according to specific business needs, such as marking all duplicates as abnormal. Primary key uniqueness verification effectively prevents duplicate data from being recorded in the database, ensuring the continuity of data verification results.

[0067] Foreign key relationship validity validation checks whether any data in the first validation result contains foreign key constraints, ensuring the integrity of data references. Foreign key relationship validity validation can be performed using a left join query, connecting the first validation result to its related main table. Further filtering is then applied to identify rows containing records whose corresponding data cannot be found in the related main table. Data where a valid relationship cannot be established is designated as "orphan data." For example, in the relationship validation between the order table and the user table, the `user_id` field in the order table should be associated with the `id` field in the user table. After performing the relationship validation, the result will return all order records for users whose corresponding data does not exist in the user table. These orphan data are uniformly marked as abnormal data and recorded in the abnormal data table. They are then excluded from the subsequent second validation result, effectively ensuring data rigor and preventing invalid relationship data from entering the system and causing interference.

[0068] It should be noted that, in the embodiments of this application, the second-level verification can also be applied to analyze the cardinality of the unique key field by performing lightweight sampling analysis on the original imported data before confirming the first scaling factor based on whether the original imported data contains a large object field. If a high number of duplicate values ​​for a certain unique key are found during sampling, it will take a lot of time to verify it, and it can be marked as a "high-cost batch file". Similarly, the first-level verification can also be applied to clean the format of the original imported data to reduce the verification time before confirming the first scaling factor based on whether the original imported data contains a large object field.

[0069] Step S403: Verify the business logic correctness of the data according to the configurable business rule expression to perform the third-level verification.

[0070] Specifically, the third-level verification verifies the correctness of the business logic of the data based on configurable business rule expressions to obtain the verification results of the batch data, and inserts the verification results from the temporary verification table into the target table. The configurable business rule expressions support quick addition, deletion and modification.

[0071] For the third level of verification, let's take data verification scenarios in e-commerce order business scenarios and financial transaction scenarios as examples: In e-commerce business scenarios, it is necessary to verify the consistency of order data business logic to ensure that it conforms to normal business processes and rules. This requires verifying the compliance of order amount and the legality of payment status. For order amount, it is necessary to verify whether the value of the order amount (order_amount) field is greater than or equal to 0 to ensure that no negative value orders are generated, which conforms to basic financial logic. For payment status, it is necessary to verify whether the value of the payment status (pay_status) field is within the predefined set of legal statuses. For example, the preset value of 0 indicates that the user has not paid, a value of 1 indicates that the user has paid, and a value of 2 indicates that the user is in the process of getting a refund, in order to prevent abnormal or undefined status values ​​from entering the system and to ensure the correctness of the order status process.

[0072] In financial transaction scenarios, it is necessary to verify the reasonableness of transaction time and the validity of transaction amount. For transaction time, it is necessary to verify whether the difference between transaction time (transaction_time) and system time (system_time) is within a reasonable range, such as whether the difference is within five minutes before or after 5. Transaction time verification can identify and eliminate invalid transactions or future transactions caused by device malfunctions or malicious forgery, thus maintaining the accuracy of transaction timing. For transaction amount, it is necessary to verify whether the transaction amount (transaction_amount) is less than or equal to the user's current available balance. By querying the user's balance, it is possible to prevent over-trading or erroneous transactions and ensure the security of transaction funds.

[0073] Step S103: Record the processing status of each batch of data in real time to support data resume after abnormal interruption.

[0074] Specifically, during the production-consumer model operation, the processing status of each batch of data is recorded in real time, and data interruption points are resumed based on the data processing status. The data processing method provided in this application embodiment can achieve data verification while ensuring efficient data import, providing effective technical support for data consistency and scalability.

[0075] As an optional embodiment, the processing status of each batch of data is recorded in real time, including: creating a record in the breakpoint status management table for each batch of data; wherein the record includes at least batch number, file path, processing status, start time and completion time; updating the processing status during processing; wherein the processing status includes waiting to process, importing, verifying, success, and failure; when an abnormal interruption occurs, the system queries the breakpoint status management table when restarting, skips batch data with a success status, and performs recovery operations on batch data with a status of importing, verifying, or failure.

[0076] Specifically, in order to record the processing status of each batch of data in real time, the system provided in this application also creates a breakpoint status management table. The breakpoint status management table is used to record detailed status information of batch files, so as to realize real-time monitoring of the processing status of batch data and provide accurate data support for the processing of the next batch of data.

[0077] The breakpoint status management table records at least the batch number, file path, processing status, start time, and completion time for each batch of data to record the complete lifecycle of that batch. The processing status can be represented by the status field, which includes: pending (indicating the slice data is waiting to be processed); importing (indicating the slice data is being imported into the data validation table); validating (indicating the slice data is being validated); success (indicating the slice data has been successfully processed); and failed (indicating the slice data has failed to process).

[0078] In this embodiment of the application, the design of the breakpoint status management table is shown in Table 2: Table 2

[0079] If the system experiences an abnormal interruption while processing batch files, the breakpoint status management table records the final processing status of the current slice data. When the system queries the breakpoint status management table, it first skips batch files that have been successfully processed. Furthermore, it identifies batch files marked with anomalies. The system can determine the start and end line numbers of the current batch file based on the breakpoint status management table, accurately locating the breakpoint in the breakpoint status management table. This allows the target data to continue to be imported and verified from the interruption point, effectively saving processing resources and time.

[0080] According to the data processing method provided in this application embodiment, after slicing the data in the system according to preset slicing rules, the processed batch files are sent sequentially to the producer thread and the consumer thread. In the consumer thread, the processed batch files are verified according to a preset verification stored procedure. The use of sliced ​​batch files can effectively reduce the import time of large amounts of data, and the parallel processing mode of the producer thread and the consumer thread can further improve the efficiency of data processing. At the same time, the verification in the consumer process can also remove unnecessary data, avoid data quality risks, and verify different business logic scenarios during the verification process, effectively improving the scalability and practicality of data import processing.

[0081] refer to Figure 5 This is a schematic diagram of a data processing apparatus provided in an embodiment of this application.

[0082] Based on the same concept, corresponding to the data processing method provided in any of the above embodiments, this application also proposes a data processing device 500, including a slice processing module 510, a verification module 520, and a continuation module 530.

[0083] The slicing module 510 is configured to slice the original imported data based on adaptive slicing rules, generating multiple batch files; the verification module 520 is configured to perform batch file import and verification in parallel based on a producer-consumer thread model that includes producer threads and consumer threads; wherein, the producer thread imports batch data into a temporary verification table and generates verification tasks; the consumer thread calls a preset verification stored procedure to perform verification on the verification tasks, and inserts the data from the temporary verification table into the target table after the verification passes; the continuation module 530 is configured to record the processing status of each batch of data in real time to support data breakpoint continuation after abnormal interruption.

[0084] In some embodiments, the adaptive slicing rules are dynamically determined based on the field characteristics of the original imported data and the hardware performance parameters of the target server.

[0085] In some embodiments, the original imported data is sliced ​​based on adaptive slicing rules. The slicing module 510 is further configured to: obtain the hardware performance parameters of the target server; dynamically determine the target number of rows for a single batch processing based on the hardware performance parameters and the field characteristics of the original imported data, so that the estimated time for a single batch processing meets the preset duration constraint; and split the original imported data according to the target number of rows to generate multiple independent batch files.

[0086] In some embodiments, when dynamically determining the target number of rows for a single batch of processing, the slice processing module 510 is further configured to: determine a basic row number constraint for processing data within a preset time constraint in the target environment; determine a first scaling factor for the basic row number constraint based on whether the original imported data contains a large object field, and perform a first scaling process on the basic row number constraint based on the first scaling factor; determine a second scaling factor for the basic row number constraint based on hardware performance parameters, and perform a second scaling process on the basic row number constraint after processing by the first scaling factor based on the second scaling factor, so as to obtain the target number of rows.

[0087] In some embodiments, when batch file import and verification operations are performed in parallel based on the producer-consumer thread model, the verification module 520 is further configured as follows: the producer thread adds unprocessed batch files to the blocking queue according to the batch list file and in the order of batch identifiers; loads batch data from the blocking queue into a temporary verification table using a target loading command, and closes the non-primary key index of the temporary verification table after the import is completed; and imports the verification task containing the batch identifier, temporary verification table name, and verification type into the task queue.

[0088] In some embodiments, when batch file import and verification operations are performed in parallel based on the producer-consumer thread model, the verification module 520 is further configured as follows: the consumer thread obtains verification tasks from the task queue, calls the verification stored procedure in the database to perform verification, and after the verification is passed, inserts the batch data from the temporary verification table into the target table and rebuilds the relevant indexes.

[0089] In some embodiments, when a validation stored procedure in the database is invoked to perform validation, the validation module 520 is further configured to: validate the format, length, and non-null constraints of the data fields to perform a first-level validation; validate the uniqueness of the primary key and the validity of the foreign key associations of the data to perform a second-level validation; and validate the correctness of the business logic of the data according to configurable business rule expressions to perform a third-level validation.

[0090] In some embodiments, the continuation module 530, when recording the processing status of each batch of data in real time, is also configured to: create a record in the breakpoint status management table for each batch of data; wherein the record includes at least the batch number, file path, processing status, start time, and completion time; update the processing status during processing; wherein the processing status includes waiting for processing, importing, verifying, success, and failure; when an abnormal interruption occurs, the system restarts and queries the breakpoint status management table, skips the batch data with a success status, and performs a recovery operation on the batch data with a status of importing, verifying, or failure.

[0091] According to the data processing apparatus provided in the embodiments of this application, after slicing the data in the system according to preset slicing rules, the processed batch files are sent sequentially to the producer thread and the consumer thread. In the consumer thread, the processed batch files are verified according to a preset verification storage process. The use of sliced ​​batch files can effectively reduce the import time of large amounts of data, and the parallel processing mode of the producer thread and the consumer thread can further improve the efficiency of data processing. At the same time, the verification in the consumer process can also remove unnecessary data, avoid data quality risks, and verify different business logic scenarios during the verification process, effectively improving the scalability and practicality of data import processing.

[0092] Based on the same concept, corresponding to the data processing method provided in any of the above embodiments, this application also provides an electronic device, including a processor and a memory, wherein the memory stores a program or instructions that can run on the processor, and the program or instructions are executed by the processor to implement the above data processing method.

[0093] Figure 6 This illustration shows a more specific hardware structure diagram of an electronic device according to an embodiment of this application. The device may include: a processor 610, a memory 620, an input / output interface 630, a communication interface 640, and a bus 650. The processor 610, memory 620, input / output interface 630, and communication interface 640 are interconnected internally via the bus 650.

[0094] The processor 610 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0095] The memory 620 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 620 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program code is stored in the memory 620 and is called and executed by the processor 610.

[0096] The input / output interface 630 is used to connect input / output modules to enable information input and output. Input / output modules can be configured as components within the device (not shown in the figure) or externally connected to the device to provide corresponding functions. Input devices may include keyboards, mice, touchscreens, microphones, various sensors, etc., while output devices may include displays, speakers, vibrators, indicator lights, etc.

[0097] The communication interface 640 is used to connect a communication module (not shown in the figure) to enable communication between this device and other devices. The communication module can communicate via wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0098] Bus 650 includes a pathway for transmitting information between various components of the device, such as processor 610, memory 620, input / output interface 630, and communication interface 640.

[0099] It should be noted that although the above-described device only shows the processor 610, memory 620, input / output interface 630, communication interface 640, and bus 650, in specific implementations, the device may also include other components necessary for normal operation. Furthermore, those skilled in the art will understand that the above-described device may only include the components necessary for implementing the embodiments of this specification, and not necessarily all the components shown in the figures.

[0100] The electronic devices described above are used to implement the corresponding data processing methods in any of the foregoing embodiments and have the beneficial effects of the corresponding data processing method embodiments, which will not be repeated here.

[0101] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, it should be noted that the scope of the methods and apparatuses in the embodiments of this application is not limited to performing functions in the order shown or discussed, but may also include performing functions substantially simultaneously or in the reverse order, depending on the functions involved. For example, the described methods may be performed in a different order than described, and various steps may be added, omitted, or combined. Additionally, features described with reference to certain examples may be combined in other examples.

[0102] From the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of computer software products plus necessary general-purpose hardware platforms, and of course, they can also be implemented by hardware. The computer software product is stored in a storage medium (such as ROM, RAM, magnetic disk, optical disk, etc.) and includes several instructions to cause the terminal or network-side device to execute the methods described in the various embodiments of this application.

[0103] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other implementations under the guidance of this application without departing from the spirit and scope of the claims. All of these implementations are within the protection scope of this application.

Claims

1. A data processing method, characterized in that, include: Based on adaptive slicing rules, the original imported data is sliced ​​to generate multiple batch files; Based on a producer-consumer thread model that includes producer threads and consumer threads, the import and verification of the batch files are performed in parallel. The producer thread imports the batch data into a temporary verification table and generates a verification task. The consumer thread calls a preset verification stored procedure to perform verification on the verification task, and inserts the batch data from the temporary verification table into the target table after the verification is successful. Record the processing status of each batch of data in real time to support data resume after abnormal interruption.

2. The data processing method according to claim 1, characterized in that, The adaptive slicing rules are dynamically determined based on the field characteristics of the original imported data and the hardware performance parameters of the target server.

3. The data processing method according to claim 2, characterized in that, The process of slicing the original imported data based on adaptive slicing rules includes: Obtain the hardware performance parameters of the target server; Based on the hardware performance parameters and the field characteristics of the original imported data, the target number of rows for a single batch processing is dynamically determined so that the estimated time for a single batch processing meets the preset duration constraint. The original imported data is split according to the target number of rows to generate multiple independent batch files.

4. The data processing method according to claim 3, characterized in that, The dynamic determination of the target number of rows for a single batch processing includes: In the target environment, determine the basic row count constraint for processing data within the preset duration constraint; Based on whether the original imported data contains a large object field, a first scaling factor is determined for the basic row number constraint, and a first scaling process is performed on the basic row number constraint based on the first scaling factor; Based on the hardware performance parameters, a second scaling factor is determined for the basic row count constraint, and the basic row count constraint processed by the first scaling factor is subjected to a second scaling process based on the second scaling factor to obtain the target row count.

5. The data processing method according to claim 1, characterized in that, The process of parallelly executing the batch file import and verification operations based on the producer-consumer thread model includes: The producer thread adds unprocessed batch files to a blocking queue according to the batch list file and in the order of batch identifiers; it loads batch data from the blocking queue into a temporary verification table using a target loading command, and closes the non-primary key index of the temporary verification table after the import is complete; and it imports the verification task containing the batch identifier, temporary verification table name, and verification type into the task queue.

6. The data processing method according to claim 5, characterized in that, The parallel execution of the batch file import and verification operations based on the producer-consumer thread model also includes: The consumer thread retrieves the verification task from the task queue, calls the verification stored procedure in the database to perform the verification, and inserts the batch data from the temporary verification table into the target table after the verification is successful, and rebuilds the relevant indexes.

7. The data processing method according to claim 6, characterized in that, The step of calling the stored procedure for verification within the database to perform verification includes: Validate the format, length, and NOT NULL constraints of the data fields to perform the first level of validation; Verify the uniqueness of the primary key and the validity of foreign key associations in order to perform a second level of verification; The business logic of the data is validated based on configurable business rule expressions to perform a third-level validation.

8. The data processing method according to claim 1, characterized in that, The real-time recording of the processing status of each batch of data includes: Create a record in the breakpoint status management table for each batch of data; wherein the record shall contain at least the batch number, file path, processing status, start time and finish time; The processing status is updated during the processing; wherein the processing status includes waiting to be processed, importing, verifying, success, and failure. When an abnormal interruption occurs, the system will query the breakpoint status management table upon restart, skip batch data with a successful status, and perform recovery operations on batch data with a status of importing, verifying, or failing.

9. A data processing apparatus, characterized in that, include: The slicing module is configured to slice the original imported data based on adaptive slicing rules, generating multiple batch files; The verification module is configured to perform the import and verification of the batch files in parallel based on a producer-consumer thread model that includes producer threads and consumer threads. The producer thread imports the batch data into a temporary verification table and generates a verification task. The consumer thread calls a preset verification stored procedure to perform verification on the verification task and inserts the data from the temporary verification table into the target table after the verification is successful. The resume module is configured to record the processing status of each batch of data in real time to support data resume after abnormal interruption.

10. An electronic device, characterized in that, It includes a memory and a processor, the memory storing a computer program, and the processor executing the program to implement the data processing method according to any one of claims 1 to 8.

Images