A batch processing method of multiple data tables, a storage medium and an electronic device

Abstract

The present application relates to a kind of batch processing method of multiple data tables, storage medium and electronic equipment, the method comprises: reading the configuration information of multiple database tables, the configuration information includes screening condition, update condition and value mode;Receive data to be updated, according to the screening condition of the configuration information, all key element data are screened out, and the key element data is loaded once;According to the update condition and value mode, the key element data is spliced into update statement, and batch creation update script is created, and the update script created in batch is added to execution queue;The update script in the execution queue is submitted to database execution.Through pre-configuration configuration information of multiple database tables and screening to be updated data according to configuration rule, update script is automatically generated, and update script is uniformly executed by database, and the data update in multiple database tables can be efficiently and quickly realized.

Description

Technical Field

[0001] This invention relates to the field of database table processing, and more specifically to a batch processing method, storage medium, and electronic device for multiple data tables. Background Technology

[0002] With the development of technology, all kinds of Internet applications have emerged. The realization of these application functions is mostly inseparable from data interaction. Data reception and processing are essential basic functions of applications. Being able to process complex data quickly and efficiently is an important foundation for improving performance.

[0003] Currently, when updating data in a database table, the process typically involves receiving the data to be updated and then processing each table separately according to its different data structure. Each table requires separate coding, and when the data structure changes, corresponding database operation code needs to be written for each table that needs to be updated. This process is cumbersome, inefficient, and lacks flexibility. Summary of the Invention

[0004] In view of the above problems, the present invention provides a batch processing method, storage medium and electronic device for multiple data tables, which solves the problem of low accuracy of existing deep learning models for detecting fake faces.

[0005] To achieve the above objectives, in a first aspect, the present invention provides a batch processing method for multiple data tables, comprising the following steps:

[0006] S1: Read the configuration information of multiple database tables, including filtering conditions, update conditions, and value retrieval methods;

[0007] S2: Receive the data to be updated, filter out all key element data according to the filtering conditions of the configuration information, and load the key element data at once;

[0008] S3: Concatenate the key element data into update statements according to the update conditions and value retrieval methods, create update scripts in batches, and add the batch-created update scripts to the execution queue;

[0009] S4: Submit the update script in the execution queue to the database for execution.

[0010] As an optional embodiment, the method further includes the following steps:

[0011] After each receipt of the data to be updated, the data to be updated is saved to the file server as a file;

[0012] Based on the primary key corresponding to the received data to be updated, set the file format version for this push, determine whether the file server already stores a higher version of the file format version for this push, if so, end the current process, otherwise continue to execute the subsequent steps, and set a success flag for this process after the subsequent steps are executed successfully, and record the failure information if it fails.

[0013] As an optional embodiment, the method further includes:

[0014] Set preset retry parameters, which include preset number of retries and / or preset retry interval time;

[0015] If the subsequent steps fail, an exception log message is output, and the subsequent steps are re-executed until the parameters for re-execution meet the preset retry parameters.

[0016] As an optional embodiment, before filtering all key element data according to the filtering conditions of the configuration information and loading the key element data all at once, the following steps are further included:

[0017] Different implementation classes are set up for data of different business types. After receiving the data to be updated, the data to be updated is assigned to the corresponding implementation class according to the business type of the data to be updated, and the data to be updated is converted into the corresponding structured data in the database.

[0018] As an optional embodiment, after batch creation of the update scripts, the following steps are also included:

[0019] The update script is verified, and if the verification is successful, the update script is added to the execution queue; otherwise, the update script is not added to the execution queue.

[0020] As an optional embodiment, the method further includes:

[0021] According to the update conditions, the data to be updated is filtered out and the data to be added is selected. The class of the corresponding database table entity is obtained from the configured mapper class. Then, the data to be added is converted into an entity class object through JSON. Based on the entity class object, the corresponding processing method of the database is called to generate an insert statement. The insert statement is then submitted to the database for execution.

[0022] As an optional embodiment, the method further includes:

[0023] Operations unrelated to data entry are managed by an event listener. Upon receiving an event notification, the event listener executes the corresponding processing strategy. The event listener is decoupled from the database and operates independently.

[0024] As an optional embodiment, the method further includes:

[0025] When a missing key element data is detected in a certain business process, after the missing key element data is supplemented, the valid data pushed in all subsequent business processes is detected and data compensation is performed one by one.

[0026] The valid data is associated with the key element data, and the data compensation includes: supplementing the missing data in the valid data based on the key element data after the supplementation is completed.

[0027] In a second aspect, the present invention also provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the method described in the first aspect.

[0028] In a third aspect, the present invention also provides an electronic device including a memory and a processor, the memory being used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method described in the first aspect.

[0029] Unlike existing technologies, the above technical solution involves a batch processing method, storage medium, and electronic device for multiple database tables. The method includes: reading configuration information from multiple database tables, where the configuration information includes filtering conditions, update conditions, and value retrieval methods; receiving data to be updated, filtering all key element data according to the filtering conditions in the configuration information, and loading the key element data all at once; concatenating the key element data into update statements according to the update conditions and value retrieval methods, creating update scripts in batches, and adding the batch-created update scripts to an execution queue; and submitting the update scripts in the execution queue to the database for execution. By pre-configuring the configuration information of multiple database tables and filtering the data to be updated according to configuration rules, automatically generating update scripts, and then having the database execute the update scripts uniformly, data updates across multiple database tables can be achieved efficiently and quickly.

[0030] The above description of the invention is merely an overview of the technical solution of the present invention. In order to enable those skilled in the art to better understand the technical solution of the present invention and to implement it based on the description and drawings, and to make the above-mentioned objectives and other objectives, features and advantages of the present invention easier to understand, the following description is provided in conjunction with the specific embodiments and drawings of the present invention. Attached Figure Description

[0031] The accompanying drawings are only used to illustrate the principles, implementation methods, applications, features, and effects of specific embodiments of the present invention and other related contents, and should not be considered as limitations on the present invention.

[0032] In the accompanying drawings of the instruction manual:

[0033] Figure 1 A flowchart illustrating a batch processing method for multiple data tables according to a first exemplary embodiment of the present invention;

[0034] Figure 2 A flowchart illustrating a batch processing method for multiple data tables according to a second exemplary embodiment of the present invention;

[0035] Figure 3 A flowchart illustrating a batch processing method for multiple data tables according to a third exemplary embodiment of the present invention;

[0036] Figure 4 A flowchart illustrating a batch processing method for multiple data tables according to a fourth exemplary embodiment of the present invention;

[0037] Figure 5 This is a schematic diagram of an electronic device according to an exemplary embodiment of the present invention.

[0038] The reference numerals used in the above figures are explained as follows:

[0039] 1. Electronic equipment;

[0040] 11. Memory;

[0041] 12. Processor. Detailed Implementation

[0042] To illustrate the possible application scenarios, technical principles, implementable specific solutions, and achievable objectives and effects of this invention in detail, the following description, in conjunction with the listed specific embodiments and accompanying drawings, provides a detailed explanation. The embodiments described herein are merely illustrative of the technical solutions of this invention and are therefore intended only as examples, not as limiting the scope of protection of this invention.

[0043] In this document, the term "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The term "embodiment" appearing in various places throughout the specification does not necessarily refer to the same embodiment, nor does it specifically limit its independence or connection with other embodiments. In principle, in this invention, as long as there are no technical contradictions or conflicts, the technical features mentioned in each embodiment can be combined in any way to form corresponding implementable technical solutions.

[0044] Unless otherwise defined, the technical terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains; the use of related terms herein is merely for the purpose of describing particular embodiments and is not intended to limit the invention.

[0045] In the description of this invention, the term "and / or" is used to describe the logical relationship between objects, indicating that three relationships can exist. For example, A and / or B means: A exists, B exists, and A and B exist simultaneously. Additionally, the character " / " generally indicates that the preceding and following objects have an "or" logical relationship.

[0046] In this invention, terms such as “first” and “second” are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any actual quantity, hierarchy, or order between these entities or operations.

[0047] Without further limitations, the use of terms such as “comprising,” “including,” “having,” or other similar open-ended expressions in this invention is intended to cover non-exclusive inclusion, which does not exclude the presence of additional elements in a process, method, or product that includes the stated elements, such that a process, method, or product that includes a list of elements may include not only those defined elements but also other elements not expressly listed, or elements inherent to such a process, method, or product.

[0048] Similar to the understanding in the Examination Guidelines, in this invention, expressions such as "greater than," "less than," and "exceeding" are understood to exclude the stated number; expressions such as "above," "below," and "within" are understood to include the stated number. Furthermore, in the description of the embodiments of this invention, "multiple" means two or more (including two), and similar expressions related to "multiple" are also understood in this way, such as "multiple groups" and "multiple times," unless otherwise explicitly specified.

[0049] Please see Figure 1 The above is a flowchart of a batch processing method for multiple data tables according to a first exemplary embodiment of the present invention. The method includes the following steps:

[0050] S1: Read the configuration information of multiple database tables, including filtering conditions, update conditions, and value retrieval methods;

[0051] S2: Receive the data to be updated, filter out all key element data according to the filtering conditions of the configuration information, and load the key element data at once;

[0052] S3: Concatenate the key element data into update statements according to the update conditions and value retrieval methods, create update scripts in batches, and add the batch-created update scripts to the execution queue;

[0053] S4: Submit the update script in the execution queue to the database for execution.

[0054] In this embodiment, for the general configuration of each data table, a certain conditional expression is preset for each data object in the database table. During data processing, a specified element in the data source is extracted according to this expression and parsed into the expression. If the conditional expression is met (i.e., the filtering condition is satisfied), the data object is processed; otherwise, subsequent steps are skipped. For example, when the procurement method is online bidding, data related to the online bidding rules is processed; otherwise, the data object is skipped.

[0055] In this embodiment, each database table typically stores a large amount of data, which may belong to different users. Therefore, update conditions need to be configured to prevent accidental manipulation of other data. Specifically, the update conditions are configured as follows: the update scope is determined based on the primary key or certain related fields of each database table, and the value retrieval method for each field is configured. These values ​​are then extracted from the received data (i.e., the data to be updated), and finally concatenated into a complete update statement. The primary key or certain related fields of the database table include project ID or plan ID.

[0056] In this embodiment, the value retrieval method refers to the value retrieval method when each data table performs an insert operation, which is used to extract the data to be inserted from the received data (i.e., the data to be updated).

[0057] In this embodiment, the configuration information also includes the identification information of the database tables. Different database tables have different identification information, which can be letters, numbers, strings, etc. In other embodiments, the configuration information also includes the abstract class name for database operation processing corresponding to each database table. Specifically, a database connector (such as SqlSessionFactory) can be configured to create a database session. Then, the processing object corresponding to each table can be obtained based on the configured class name. Update and insert operations both rely on this object to complete.

[0058] The above solution pre-configures the configuration information of multiple database tables and filters the data to be updated according to the configuration rules, automatically generates update scripts, and then the database executes the update scripts uniformly. When there are new database table operation requirements in the future, only rules need to be added in the configuration to realize various data processing actions, which can efficiently and quickly realize data updates in multiple database tables.

[0059] Please see Figure 2 The flowchart below shows a batch processing method for multiple data tables according to a second exemplary embodiment of the present invention. The method includes the following steps:

[0060] First, proceed to step S201: after receiving the data to be updated each time, save the data to be updated to the file server in the form of a file;

[0061] Then proceed to step S202 to set the file format version of this push according to the primary key corresponding to the data to be updated received this time;

[0062] Then, proceed to step S203 to determine whether the file server already stores data of a higher version of the file format pushed this time. If so, proceed to step S204 to end this process. Otherwise, proceed to step S205 to continue executing the subsequent steps. If the subsequent steps are executed successfully, set the success flag for this process. If it fails, record the failure information.

[0063] In this embodiment, the subsequent steps refer to "filtering all key element data according to the filtering conditions of the configuration information and loading the key element data at once" in step S2, as well as steps S3 and S4. The key element data refers to the data that needs to be executed with corresponding operation instructions after being filtered by the filtering conditions. The operation instructions include adding, deleting, modifying (or replacing), and querying the data.

[0064] In this embodiment, the updated data is stored in file format. The database only stores the file's storage address. Each file format has a corresponding matching version, and versions are upgraded sequentially based on time. Setting the file format version based on the primary key corresponding to the received data to be updated means storing and backing up the data to be updated as a file and recording the file storage path in a log table. If a set of data to be updated has a higher version, and the update is still performed, the updated data will be replaced by the higher version in subsequent processing, rendering the update worthless. Therefore, ending the process promptly in this case can effectively save system computing resources. Conversely, if the updated data version is the latest version, subsequent steps continue, and a success flag is set upon successful execution of subsequent steps. If a failure occurs, failure information is recorded, allowing technicians to promptly know whether the update operation has been processed.

[0065] Please see Figure 3 The flowchart below shows a batch processing method for multiple data tables according to a third exemplary embodiment of the present invention. The method includes the following steps:

[0066] First, proceed to step S301 to set preset retry parameters, which include preset retry count and / or preset retry interval time;

[0067] Then, in step S302, if the subsequent steps fail, an exception log message is output, and the subsequent steps are re-executed until the parameters for re-execution meet the preset retry parameters.

[0068] In this embodiment, the preset number of retries and / or the preset retry interval can be set according to actual needs. Any exceptions during this processing will also be registered in the data version and an exception log will be output for subsequent troubleshooting and continuous improvement of data processing capabilities.

[0069] In some embodiments, before filtering all key element data according to the filtering conditions of the configuration information and loading the key element data all at once, the method further includes the step of setting different implementation classes for data of different business types. The implementation class refers to a dedicated processing method for handling a specific type of business. After receiving the data to be updated, it is assigned to the corresponding implementation class according to the business type of the data to be updated, and the data to be updated is converted into the corresponding structured data in the database. In this way, through preprocessing of the data to be updated, the received data to be updated can be selectively divided, and then different processing methods can be used to process the corresponding data in a targeted manner, improving data processing efficiency.

[0070] Since the current database table may contain data that has been replaced, in order to preserve the previous state of the data and record its evolution, it is necessary to perform historical processing on the current valid data. Therefore, in some embodiments, after batch creation of update scripts, the following steps are also included: verifying the update scripts, and adding the update scripts to the execution queue if the verification is successful; otherwise, not adding the update scripts to the execution queue.

[0071] Specifically, update conditions and value retrieval methods for each database table can be pre-defined in the configuration information. Then, update statements are constructed based on these conditions and values. If not all conditions for a particular table are met, the system checks if the underlying primary key condition (such as a project ID) is present. If it is, the corresponding update script is added to the execution queue; otherwise, it is not. This method enables batch submission and execution of update scripts, reducing the number of database interactions, significantly improving system performance, and reducing processing time.

[0072] To enable the batch import of large amounts of data into the database in the shortest possible time, the method further includes: reading the data from the data source and, according to the abstract methods in the corresponding configuration, transforming the complex data structure into database table entity objects. Specifically, it involves locating and retrieving the required dataset from the data source based on the configuration, and then converting the dataset into corresponding Java objects. Finally, it calls the database processing method to generate insert statements and submits them to the database for batch execution. During this process, data type conversion, data cleaning, and data normalization are required.

[0073] Specifically, the process involves reading data from the data source, transforming complex data structures into database table entity objects according to the abstract methods in the corresponding configuration, calling database processing methods to generate insert statements, and submitting them to the database in batches for execution.

[0074] According to the update conditions, the data to be updated is filtered out and the data to be added is selected. The class of the corresponding database table entity is obtained from the configured mapper class. Then, the data to be added is converted into an entity class object through JSON. Based on the entity class object, the corresponding processing method of the database is called to generate an insert statement. The insert statement is then submitted to the database for execution.

[0075] In some embodiments, the method further includes managing operations unrelated to data entry within an event listener. Upon receiving an event notification, the event listener executes the corresponding processing strategy (e.g., automatically creating an online discussion group (group chat) after project creation, or automatically forwarding announcement data to the expert database upon receipt). The event listener is decoupled from the database and runs independently. This approach reduces code coupling and increases scalability, leaving more room for subsequent data reuse.

[0076] When an application needs to receive and process data sequentially in stages, if a preceding stage pushes data repeatedly, the data status of subsequent stages will be overwritten. For example, the service receives project data pushed by the bidding application with the data version in the "Document Compilation" state; at this point, the data status is "Document Compilation." Then, the bidding application pushes data for the same project with the status "Bidding Open," updating the data status to "Bidding Open." If the bidding application pushes data for the same project again, the data status will be overwritten as "Document Compilation."

[0077] In the above example, data overwriting occurred. To address this issue, this application also designs an automatic data compensation scheme, specifically including: when a missing key element data in a certain business process is detected, after the missing key element data is supplemented, the valid data pushed in all subsequent business processes is detected, and data compensation is performed one by one; the valid data is associated with the key element data, and the data compensation includes: supplementing the missing data in the valid data based on the supplemented key element data.

[0078] Typically, data for a bidding project is multi-dimensional, generated chronologically, and often overlaps between dimensions. If subsequent dimensions already exist after the data for a preceding dimension is pushed, they need to be overwritten. The preceding and subsequent dimensions constitute the "valid data" mentioned earlier. The association between valid data and key element data can be understood as follows: key element data is part of or consistent with valid data, and can be obtained by filtering based on the association ID of the valid data.

[0079] For example, if a project has completed the bidding process but some data is missing and needs to be supplemented, the bidding service pushes the data for the bidding stage. The system automatically detects the data for subsequent stages, such as bid opening, bid evaluation, and bid awarding. After processing the bidding data, it automatically supplements the data for the subsequent stages in sequence.

[0080] In this way, by querying the valid data pushed in all subsequent stages based on the key elements of this data push, and compensating for the data one by one according to the application order, the data status can be restored, thus avoiding data overwriting issues and ensuring data integrity.

[0081] like Figure 4 The diagram shown is a flowchart of a batch processing method for multiple data tables according to a fourth exemplary embodiment of the present invention. The method includes the following steps:

[0082] The system pre-configures the settings. Upon receiving data (the data to be updated), it reads the general configuration, creates a data version, and checks if the database version is the latest. If it is, it proceeds to the processing flow, which includes: data preprocessing, creating a data change record, data import, data compensation, and publishing the processing result event. If the database version is not the latest, the processing of this data is skipped. If the data processing is successful, a success message is output and the current processing ends. If processing fails, exception information is captured, and the process is retried. The system checks if the preset number of retries has been reached. If not, the processing flow is re-executed; if it has been reached, the current data processing ends.

[0083] In a second aspect, this embodiment also provides a computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, implement the method described in the first aspect.

[0084] Please see Figure 5 In a third aspect, this embodiment also provides an electronic device 1, including a memory 11 and a processor 12, wherein the memory 11 is used to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor 12 to implement the method described in the first aspect.

[0085] The electronic device 1 can be a tablet, mobile phone, laptop, desktop computer, etc., and the storage medium / memory 11 includes, but is not limited to: RAM, ROM, magnetic disk, magnetic tape, optical disk, flash memory, USB flash drive, portable hard drive, memory card, memory stick, network server storage, network cloud storage, etc. The processor 12 includes, but is not limited to, CPU (Central Processing Unit 12), GPU (Graphics Processing Unit 12), MCU (Microprocessor 12), etc.

[0086] The above technical solution can fully aggregate diverse facial feature information, effectively enhancing the expressive power of facial features and thus improving the accuracy of fake face detection. By adding a spatial-frequency domain feature fusion module, the spatial information of the image is fully utilized. At the same time, the forgery traces generated by upsampling operations during the fake face generation process can be observed through frequency domain information, achieving multi-dimensional information aggregation. The variant attention module can obtain the optimal feature representation through enhanced attention operations, greatly improving the generalization of fake face detection and its robustness against various unknown tampering methods in real-world scenarios, thus improving the model's discrimination ability and enabling it to maintain good performance in real-world scenarios.

[0087] Finally, it should be noted that although the above embodiments have been described in the description and drawings of this invention, this should not limit the scope of patent protection of this invention. Any technical solutions that are based on the essential concept of this invention, utilize the content described in the description and drawings of this invention to make equivalent structural or procedural substitutions or modifications, as well as the direct or indirect application of the technical solutions of the above embodiments to other related technical fields, are all included within the scope of patent protection of this invention.

Claims

1. A batch processing method for multiple data tables, characterized in that, The method comprises the following steps: S1: reading the configuration information of multiple database tables, the configuration information comprising filtering conditions, update conditions and value modes; S2: receiving to-be-updated data, screening all key element data according to the filtering conditions of the configuration information, and loading the key element data at one time; S3: splicing the key element data into update statements according to the update conditions and value modes, creating update scripts in batches, and adding the update scripts created in batches to an execution queue; S4: submitting the update scripts in the execution queue to a database for execution; The method further comprises the following steps: After receiving the to-be-updated data each time, saving the to-be-updated data in a file form on a file server; Setting a file form version of this time of pushing according to a primary key corresponding to the to-be-updated data received this time, judging whether there is a version data of a higher level than the file form version of this time of pushing stored on the file server, ending this time of processing if yes, otherwise, continuing to execute subsequent steps, and setting a this-time-of-processing-successful identifier after the subsequent steps are executed successfully, and recording failure information if it fails; The method further comprises: Screening warehousing data from the to-be-updated data according to the update conditions, obtaining a class class corresponding to a database table entity in a configured mapper class, then converting the warehousing data into an entity class object through json, calling a processing method corresponding to the database based on the entity class object to generate an insertion statement, and submitting the insertion statement to the database for execution; The method further comprises: When it is detected that the key element data of a certain business link is missing, detecting valid data pushed in all links after the current business link one by one and performing data compensation after the missing key element data is pushed; The valid data is associated with the key element data, and the data compensation comprises supplementing missing data in the valid data according to the key element data after the pushing is completed.

2. The batch processing method of multiple data tables of claim 1, wherein, The method further comprises: Setting preset retry parameters, the preset retry parameters comprising a preset number of retries and / or a preset retry interval; When the subsequent steps fail, outputting abnormal log information, and re-executing the subsequent steps until the parameters of the re-execution meet the preset retry parameters.

3. The batch processing method of multiple data tables of claim 1, wherein, Before screening all key element data according to the filtering conditions of the configuration information and loading the key element data at one time, the method further comprises the following steps: Setting different implementation classes for data of different business types, after receiving the to-be-updated data, distributing the to-be-updated data to the corresponding implementation classes according to the business types of the to-be-updated data, and converting the to-be-updated data into corresponding structure data in the database.

4. The batch processing method of multiple data tables of claim 1, wherein, After creating the update scripts in batches, the method further comprises the following steps: Verifying the update scripts, and adding the update scripts to the execution queue after the verification passes, otherwise, not adding the update scripts to the execution queue.

5. The batch processing method of multiple data tables of claim 1, wherein, The method further comprises: Operations irrelevant to data warehousing are uniformly put into an event listener for management, the event listener executes a processing strategy corresponding to the event after receiving an event notification, and the event listener and the database are decoupled and independently run.

6. A computer readable storage medium having stored thereon computer program instructions, wherein, The computer program instructions, when executed by a processor, implement the method of any one of claims 1-5.

7. An electronic device comprising a memory and a processor, characterized in that The memory is configured to store one or more computer program instructions, wherein the one or more computer program instructions are executed by the processor to implement the method of any one of claims 1-5.

Images