Multi-table universal efficient data processing method based on batch processing

Abstract

The invention belongs to the field of data processing, and particularly relates to a batch processing-based multi-table universal efficient data processing method, which comprises the following steps of S1, data acquisition and distribution; s2, data preprocessing and insertion data construction; s3, extracting and processing batch data; and S4, batch data insertion and log recording. According to the method, four dimensions of visual configuration, full-process automation, data processing refinement and batch high efficiency are concentrated, through systematic process design and technical optimization, the manual intervention cost and dependence on bottom layer coding are greatly reduced, and synchronous upgrading of performance, stability and operation and maintenance efficiency is achieved in a large-scale data processing scene; in the aspect of a configuration mode, the scheme breaks through the limitation that traditional data processing depends on coding, and a visual operation interface is provided; on the level of flow operation, a full-closed-loop automatic system of data distribution, preprocessing, batch extraction and insertion landing is constructed in the scheme.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a high-efficiency data processing method based on batch processing that is applicable to multiple tables. Background Technology

[0002] In traditional database operations, each table requires writing related SQL for write, update, and delete operations, which can be done for some data. The informatization of hospital equipment management, fault prediction, and maintenance has certain limitations in terms of efficiency, accuracy, and real-time performance, and cannot adapt to the rapidly developing medical technologies and demands. The following is the current situation and some of its shortcomings: (1) Low processing efficiency: When the amount of data is around one million during the data docking process, the data processing and storage speed is slow, which affects the use of equipment data; (2) High resource consumption: Traditional systems usually cannot provide real-time monitoring of equipment status, making it difficult to respond quickly to abnormal equipment conditions, which increases the time required to handle faults; (3) Poor flexibility of multi-table operations: Traditional tables require writing related SQL statements when writing data; (4) Inefficient single-table processing mode: Traditional systems have difficulty predicting equipment maintenance needs in advance and cannot take measures before equipment failure occurs, resulting in low maintenance efficiency; To address the issues raised above, we propose a high-efficiency data processing method that is applicable to multiple tables based on batch processing. Summary of the Invention

[0003] This invention proposes a high-efficiency data processing method based on batch processing that is applicable to multiple tables, in order to solve the problems mentioned in the background art.

[0004] To achieve the above objectives, the present invention adopts the following technical solution: A batch-processing-based, multi-table, high-efficiency data processing method includes the following steps: S1: Data acquisition and distribution; S2: Data preprocessing and insertion data construction, which mainly handles data cleaning, transformation and field binding, providing a standardized data structure for subsequent batch data processing and insertion; S3: Batch data extraction and processing, which is used to extract data in batches, standardize the data, and record abnormal data. S4: Batch data insertion and logging, which is used to efficiently write cleaned data into the target database in batches and to perform full-process log tracking.

[0005] Preferably, in step S1, data acquisition and distribution specifically includes the following steps: S101: Configure the data source, including request method, database type, request parameters, and data source information, to obtain the data provided by the hospital. S102: Configure table mapping to support mapping the table structure in the data source to the data model of the target system. The purpose is to achieve structural alignment and semantic unity between the source data structure and the target system data model. Users can perform field-level mapping and renaming through the interface. S103: Implement a data splitting strategy, which can dynamically allocate data based on table name or custom rules and assign it to the corresponding target business table. The data separation strategy will also automatically match data from different data sources to the target business table, providing a structured foundation for subsequent data cleaning and warehousing processing.

[0006] Preferably, in step S101, the request method configuration is achieved through a visual system configuration page, enabling unified access and management of multiple types of data sources. The data source type in the database type configuration includes, but is not limited to, relational databases, distributed file systems, and non-relational databases; The data source information configuration allows users to complete the filling and verification of the above data source connection parameters in the system's backend visual configuration interface, including IP address, port number, username, password, file path, access protocol, and authentication method. The data source information configuration also supports configuring data update frequency parameters to adapt to the data update needs of different business scenarios.

[0007] Preferably, in step S2, the data preprocessing and insertion data construction includes the following steps: S201: Data Cleaning Rule Definition: The system uses a configurable cleaning rule definition module to achieve automated quality control and standardized processing of source data. It also supports multi-dimensional cleaning strategy configuration, including missing value handling, outlier identification and correction, and format standardization, to ensure the integrity and consistency of data before it enters subsequent processing stages. S202: Data Conversion Rule Definition; The system provides data conversion functions such as field concatenation, numerical calculation, encoding conversion, and multi-field mapping, and supports field-level calculation and mapping rule definition, thereby meeting the data structure integration and semantic fusion needs in complex business scenarios; S203: Data Object Construction: After data cleaning and transformation, the system enters the data object construction phase, which generates data carrier objects that can be directly used for batch insertion. Additionally, the system uses a Map-based approach.<Key, Value> The system uses a key-value mapping data structure to construct intermediate data objects. During the object construction process, the system fills the source data into the corresponding target fields according to the field mapping relationship. If the field defines a default value or calculation rule in the mapping relationship, the system will execute the calculation logic and fill the result at this stage to ensure that the generated data object is completely consistent with the target table structure.

[0008] Preferably, in step S203, Key represents the field name of the target table, and Value represents the field value after processing by the cleaning and transformation rules.

[0009] Preferably, in step S3, the batch data extraction and processing includes the following steps: S301: Preliminary verification: During the data extraction stage, the system performs a dual verification mechanism of the structure layer and content layer on the source data to ensure the integrity and accuracy of the data before it enters the processing flow; S302: Deep cleaning and transformation: Determine whether the current data already exists in the database, and perform duplicate data detection and deduplication to ensure that the data that finally enters the batch storage stage has high consistency and high quality.

[0010] Preferably, in step S301, the dual-layer verification mechanism includes structural verification and content verification, which provides a preliminary guarantee for data quality and ensures that the data has structural correctness and content legality before entering the cleaning and transformation stage, thereby improving the overall data processing stability and traceability of the system.

[0011] Preferably, in step S302, the deduplication strategy can be executed based on the primary key, unique index, or business logic field, retaining the latest or valid record, or an update field can be set.

[0012] Preferably, in step S4, the batch data insertion and log recording includes the following steps: S401: Batch processing strategy definition: Users can define a single batch processing threshold in the system. The system temporarily stores the cleaned and transformed data in the memory cache. When the cumulative data volume reaches the set threshold, the batch insertion operation is automatically triggered. At the same time, the system has a built-in transaction control mechanism in the batch processing process. When any batch of data encounters an exception during the insertion stage, the transaction rollback will be automatically triggered to ensure the consistency and atomicity of all data in the batch and prevent data incompleteness caused by the successful writing of some data. S402: Batch Insertion and Dynamic SQL Generation Mechanism: Before performing the insertion operation, the system will automatically generate a compatible batch insert statement template based on the target database type and intelligently adapt to the syntax differences of different databases. During the data insertion stage, the system dynamically concatenates the field names and field value lists by iterating through the Map collection using foreach, thereby automatically assembling the SQL statement. In addition, when performing batch insertion, the system will uniformly assemble multiple records according to the batch processing threshold configured above and write them using a batch pre-compilation method, thereby effectively reducing the number of database interactions. S403: Log Recording and Monitoring: Throughout the entire data processing and insertion process, the system has a built-in comprehensive log recording and operation monitoring mechanism to ensure the traceability of task execution and the visualized management of system operation and maintenance.

[0013] Preferably, in step S401, the system can dynamically adjust the batch processing threshold according to the runtime load to achieve adaptive optimization for different data volumes and network environments.

[0014] Compared with existing technologies, this invention focuses on four dimensions: visualized configuration, full-process automation, refined data processing, and efficient batch processing. Through systematic process design and technology optimization, it significantly reduces the cost of manual intervention and dependence on underlying coding, achieving simultaneous upgrades in performance, stability, and operational efficiency in large-scale data processing scenarios. In terms of configuration mode, the solution breaks through the limitations of traditional data processing that relies on coding, providing an intuitive visual operation interface. At the process operation level, the solution constructs a fully closed-loop automated system of "data diversion - preprocessing - batch extraction - insertion and deployment". Attached Figure Description

[0015] Figure 1 This is a flowchart of a high-efficiency data processing method for multiple tables based on batch processing proposed in this invention; Figure 2 This is a flowchart of data acquisition and distribution in a batch-based, multi-table general-purpose, high-efficiency data processing method proposed in this invention; Figure 3 This is a flowchart illustrating the data preprocessing and data insertion construction in a batch-based, multi-table, high-efficiency data processing method proposed in this invention. Figure 4 This is a flowchart of batch data extraction and processing in a multi-table general-purpose efficient data processing method based on batch processing proposed in this invention; Figure 5 This is a flowchart of batch data insertion and log recording in a batch processing-based, multi-table, high-efficiency data processing method proposed in this invention. Detailed Implementation

[0016] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0017] Reference Figure 1-5 A high-efficiency data processing method applicable to multiple tables based on batch processing includes the following steps: S1: Data acquisition and distribution, which specifically includes the following steps: S101: Configure the data source, including request method, database type, request parameters, and data source information, to obtain the data provided by the hospital. S102: Configure table mapping to support mapping the table structure in the data source to the data model of the target system. The purpose is to achieve structural alignment and semantic unity between the source data structure and the target system data model. Users can perform field-level mapping and renaming through the interface. S103: Implement a data splitting strategy, which can dynamically allocate data based on table name or custom rules and assign it to the corresponding target business table. The data separation strategy will also automatically match data from different data sources to the target business table, providing a structured foundation for subsequent data cleaning and warehousing. S2: Data preprocessing and insertion data construction, mainly handles data cleaning, transformation, and field binding, providing a standardized data structure for subsequent batch data processing and insertion. It includes the following steps: S201: Data Cleaning Rule Definition: The system uses a configurable cleaning rule definition module to achieve automated quality control and standardized processing of source data. It also supports multi-dimensional cleaning strategy configuration, including missing value handling, outlier identification and correction, and format standardization, to ensure the integrity and consistency of data before it enters subsequent processing stages. S202: Data Conversion Rule Definition; The system provides data conversion functions such as field concatenation, numerical calculation, encoding conversion, and multi-field mapping, and supports field-level calculation and mapping rule definition, thereby meeting the data structure integration and semantic fusion needs in complex business scenarios; S203: Data Object Construction: After data cleaning and transformation, the system enters the data object construction phase, which generates data carrier objects that can be directly used for batch insertion. Additionally, the system uses a Map-based approach.<Key, Value> The system uses a key-value mapping data structure to construct intermediate data objects, where the key represents the field name of the target table and the value represents the field value after cleaning and transformation rules. During the object construction process, the system fills the source data into the corresponding target fields according to the field mapping relationship. If the field defines a default value or calculation rule in the mapping relationship, the system will execute the calculation logic and fill the result at this stage to ensure that the generated data object is completely consistent with the target table structure. S3: Batch data extraction and processing, which is used to extract data in batches, perform normalization, and record abnormal data. It includes the following steps: S301: Preliminary verification: During the data extraction stage, the system performs a dual verification mechanism of structure layer and content layer on the source data to ensure the integrity and accuracy of the data before it enters the processing flow. The dual verification mechanism includes structure verification and content verification, which realizes the pre-guarantee of data quality and ensures that the data has the correct structure and legal content before entering the cleaning and transformation stage, thereby improving the overall data processing stability and traceability of the system. S302: Deep cleaning and transformation: Determine whether the current data already exists in the database, and perform duplicate data detection and deduplication. The deduplication strategy can be based on the primary key, unique index or business logic field, retaining the latest or valid records, or setting update fields, thereby ensuring that the data that finally enters the batch storage stage has high consistency and high quality. S4: Batch Data Insertion and Log Recording, used to efficiently write cleaned data into the target database in batches and perform full-process log tracking, including the following steps: S401: Batch processing strategy definition: Users can define a single batch processing threshold in the system. The system temporarily stores the cleaned and transformed data in the memory cache. When the cumulative data volume reaches the set threshold, the batch insertion operation is automatically triggered. At the same time, the system has a built-in transaction control mechanism in the batch processing process. When any batch of data encounters an exception during the insertion stage, the transaction rollback will be automatically triggered to ensure the consistency and atomicity of all data in the batch and prevent data incompleteness caused by the successful writing of some data. S402: Batch Insertion and Dynamic SQL Generation Mechanism: Before performing the insertion operation, the system will automatically generate a compatible batch insert statement template based on the target database type and intelligently adapt to the syntax differences of different databases. During the data insertion stage, the system dynamically concatenates the field names and field value lists by iterating through the Map collection using foreach, thereby automatically assembling the SQL statement. In addition, when performing batch insertion, the system will uniformly assemble multiple records according to the batch processing threshold configured above and write them using a batch pre-compilation method, thereby effectively reducing the number of database interactions. S403: Log Recording and Monitoring: Throughout the entire data processing and insertion process, the system has a built-in comprehensive log recording and operation monitoring mechanism to ensure the traceability of task execution and the visualized management of system operation and maintenance.

[0018] Specifically, in S101 of this embodiment, the request method configuration is achieved through a visual system configuration page to realize unified access and management of multiple types of data sources; The data source type in the database type configuration includes, but is not limited to, relational databases, distributed file systems, and non-relational databases; The data source information configuration allows users to complete the filling and verification of the above data source connection parameters in the system's backend visual configuration interface, including IP address, port number, username, password, file path, access protocol, and authentication method. The data source information configuration also supports configuring data update frequency parameters to adapt to the data update needs of different business scenarios.

[0019] Furthermore, in S401 of this embodiment, the system can dynamically adjust the batch processing threshold according to the runtime load to achieve adaptive optimization for different data volumes and network environments.

[0020] The core advantages of this embodiment lie in four dimensions: visualized configuration, full-process automation, refined data processing, and high-efficiency batch processing. Through systematic process design and technical optimization, it significantly reduces the cost of manual intervention and dependence on underlying coding, achieving simultaneous upgrades in performance, stability, and operational efficiency in large-scale data processing scenarios. In terms of configuration mode, the solution breaks through the limitations of traditional data processing that relies on coding, providing an intuitive visual operation interface. At the process operation level, the solution constructs a fully closed-loop automated system of "data diversion - preprocessing - batch extraction - insertion and deployment".

[0021] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A high-efficiency data processing method applicable to multiple tables based on batch processing, characterized in that, Includes the following steps: S1: Data acquisition and distribution; S2: Data preprocessing and insertion data construction, which mainly handles data cleaning, transformation and field binding, providing a standardized data structure for subsequent batch data processing and insertion; S3: Batch data extraction and processing, which is used to extract data in batches, standardize the data, and record abnormal data. S4: Batch data insertion and logging, which is used to efficiently write cleaned data into the target database in batches and to perform full-process log tracking.

2. The efficient data processing method for multiple tables based on batch processing according to claim 1, characterized in that, In step S1, data acquisition and distribution specifically include the following steps: S101: Configure the data source, including request method, database type, request parameters, and data source information, to obtain the data provided by the hospital. S102: Configure table mapping to support mapping the table structure in the data source to the data model of the target system. The purpose is to achieve structural alignment and semantic unity between the source data structure and the target system data model. Users can perform field-level mapping and renaming through the interface. S103: Implement a data splitting strategy, which can dynamically allocate data based on table name or custom rules and assign it to the corresponding target business table. The data separation strategy will also automatically match data from different data sources to the target business table, providing a structured foundation for subsequent data cleaning and warehousing processing.

3. The efficient data processing method for multiple tables based on batch processing according to claim 2, characterized in that, In step S101, the request method configuration enables unified access and management of multiple types of data sources through a visual system configuration page. The data source type in the database type configuration includes, but is not limited to, relational databases, distributed file systems, and non-relational databases; The data source information configuration allows users to complete the filling and verification of the above data source connection parameters in the system's backend visual configuration interface, including IP address, port number, username, password, file path, access protocol, and authentication method. The data source information configuration also supports configuring data update frequency parameters to adapt to the data update needs of different business scenarios.

4. The efficient data processing method for multiple tables based on batch processing according to claim 2, characterized in that, In step S2, data preprocessing and insertion data construction include the following steps: S201: Data Cleaning Rule Definition: The system uses a configurable cleaning rule definition module to achieve automated quality control and standardized processing of source data. It also supports multi-dimensional cleaning strategy configuration, including missing value handling, outlier identification and correction, and format standardization, to ensure the integrity and consistency of data before it enters subsequent processing stages. S202: Data Conversion Rule Definition; The system provides data conversion functions such as field concatenation, numerical calculation, encoding conversion, and multi-field mapping, and supports field-level calculation and mapping rule definition, thereby meeting the data structure integration and semantic fusion needs in complex business scenarios; S203: Data Object Construction: After data cleaning and transformation, the system enters the data object construction phase, which generates data carrier objects that can be directly used for batch insertion. Additionally, the system uses a Map-based approach.<Key, Value> The system uses a key-value mapping data structure to construct intermediate data objects. During the object construction process, the system fills the source data into the corresponding target fields according to the field mapping relationship. If the field defines a default value or calculation rule in the mapping relationship, the system will execute the calculation logic and fill the result at this stage to ensure that the generated data object is completely consistent with the target table structure.

5. The efficient data processing method for multiple tables based on batch processing according to claim 4, characterized in that, In S203, Key represents the field name of the target table, and Value represents the field value after being processed by the cleaning and transformation rules.

6. The efficient data processing method for multiple tables based on batch processing according to claim 4, characterized in that, In step S3, batch data extraction and processing includes the following steps: S301: Preliminary verification: During the data extraction stage, the system performs a dual verification mechanism of the structure layer and content layer on the source data to ensure the integrity and accuracy of the data before it enters the processing flow; S302: Deep cleaning and transformation: Determine whether the current data already exists in the database, and perform duplicate data detection and deduplication to ensure that the data that finally enters the batch storage stage has high consistency and high quality.

7. The efficient data processing method for multiple tables based on batch processing according to claim 6, characterized in that, In S301, the dual-layer verification mechanism includes structural verification and content verification, which realizes the pre-guarantee of data quality and ensures that the data has structural correctness and content legality before entering the cleaning and transformation stage, thereby improving the overall data processing stability and traceability of the system.

8. The efficient data processing method for multiple tables based on batch processing according to claim 6, characterized in that, In step S302, the deduplication strategy can be executed based on the primary key, unique index, or business logic field, retaining the latest or valid record, or an update field can be set.

9. A high-efficiency data processing method based on batch processing and applicable to multiple tables, as described in claim 6, is characterized in that... In step S4, batch data insertion and log recording include the following steps: S401: Batch processing strategy definition: Users can define a single batch processing threshold in the system. The system temporarily stores the cleaned and transformed data in the memory cache. When the cumulative data volume reaches the set threshold, the batch insertion operation is automatically triggered. At the same time, the system has a built-in transaction control mechanism in the batch processing process. When any batch of data encounters an exception during the insertion stage, the transaction rollback will be automatically triggered to ensure the consistency and atomicity of all data in the batch and prevent data incompleteness caused by the successful writing of some data. S402: Batch Insertion and Dynamic SQL Generation Mechanism: Before performing the insertion operation, the system will automatically generate a compatible batch insert statement template based on the target database type and intelligently adapt to the syntax differences of different databases. During the data insertion stage, the system dynamically concatenates the field names and field value lists by iterating through the Map collection using foreach, thereby automatically assembling the SQL statement. In addition, when performing batch insertion, the system will uniformly assemble multiple records according to the batch processing threshold configured above and write them using a batch pre-compilation method, thereby effectively reducing the number of database interactions. S403: Log Recording and Monitoring: Throughout the entire data processing and insertion process, the system has a built-in comprehensive log recording and operation monitoring mechanism to ensure the traceability of task execution and the visualized management of system operation and maintenance.

10. A high-efficiency data processing method based on batch processing and applicable to multiple tables, as described in claim 9, is characterized in that... In S401, the system can dynamically adjust the batch processing threshold according to the runtime load to achieve adaptive optimization for different data volumes and network environments.

Images