Enterprise-level unified templated data import and export method and system

Abstract

The application relates to the technical field of enterprise-level information system software development, and discloses an enterprise-level unified templating data import and export method and system, which comprises the following steps: receiving a request containing a template number sent by a front end through an access layer; searching a corresponding business processor instance from a global template register according to the template number; obtaining template configuration information and a business data object; performing file analysis or file generation operation of the business data object according to a file format type through a unified file reading and writing abstract layer, so that the business logic is decoupled from the file format; creating an asynchronous job through a unified asynchronous job engine to perform page batch processing; providing job whole life cycle management; and restoring the language environment and time zone context of the front end request when the job is executed; the application realizes full-system reuse of import and export capabilities, and solves the problems of repeated development, strong coupling between business logic and file format, lack of unified asynchronous control and weak cross-time zone adaptation capability.

Description

Technical Field

[0001] This invention relates to the field of enterprise-level information system software development technology, and in particular to an enterprise-level unified templated data import and export method and system. Background Technology

[0002] In enterprise-level business systems such as retail, supply chain, and ERP, almost all business modules have frequent data import and export requirements, covering application scenarios such as batch import of business documents, detailed data verification, export of business reports, and cross-system data integration. The import and export requirements of different business modules share common technical implementation aspects. However, current enterprise-level software generally adopts a "single-module independent implementation" development model for import and export functions. This means that each business module independently completes the entire development process from file parsing, data verification, business processing to file generation, resulting in code scattered across various modules. Maintenance costs increase linearly with the growth of business scale.

[0003] At the technical implementation level, existing solutions have several shortcomings. File parsing and generation logic is deeply tied to business data processing logic; adding or modifying file formats requires modification of core business code, resulting in extremely poor scalability. Furthermore, most implementations only support a single data format, failing to adapt to different business scenarios such as summary documents and detailed data. Most import / export operations use a synchronous execution mode, which easily leads to interface timeouts and memory overflows in large-scale data scenarios. Even if some modules implement asynchronous execution, they still require separate development of asynchronous tasks, progress queries, and interrupt control logic, lacking a unified system-wide job lifecycle management capability. Data validation and numerical precision management are also severely fragmented; validation logic cannot be uniformly reused, and decimal precision and rounding rules are inconsistent between different modules, making it highly susceptible to business errors due to inconsistencies in precision between detailed and summary data.

[0004] At the security and enterprise-level deployment level, permission verification is mostly implemented independently by each module, with some modules even lacking verification altogether, posing significant security risks. Internationalization configurations are scattered across various locations, resulting in extremely high costs for adapting to multi-language scenarios; simultaneously, the lack of a unified timezone resolution mechanism easily leads to time field offsets when deploying across time zones. Furthermore, the generation of business reports in formats such as PDF and HTML requires separate development, completely disconnected from regular data export logic, making it impossible to reuse existing data query logic, resulting in significant duplication of development work and difficulty in ensuring data consistency. Therefore, there is an urgent need for a technical solution that can uniformly handle the entire import and export process, enabling capability reuse and flexible expansion. Summary of the Invention

[0005] This invention addresses the problems of repetitive development, logical coupling, fragmented capabilities, and lack of unified management mechanisms in the import and export functions of existing enterprise-level business systems. It provides an enterprise-level unified templated data import and export method and system, aiming to achieve full system reuse and flexible expansion of import and export capabilities through standardized interfaces, file read and write abstraction layers, and asynchronous job engines, thereby improving development efficiency and system maintainability.

[0006] On the one hand, this invention provides an enterprise-level unified templated data import and export method, including the following steps: S1: Receive import or export requests sent by the front end through the access layer, wherein the requests contain a template number; S2: Based on the template number, search for the business processor instance corresponding to the template number in the global template registry; S3: Use the business processor instance to parse the request and obtain template configuration information and business data object, wherein the template configuration information includes at least the file format type; S4: Through a unified file read and write abstraction layer, the corresponding file parsing or file generation operation is performed on the business data object according to the file format type. The unified file read and write abstraction layer encapsulates the parsing logic and generation logic of multiple file formats, thereby decoupling the business logic from the specific file format. S5: Create asynchronous jobs for the operation through a unified asynchronous job engine, and perform paginated batch processing on the business data object based on the asynchronous job. The asynchronous job engine provides full lifecycle management of the job and restores the language environment and time zone context of the front-end request when executing the job.

[0007] Furthermore, in step S2, the global template registry is based on Spring Bean lifecycle management. When the system starts, it automatically scans all Spring Beans that implement the standardized processor interface or exporter interface and establishes a mapping relationship with the template number of each Bean as the key. The standardized processor interface includes at least MapDataProcessor for MAP format, PojoDataBatchProcessor for batch import of POJOs, and PojoDataDisposableProcessor for one-time processing of POJOs. The exporter interface includes at least MapDataExporter and PojoDataExporter.

[0008] Further, in step S4, through a unified file read / write abstraction layer, the corresponding file parsing or file generation operation is performed on the business data object according to the file format type, specifically including: When an import operation is performed, the unified file read and write abstraction layer calls the corresponding file parser according to the format type of the source file, and converts the content of the source file in xls, xlsx, csv or json format into a DataRow object stream or POJO object stream and passes it to the upper layer. When an export operation is performed, the unified file read and write abstraction layer calls the corresponding file generator according to the format type of the target file, and writes the DataRow object stream or POJO object stream output by the upper layer into the target file in xls, xlsx, csv, json, pdf or html format. When a business module modifies the imported or exported file format, it only needs to modify the file extension in the template configuration information without modifying the business data processing logic; when adding new file format support, it only needs to extend the corresponding parser or generator in the unified file read / write abstraction layer, and the already connected business modules will automatically be compatible with the new format.

[0009] Furthermore, in step S5, creating an asynchronous job for the operation using the unified asynchronous job engine specifically includes: Create job progress records and asynchronous processing task records. The job progress record shall at least include job name, job ID, template number, progress status, progress JSON, start time, end time, interruption information, progress result, and job execution result. The asynchronous processing task record shall at least include asynchronous executor identifier and asynchronous task parameter JSON. The asynchronous task parameter JSON shall include template number, job ID, file storage key, locale, and time zone. The asynchronous job is submitted to the built-in asynchronous thread pool. During execution, a paginated batch processing mode is used. After each batch of data is processed, the progress percentage and processing quantity status in the job progress record are updated. An interruptible checkpoint is set in the job execution process. Before each batch of data is processed, it is checked whether the job is marked as interrupted. If it is marked, the execution is terminated and the job status is updated to interrupted.

[0010] Furthermore, in step S5, restoring the locale and timezone context of the front-end request during job execution specifically includes: Upon receiving a request from the front end, the Locale language environment and TimeZone time zone information are parsed from the request, encapsulated into a job context, and persistently stored together with the asynchronous job record; When an asynchronous thread executes a job, it reads the Locale and TimeZone from the job context and restores them to the LocaleContextHolder and the current execution thread, respectively. During import, it converts the time values ​​in the file into time objects of the server time zone according to the front-end time zone and persists them to the database. During export, it converts the time values ​​in the database into the time format corresponding to the front-end time zone and writes them to the export file.

[0011] Preferably, during the import operation, the business data object is validated using a three-tier data validation system: The first level is basic format validation, which automatically checks the validity of number and date formats during the file parsing stage. Data with incorrect formats is marked as failed and format error feedback information is generated. The second level is field validity validation, which automatically parses and executes the validation rules defined by the BeanValidation annotation for POJO format data, including field length, value range and required field validation for @Size, @Max and @NotNull. Data that fails validation is marked as failed and filled with error messages. The third level is business logic verification. In the processData method, the business processor instance marks the data status and fills in the business verification feedback information by using the context.ignore or context.fail method.

[0012] Furthermore, the method also includes numerical precision management: Define a global numerical precision configuration interface, load the global default decimal places (scale) and rounding mode (roundingMode) as the default configuration for all templates when the system starts; provide a single-template personalized numerical precision configuration extension point, where business processor instances define the numerical precision configuration of the current template by overriding the getBizValueSettings method; During the import or export process, the single template configuration is read first, and if the single template is not configured, it will fall back to the global default configuration.

[0013] Preferably, the method further includes internationalization management: Construct a three-level multilingual text management system: global general configuration corresponds to the import-template-0 level, template-level configuration corresponds to the import-template-{template number} level, and enumeration value configuration is for job status and error code; When generating import templates or export files, the corresponding level of internationalized text is read from the three-level management system based on the Locale of the current thread, and the column names, file names, and sheet names are replaced.

[0014] More preferably, the method further includes an access control step and a report generation step: Before performing import, export, or interrupt job operations, the global permission validator is automatically invoked to perform unified permission verification; when the business processor instance overrides the authentication method, the personalized authentication logic of the business processor instance is executed, otherwise the logic of the global permission validator is executed, and the single-template authentication has higher priority than the global configuration. When the export business implementation class also implements the CsvToJasperReportTransformer interface, the data queried by the business is first standardized and exported as a CSV temporary file during the export process. Then, the JasperReport template engine reads the CSV temporary file and generates a PDF or HTML report file based on the report parameters. The interface includes the getReportTemplate method that returns a report template instance and the resolveReportArguments method that defines the report parameters and resource paths.

[0015] On the other hand, the present invention provides a method comprising: The access request module is used to receive import or export requests sent by the front end through the access layer, wherein the request contains a template number; The template routing module is used to search for the business processor instance corresponding to the template number from the global template registry. The configuration parsing and data adaptation module is used to parse the request using the business processor instance to obtain template configuration information and business data objects. The template configuration information includes at least the file format type. The file read / write abstraction module is used to perform corresponding file parsing or file generation operations on the business data object according to the file format type through a unified file read / write abstraction layer. The unified file read / write abstraction layer encapsulates the parsing logic and generation logic of various file formats, thereby decoupling the business logic from the specific file format. The asynchronous job scheduling and execution module is used to create asynchronous jobs for the operation through a unified asynchronous job engine, and perform paginated batch processing on the business data object based on the asynchronous job. The asynchronous job engine provides full lifecycle management of the job and restores the language environment and time zone context of the front-end request when executing the job.

[0016] Compared with the prior art, the beneficial effects of the present invention are: (1) This invention automatically scans and registers business processor instances through a global template registry, so that each business module only needs to implement a standardized interface and specify a template number to reuse all underlying capabilities without having to repeatedly develop common logic such as file parsing and pagination, which effectively shortens the development cycle and reduces maintenance costs. (2) This invention encapsulates the parsing and generation logic of multiple file formats through a unified file reading and writing abstraction layer, so that business code only programs towards standardized data object streams, and file format changes are only completed at the configuration level, which significantly reduces the risk of system changes and expansion costs; (3) This invention creates asynchronous jobs with full lifecycle management capabilities for each operation through a unified asynchronous job engine. Combined with a paginated batch processing mechanism, it effectively avoids interface timeout and memory overflow problems in large data scenarios under synchronous execution mode, thereby improving system stability and user experience. (4) This invention automatically restores the language environment and time zone context of the front-end request through the asynchronous job engine, so that the internationalized text reading and time format conversion in the entire process of import and export operation are consistent with the user request, eliminating the problem of data inconsistency such as time field offset when deploying across time zones; (5) This invention enables system maintenance personnel to query job progress and monitor execution status in real time through the job life cycle status model and interruption checkpoint mechanism built into the asynchronous job engine, and interrupt abnormal jobs when necessary, thereby improving the maintainability and resource utilization efficiency of the system. Attached Figure Description

[0017] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of an enterprise-level unified templated data import and export method according to the present invention; Figure 2 This is a timing diagram of an import operation according to the present invention. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] The specific embodiments of the present invention will be described below with reference to the accompanying drawings and examples.

[0020] Example 1

[0021] This embodiment is developed based on the Java Spring framework, and its overall architecture is divided into three layers: the access layer, the core engine layer, and the business extension layer. The access layer is responsible for receiving frontend requests and performing initial parsing and routing; the core engine layer includes core components such as a unified file read / write abstraction layer, an asynchronous job engine, and a data validation engine; the business extension layer provides standardized processor and exporter interfaces for each business module to implement specific import / export logic.

[0022] Please see Figure 1 The technical solution for an enterprise-level unified templated data import and export method provided in this embodiment includes the following steps: S1: Receive import or export requests sent by the front end through the access layer, wherein the requests contain a template number; S2: Based on the template number, search for the business processor instance corresponding to the template number in the global template registry; S3: Use the business processor instance to parse the request and obtain template configuration information and business data object, wherein the template configuration information includes at least the file format type; S4: Through a unified file read and write abstraction layer, the corresponding file parsing or file generation operation is performed on the business data object according to the file format type. The unified file read and write abstraction layer encapsulates the parsing logic and generation logic of multiple file formats, thereby decoupling the business logic from the specific file format. S5: Create asynchronous jobs for the operation through a unified asynchronous job engine, and perform paginated batch processing on the business data object based on the asynchronous job. The asynchronous job engine provides full lifecycle management of the job and restores the language environment and time zone context of the front-end request when executing the job.

[0023] The first step is the request access in step S1. The outermost entry point of this invention is responsible for receiving import or export requests sent by the frontend. The frontend request is sent to the system via the HTTP protocol, and the request must at least contain a template number (templateNo) parameter, and may optionally contain other business request parameters. The request access module parses the request, extracts the template number and request parameters, and passes them to the downstream modules for processing.

[0024] Based on the request access in step S1, template routing in step S2 is performed. The global template registry is based on Spring Bean lifecycle management, automatically scanning all Spring Beans that implement standardized processor or exporter interfaces at system startup, and establishing mapping relationships using the template number of each Bean as the key. The standardized processor interface includes at least MapDataProcessor for MAP format, PojoDataBatchProcessor for batch import of POJOs, and PojoDataDisposableProcessor for one-time processing of POJOs. The exporter interface includes at least MapDataExporter and PojoDataExporter.

[0025] Specifically, this invention is based on Spring Bean lifecycle management, automatically scanning all Spring Beans that implement standardized handler or exporter interfaces when a component starts. The system obtains all Bean instances implementing these interfaces through Spring's ApplicationContext, calls each Bean's getTemplateNo method to retrieve its unique template number, and stores this template number as the key and the Bean instance as the value in the global template registry, forming a one-to-one mapping between template numbers and business implementation classes. When the system receives an import or export request, the template routing module searches for and retrieves the corresponding business handler instance from the global template registry based on the template number in the request.

[0026] For import scenarios, this invention defines three types of standardized processor interfaces: The first type is MapDataProcessor, adapted for import scenarios with simple structures such as summary documents. This interface requires implementing classes to override the getTemplateNo method to return a unique template number, the getParamsType method to return the request parameter type, the resolveTemplateFile method to define the import template configuration, and the processData method to process MAP format data. During processing, business code reads data by field name using the get method of the DataRow object, converts it into a business entity, and then executes validation and saving logic. The second type is PojoDataBatchProcessor, adapted for batch import scenarios with complex structures such as detailed data. This interface adds a getPojoType method to MapDataProcessor to return the POJO object type, and its processData method receives a list of POJO objects as input, supporting batch processing. The business code performs validation on each POJO object in the list, obtains the context corresponding to the current record through the get method of BatchProcessContext, and calls the ignore or fail method to mark the data status. The third category is PojoDataDisposableProcessor, which is suitable for one-time full data processing scenarios such as data verification and full data processing.

[0027] Meanwhile, for export scenarios, this invention defines two types of standardized exporter interfaces: The first scenario, MapDataExporter adapted for report export with custom columns, requires the implementing class to override the getTemplateNo, getParamsType, and resolveExportFile methods to define the export template configuration, the queryExportedTotalCount method to query the total number of exported rows, and the queryExportedRows method to query exported data in paginated form. In the queryExportedRows method, DataRowJdbcMapper is used to map the database query results to DataRow objects named in camelCase. The second type of PojoDataExporter is adapted for exporting detailed data with a fixed structure.

[0028] Next, step S3, configuration parsing and data adaptation, is performed. After the business processor instance is obtained by the route, the configuration parsing and data adaptation module calls the `resolveTemplateFile` method (for importing scenarios) or the `resolveExportFile` method (for exporting scenarios) of the business processor instance to obtain the template configuration information. The template configuration information includes at least the file format type (such as xls, xlsx, csv, json, pdf, html, etc.) and field mapping rules.

[0029] For MAP format data, this embodiment encapsulates a DataRow object to uniformly carry row and column data, and provides standardized get methods (such as data.get("order number", String.class)) and set methods to implement field reading and writing. Business code can directly manipulate data through field names without having to worry about file parsing details.

[0030] For POJO format data, this embodiment implements an annotation-based object mapping mechanism. The `@XlColumn` annotation defines the mapping relationship, index, and required attributes between POJO fields and Excel columns. The component automatically converts file row data to POJO objects. It also natively supports field validation using Bean Validation annotations such as `@Size`, `@Max`, and `@NotNull`. Specifically, during import, the component reads the `@XlColumn` annotations on the POJO class fields, obtains the column index and column name for each field, reads the corresponding value from the data row by index, converts it to the POJO field type using a type converter, and then sets it into the POJO object.

[0031] Next, step S4, file read / write abstraction, is performed. This involves using a unified file read / write abstraction layer to perform corresponding file parsing or file generation operations on the business data object based on the file format type. Specifically, this includes: When an import operation is performed, the unified file read and write abstraction layer calls the corresponding file parser according to the format type of the source file, and converts the content of the source file in xls, xlsx, csv or json format into a DataRow object stream or POJO object stream and passes it to the upper layer. When an export operation is performed, the unified file read and write abstraction layer calls the corresponding file generator according to the format type of the target file, and writes the DataRow object stream or POJO object stream output by the upper layer into the target file in xls, xlsx, csv, json, pdf or html format. When a business module modifies the imported or exported file format, it only needs to modify the file extension in the template configuration information without modifying the business data processing logic; when adding new file format support, it only needs to extend the corresponding parser or generator in the unified file read / write abstraction layer, and the already connected business modules will automatically be compatible with the new format.

[0032] Specifically, this embodiment defines a unified file reading interface (FileReader) that encapsulates the parsing logic for xls, xlsx, csv, and json file formats. For Excel format, it is parsed using the Apache POI component; for CSV format, it is parsed by the delimiter using a CSV parser; and for JSON format, it is parsed using a JSON parser. Each format parser uniformly converts the content of source files of different formats into a DataRow object stream or a POJO object stream, providing standardized data input to the upper-layer business logic, without requiring the business code to be aware of the source file format.

[0033] Simultaneously, a unified file generation interface (FileWriter) is defined, encapsulating the file generation logic for xls, xlsx, csv, json, pdf, and html formats. Each format generator receives a DataRow or POJO object stream output from the upper-layer business logic and writes it uniformly to the target format file. For PDF and HTML formats, generation is performed using the JasperReport template engine, and the business code does not need to concern itself with the details of target file generation.

[0034] The core methods of the business implementation class, such as the processData method and queryExportedRows method, only handle standardized business data objects (DataRow or POJO) and do not involve any file format operations. That is, these methods do not include POI Workbook object operations, CSVWriter write operations, or JSON serialization operations.

[0035] When a business module needs to modify the import / export file format, it only needs to change the file extension (e.g., change ".xls" to ".xlsx" or ".csv") in the `resolveTemplateFile` or `resolveExportFile` method, without modifying the business data processing logic. When adding support for a new file format, it only needs to extend the corresponding format parser or generator in the file read / write abstraction layer, implementing the `FileReader` or `FileWriter` interface. All existing business modules will automatically be compatible with the new format without modifying their code.

[0036] Upon receiving a frontend request, step S5, asynchronous job scheduling and execution, is performed. The asynchronous job scheduling and execution module immediately creates an asynchronous job with a globally unique job ID for the import or export operation. Specifically, creating an asynchronous job for the operation using a unified asynchronous job engine includes: Create job progress records and asynchronous processing task records. The job progress record shall at least include job name, job ID, template number, progress status, progress JSON, start time, end time, interruption information, progress result, and job execution result. The asynchronous processing task record shall at least include asynchronous executor identifier and asynchronous task parameter JSON. The asynchronous task parameter JSON shall include template number, job ID, file storage key, locale, and time zone. The asynchronous job is submitted to the built-in asynchronous thread pool. During execution, a paginated batch processing mode is used. After each batch of data is processed, the progress percentage and processing quantity status in the job progress record are updated. An interruptible checkpoint is set in the job execution process. Before each batch of data is processed, it is checked whether the job is marked as interrupted. If it is marked, the execution is terminated and the job status is updated to interrupted.

[0037] The specific import operation sequence is as follows: Figure 2 As stated above.

[0038] First, the system creates two database records simultaneously. The first record is the job progress record (rb_progress_instance), and its fields are defined as follows:

[0039] The job status is derived from the combined calculation of three fields: state, result, and job_result. The mapping relationship is shown in the table below:

[0040] The progress of the import job is estimated in JSON format in three stages: 0%~10% is the stage of importing Excel data, which uses a time progress estimation mode to estimate the total time based on the file size; 10%~95% is the stage of processing database records, which uses an exact counting mode to calculate based on the number of processed records / total number of records; and 95%~100% is the stage of generating feedback files, which also uses an exact counting mode.

[0041] The second record is for asynchronous processing tasks (rbev_r), and its fields are defined as follows:

[0042] The contents of job progress records and asynchronous processing task records are immutable once created, ensuring the complete traceability of the job execution process.

[0043] Once the job is created, the asynchronous job scheduling and execution module submits the job to the built-in asynchronous thread pool for execution and immediately returns the job ID to the front end. The front end can then use this job ID to poll and query the job progress, avoiding interface timeouts caused by long-term blocking and waiting in synchronous execution mode.

[0044] The job execution adopts a paginated batch processing mode, and the amount of data processed per batch is configurable (1000 records per batch by default). After each batch of data is processed, the module automatically updates the progress percentage, number of records processed, and other statuses in the job progress record, and updates the successCount, failureCount, and ignoreCount fields in the job execution result JSON. When the frontend calls the progress query interface using the job ID, the module reads the latest job progress record from the database and returns it to the frontend.

[0045] In addition, interruptible checkpoints are set in the job execution process. Specifically, before processing each batch of data, the module checks whether the job progress record is marked as interrupted. If an interruption mark is detected, the execution of the current batch and subsequent batches of the job is immediately terminated, the occupied system resources (such as file handles, database connections, etc.) are released, and the job status is updated to INTERRUPTED.

[0046] The system provides a unified job interruption API, which receives interruption requests sent from the front end. After completing permission verification, it sets the interruption flag field in the progress record of the corresponding job to a valid state, thereby realizing externally controllable interruption of the job.

[0047] Secondly, restoring the locale and timezone context of the front-end request during job execution specifically includes: Upon receiving a request from the front end, the Locale language environment and TimeZone time zone information are parsed from the request, encapsulated into a job context, and persistently stored together with the asynchronous job record; When an asynchronous thread executes a job, it reads the Locale and TimeZone from the job context and restores them to the LocaleContextHolder and the current execution thread, respectively. During import, it converts the time values ​​in the file into time objects of the server time zone according to the front-end time zone and persists them to the database. During export, it converts the time values ​​in the database into the time format corresponding to the front-end time zone and writes them to the export file.

[0048] When a request is received from the front end, the module parses the Locale language environment (such as zh_CN, en_US) and TimeZone time zone information (such as Asia / Shanghai, UTC) from the Header or Query parameters of the request, encapsulates them into a job context object, and persists them along with the job record.

[0049] When an asynchronous thread executes a job, it first reads the Locale and TimeZone from the job context before executing the business logic, and sets them into the LocaleContextHolder and the ThreadLocal of the currently executing thread, respectively. Throughout the job execution process, Spring MessageSource automatically retrieves the Locale from the LocaleContextHolder when reading internationalized text, and automatically retrieves the TimeZone from the thread context when the time formatting tool performs time format conversion. This ensures that the internationalization configuration and time format conversion perfectly match the context of the frontend request, achieving automatic restoration of the language environment and time zone in asynchronous scenarios.

[0050] In the application.yml configuration file of this embodiment, the maximum size (unit: MB) and maximum number of lines to import for each imported template configuration file can be set. If not configured, the system default values ​​are used: maximum file size 10MB, maximum number of lines 1 million. When the imported file exceeds the configured threshold, the system directly reports an error and refuses to process it, avoiding the impact of excessively large files on system memory and database. Uploading and downloading OSS files uses the Rumba OSS component, and the failure retry strategy is determined by its internal implementation. This invention does not reimplement the retry mechanism for OSS operations. The import job processing method allows throwing Exception exceptions. When an exception occurs in the processing of the N+1th batch of data, the data processed in the previous N batches will not be rolled back, ensuring that the successfully processed data is persisted and avoiding the rollback of all data due to partial data exceptions. The execution framework code of the asynchronous job will catch Throwable exceptions, but only automatically triggers retries for database exceptions TransactionException and DataAccessException. The retry strategy uses the exponential backoff algorithm by default, and the retry interval is calculated according to the formula. Calculation, where For the first The interval between retry (in milliseconds). This is the initial retry interval (default 1000 milliseconds). This is the maximum retry interval (2 hours, or 7,200,000 milliseconds). Start counting from 0. When the calculated retry interval exceeds... When that happens, subsequent retries will all be based on The interval is specified. Other types of exceptions are not automatically retried; the job is directly marked as failed.

[0051] Furthermore, during the import operation, this invention verifies the business data object through a three-level data verification system, including: The first level is basic format validation, which automatically checks the validity of number and date formats during the file parsing stage. Data with incorrect formats is marked as failed and format error feedback information is generated. The second level is field validity validation, which automatically parses and executes the validation rules defined by the BeanValidation annotation for POJO format data, including field length, value range and required field validation for @Size, @Max and @NotNull. Data that fails validation is marked as failed and filled with error messages. The third level is business logic verification. In the processData method, the business processor instance marks the data status and fills in the business verification feedback information by using the context.ignore or context.fail method.

[0052] Specifically, the first level is basic format validation: During the file parsing phase, the component automatically performs validity checks on numeric and date formats based on the field's configuration type. Specifically, for fields annotated with `@XlDateTimeFormat`, the component attempts to parse the string value according to the specified date format; if parsing fails, the data is marked as failed. For fields annotated with `@XlDecimalFormat`, the component attempts to parse the string value as a BigDecimal; if parsing fails, it is marked as failed. Data with incorrect formats is directly marked as failed and does not proceed to subsequent business processing, while automatically generating format error feedback information (e.g., "Data in row X, column Y has an incorrect format; expected date format yyyy-MM-dd").

[0053] The second level is field validity validation: For POJO format data, after completing object mapping, the component uses a Java Bean Validation framework (such as Hibernate Validator) to automatically parse and execute the validation rules defined by annotations such as @Size, @Max, and @NotNull on the POJO class fields. Specifically, @Size checks if the string length is within the specified range, @Max checks if the value does not exceed the specified maximum value, and @NotNull checks if the field is not empty. Data that fails validation is automatically marked as invalid and populated with the corresponding error message (e.g., "Product code length cannot exceed 13 characters").

[0054] The third layer is business logic validation: The business processor instance implements custom business validation logic in the `processData` method. It obtains the context object corresponding to the current record through the `BatchProcessContext.get` method, calls the `context.ignore` method to mark ignored data (e.g., "Retail order already exists") or calls the `context.fail` method to mark failed data (e.g., "Product does not exist"), and populates the business validation feedback information. The three layers of validation form a progressive filtering to ensure that the data that finally enters the business saving stage is complete and valid.

[0055] The above-mentioned invention method also includes numerical precision management: Define a global numerical precision configuration interface, load the global default decimal places (scale) and rounding mode (roundingMode) as the default configuration for all templates when the system starts; provide a single-template personalized numerical precision configuration extension point, where business processor instances define the numerical precision configuration of the current template by overriding the getBizValueSettings method; During the import or export process, the single template configuration is read first, and if the single template is not configured, it will fall back to the global default configuration.

[0056] The method also includes internationalization management: Construct a three-level multilingual text management system: global general configuration corresponds to the import-template-0 level, template-level configuration corresponds to the import-template-{template number} level, and enumeration value configuration is for job status and error code; When generating import templates or export files, the corresponding level of internationalized text is read from the three-level management system based on the Locale of the current thread, and the column names, file names, and sheet names are replaced.

[0057] Specifically, this invention defines the GlobalBizValueSettingsSupplier global configuration interface to implement the system-wide default numerical precision and rounding rule configuration. The global default configuration is loaded when the component starts, serving as the default rule for all templates. In the specific implementation of this embodiment, the BizValueSettingsSupplierByProperties class inherits from GlobalBizValueSettingsSupplier and maintains a ConcurrentHashMap as a template cache (mapping precision configurations through template numbers). The processing logic of its getBizValueSettings method is as follows: First, it checks whether the passed-in Bean implements the IsTemplateTool interface; if not, it returns an empty Optional. If the interface is implemented, it calls the getTemplateNo method to obtain the template number, searches for the precision configuration corresponding to that template in the ConcurrentHashMap cache, and if it does not exist in the cache, it uses SpringPropertiesResolver to parse the corresponding precision configuration from application.yml and stores it in the cache, finally returning the configuration result. BizValueSettings is an interface that defines methods to obtain the number of decimal places (scale) and the rounding mode (roundingMode). BizValueProperties implements this interface and sets default values ​​(scale defaults to 2, roundingMode defaults to HALF_UP). In a SaaS multi-tenant scenario, it can be replaced with an implementation class for tenant configuration isolation.

[0058] It provides a single-template personalized configuration extension point, where business processor instances can override the getBizValueSettings method to define the numerical precision and rounding rules of the current template. When the component performs numerical processing, it first reads the single-template configuration. If the single template is not configured, it falls back to the global default configuration, realizing a two-layer priority control of "global default configuration + single-template personalized configuration".

[0059] The method also includes internationalization management: Construct a three-level multilingual text management system: global general configuration corresponds to the import-template-0 level, template-level configuration corresponds to the import-template-{template number} level, and enumeration value configuration is for job status and error code; When generating import templates or export files, the corresponding level of internationalized text is read from the three-level management system based on the Locale of the current thread, and the column names, file names, and sheet names are replaced.

[0060] Specifically, this invention constructs a three-level multilingual text management system based on the Spring MessageSource mechanism.

[0061] The first level is a global general configuration: Configure the messages.properties file with "import-template-0" as the prefix to achieve global internationalization for common fields across the entire system, such as import results and import feedback. For example, "import-template-0.column-order number" uniformly defines the internationalized text of the "order number" column in all templates.

[0062] The second level is template-level configuration: This is configured in the messages.properties file with the prefix "import-template-{template number}". It implements template-specific internationalization configurations for column names and filenames specific to a particular template number. For example, "import-template-111.column-filling time" defines the internationalized text for the "filling time" column in template 111. Template-level configurations have higher priority than global general configurations.

[0063] The third level is the enumeration value configuration: internationalization configuration at the enumeration level is implemented for enumeration values ​​such as job status (PENDING, SUCCESSFUL, FAILED, etc.) and error codes.

[0064] When generating import templates and exporting files, the component automatically reads the corresponding internationalized text from the three-level management system mentioned above according to priority (template level > global general > enumeration level) based on the Locale stored in the LocaleContextHolder of the current thread, and replaces the column names, file names and sheet names, realizing automatic internationalization adaptation across the entire chain. Business code does not need to handle any internationalization logic.

[0065] In addition, the method of the present invention also includes an access control step and a report generation step: Before performing import, export, or interrupt job operations, the global permission validator is automatically invoked to perform unified permission verification; when the business processor instance overrides the authentication method, the personalized authentication logic of the business processor instance is executed, otherwise the logic of the global permission validator is executed, and the single-template authentication has higher priority than the global configuration. When the export business implementation class also implements the CsvToJasperReportTransformer interface, the data queried by the business is first standardized and exported as a CSV temporary file during the export process. Then, the JasperReport template engine reads the CSV temporary file and generates a PDF or HTML report file based on the report parameters. The interface includes the getReportTemplate method that returns a report template instance and the resolveReportArguments method that defines the report parameters and resource paths.

[0066] Specifically, the access control process is automatically triggered before importing, exporting, or interrupting a job.

[0067] First, the system defines the GlobalPermissionValidator global permission validation interface, whose core method is assertOperationPermitted, which receives parameters such as the current user information, operation type (import / export / interrupted), and template number. The implementation class of the global permission validator can implement system-wide unified permission validation logic in this method, such as checking whether the current user has "import / export" permission or data operation permission for the corresponding module. If the validation fails, a permission exception is thrown, and the operation is terminated.

[0068] Building upon this, business processor instances can override the `assertOperationPermitted` method to implement personalized authentication logic for the current template. When performing permission verification, the system first checks whether the current business processor instance has overridden the `assertOperationPermitted` method (i.e., whether this method is overridden in a subclass). If it has, the system executes the business processor instance's authentication logic; otherwise, it executes the logic of the global permission validator. Single-template authentication has higher priority than global configuration, ensuring basic security control across the entire system while meeting flexible authentication needs across different business scenarios.

[0069] When the exported business implementation class also implements the CsvToJasperReportTransformer interface, the system automatically enables the report generation process.

[0070] The core processing flow of the report conversion interface is as follows: The business implementation class returns a JasperReportTemplate instance by overriding the getReportTemplate method (this instance encapsulates the JasperReport compilation template, font configuration, etc., to solve the problem of Chinese display in PDF), and overriding the resolveReportArguments method to define the name of the report template (such as "deliveryReport.jrxml"), resource path and report parameters (such as report title).

[0071] When the component performs an export, it first standardizes and exports the data retrieved from the business query into a temporary CSV file according to the usual export logic. Then, it loads the .jrxml template file specified in the report parameters and compiles it into .jasper format using JasperCompileManager. Next, it uses JasperFillManager to populate the report based on the CSV temporary file data source (read via JRCsvDataSource) and report parameters, generating a JasperPrint object. Finally, according to the target format (PDF or HTML), it exports the JasperPrint object into the final format report file using JasperExportManager.exportReportToPdfStream or JasperExportManager.exportReportToHtmlStream. The entire process is completely transparent to the business code; the business implementation class only needs to configure the report template and parameters, without needing to write any report generation logic.

[0072] Based on the above methods, a complete implementation case is presented: The first step is to import retail order details. Adding "-Duser.timezone=UTC" to the JVM startup parameters makes the application server and database operate in the UTC time zone.

[0073] Retail order details POJOs are defined as follows, where the product code has a maximum length of 13 characters, the quantity has a maximum value of 10000 with precision of Quantity, and the amount has precision of Amount: Java public class RetailOrderLine { @XlColumn(value = "order number", index = 1, required = true) private String billNumber; @XlColumn(value = "line number", index = 2, required = true) private int lineNo; @XlColumn(value = "product code", index = 3, required = true) @Size(max = 13) private String productCode; @XlColumn(value = "quantity", index = 4, required = true) @XlDecimalFormat(Quantity.class) @Max(10000) private BigDecimal qty; @XlColumn(value = "amount", index = 5, required = true) @XlDecimalFormat(Amount.class) private BigDecimal amount; } The retail order details import program uses the PojoDataBatchProcessor interface, template number 222. In the `processData` method, all retail order numbers and product codes are first retrieved from `objectList`, and the existing order number and product code lists are queried from the database. Then, each `RetailOrderLine` object in `objectList` is iterated. If the retail order number already exists, it is marked as ignored using `context.get(line).ignore`; if the product code does not exist, it is marked as failed using `context.get(line).fail`. Other data is saved (existing details are deleted before inserting new details). According to the configuration in `template-222` in `application.yml`, quantities are rounded to three decimal places.

[0074] Add the file "Retail Order Details Import Template.xlsx" to the classpath. Use this file when downloading template files in a Chinese environment. After importing the Excel file, the system performs three-level validation and generates a feedback file. Each line in the feedback file corresponds to the status (success / failure / ignore) of an imported record and the corresponding feedback information.

[0075] The second step is importing retail orders. In the retail order POJO definition, fillTime is of type Date and the @XlDateTimeFormat is specified as "yyyy-MM-dd HH:mm:ss", and accountingDate is of type LocalDate and is annotated with @XlDateTimeFormat (no parameters, using the ISO default format).

[0076] The retail order import program switched from MapDataProcessor to PojoDataProcessor, using the @XlColumn annotation to automatically map Excel columns to POJO fields. In the processData method, it first checks if the order document already exists using orderDao; if so, it calls context.ignore to mark it as ignored. Then, it checks if the document details have been imported; if the details are empty, it calls context.fail to mark it as failed. Next, it verifies the consistency between the summary data and the detail data (number of details, total quantity, total amount); finally, it saves the document.

[0077] Add a `messages_en.properties` configuration file, where `"import-template-0.column-xxx"` is set as the default value for all templates. When neither the `"RetailOrder Import Template.xls"` nor the `"RetailOrder Import Template.xls"` file exists in the classpath, the tool automatically generates the column names for the template file based on the `value` attribute of the `@XlColumn` annotation on the POJO. A template file with Chinese column names is generated when the browser's preferred language is Chinese (Simplified), and a template file with English column names is generated when the preferred language is English.

[0078] After importing the Excel file, the quantity precision of template 111 was set to 4 by default (the global default value is 4). The total quantity of the first document, 4.1234, was rounded down to 4.123 (3 decimal places), causing a discrepancy with the 4.1234 in the detailed summary and resulting in import failure. Modifying application.yml to change the quantity precision of template 111 to 3 and then importing again, the first document imported successfully. Finally, in the retail order table, fill_time is converted to UTC timezone according to the front-end timezone (automatically subtracting 8 hours if the front-end timezone is UTC+8), while accounting_date, defined as LocalDate, is independent of timezone.

[0079] Finally, there's report generation. When a business module needs to generate delivery reports in PDF or HTML format, the implementation team first converts the exported standard report into a CSV file. They then develop the report using Jaspersoft Studio based on the CSV data source and submit the request along with the .jrxml report template file. Developers add PDF and HTML export formats as parameters in the request and use the .jrxml file from the request as the default export template. If another project needs to modify the report display, the implementation team can develop their own .jrxml file and override the default template through configuration.

[0080] The DeliveryReportGenerator implements the MapDataExporter interface and the CsvToJasperReportTransformer interface, injects the JasperReportTemplate instance via @Autowired, overrides the getReportTemplate method to return this instance, and overrides the resolveReportArguments method to generate a title in the format of "xx~xx Delivery Report" according to the date range, and specify the "deliveryReport.jrxml" template file and resource path. The resolveExportFile method returns "Delivery Report.pdf" and adds support for .pdf and .html formats. When performing export, the queryExportedRows method executes the SQL query "select m.route_name, m.store_name,d.product_name, sum(d.qty) as qty from 送货单 m, 送货单明细 d where m.bill_number = d.bill_number and m.delivery_date between ? and ? group by m.route_name, m.store_name, d.product_name", automatically maps database fields such as route_name to column names in lower camel case such as routeName via DataRowJdbcMapper, and returns a list of DataRow objects. The component exports the data as a temporary CSV file, and then generates the final PDF or HTML report based on the .jrxml template through the JasperReport template engine, realizing the complete integration of data export and report generation.

[0081] Based on the above method, this embodiment provides an enterprise-level unified templated data import and export system, including: A request access module, configured to receive an import or export request sent by a front end through an access layer, where the request includes a template number; A template routing module, configured to search a global template registry for a business processor instance corresponding to the template number according to the template number; A configuration parsing and data adaptation module, configured to parse the request using the business processor instance to obtain template configuration information and a business data object, where the template configuration information at least includes a file format type; The file read / write abstraction module is used to perform corresponding file parsing or file generation operations on the business data object according to the file format type through a unified file read / write abstraction layer. The unified file read / write abstraction layer encapsulates the parsing logic and generation logic of various file formats, thereby decoupling the business logic from the specific file format. The asynchronous job scheduling and execution module is used to create asynchronous jobs for the operation through a unified asynchronous job engine, and perform paginated batch processing on the business data object based on the asynchronous job. The asynchronous job engine provides full lifecycle management of the job and restores the language environment and time zone context of the front-end request when executing the job.

[0082] It should be noted that the steps in the enterprise-level unified templated data import and export method provided in this embodiment can be implemented based on the corresponding modules in the enterprise-level unified templated data import and export system. Those skilled in the art can refer to the technical solution of the system to implement the steps of the method. That is, the embodiments in the system can be understood as preferred examples of implementing the method, and will not be elaborated here.

[0083] Besides implementing the system and its various devices provided by this invention in purely computer-readable program code, the same functions can be achieved by logically programming the method steps, making the system and its various devices of this invention appear as logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices provided by this invention can be considered as a hardware component, and the devices included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0084] Finally, it should be noted that the above description is only a preferred embodiment of the present invention, and the scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be pointed out that for those skilled in the art, any improvements and modifications made without departing from the principle of the present invention should also be considered within the scope of protection of the present invention.

[0085] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. An enterprise-level unified templated data import and export method, characterized in that, Includes the following steps: S1: Receive import or export requests sent by the front end through the access layer, wherein the requests contain a template number; S2: Based on the template number, search for the business processor instance corresponding to the template number in the global template registry; S3: Use the business processor instance to parse the request and obtain template configuration information and business data object, wherein the template configuration information includes at least the file format type; S4: Through a unified file read and write abstraction layer, the corresponding file parsing or file generation operation is performed on the business data object according to the file format type. The unified file read and write abstraction layer encapsulates the parsing logic and generation logic of multiple file formats, thereby decoupling the business logic from the specific file format. S5: Create asynchronous jobs for the operation through a unified asynchronous job engine, and perform paginated batch processing on the business data object based on the asynchronous job. The asynchronous job engine provides full lifecycle management of the job and restores the language environment and time zone context of the front-end request when executing the job.

2. The enterprise-level unified templated data import and export method according to claim 1, characterized in that, In step S2, the global template registry is based on Spring Bean lifecycle management. When the system starts, it automatically scans all Spring Beans that implement the standardized processor interface or exporter interface and establishes a mapping relationship with the template number of each Bean as the key. The standardized processor interface includes at least MapDataProcessor for MAP format, PojoDataBatchProcessor for batch import of POJOs, and PojoDataDisposableProcessor for one-time processing of POJOs. The exporter interface includes at least MapDataExporter and PojoDataExporter.

3. The enterprise-level unified templated data import and export method according to claim 1, characterized in that, In step S4, through the unified file read / write abstraction layer, the corresponding file parsing or file generation operation is performed on the business data object according to the file format type, specifically including: When an import operation is performed, the unified file read and write abstraction layer calls the corresponding file parser according to the format type of the source file, and converts the content of the source file in xls, xlsx, csv or json format into a DataRow object stream or POJO object stream and passes it to the upper layer. When an export operation is performed, the unified file read and write abstraction layer calls the corresponding file generator according to the format type of the target file, and writes the DataRow object stream or POJO object stream output by the upper layer into the target file in xls, xlsx, csv, json, pdf or html format. When a business module modifies the imported or exported file format, it only needs to modify the file extension in the template configuration information without modifying the business data processing logic; when adding new file format support, it only needs to extend the corresponding parser or generator in the unified file read / write abstraction layer, and the already connected business modules will automatically be compatible with the new format.

4. The enterprise-level unified templated data import and export method according to claim 1, characterized in that, In step S5, creating an asynchronous job for the operation using the unified asynchronous job engine specifically includes: Create job progress records and asynchronous processing task records. The job progress record shall at least include job name, job ID, template number, progress status, progress JSON, start time, end time, interruption information, progress result, and job execution result. The asynchronous processing task record shall at least include asynchronous executor identifier and asynchronous task parameter JSON. The asynchronous task parameter JSON shall include template number, job ID, file storage key, locale, and time zone. The asynchronous job is submitted to the built-in asynchronous thread pool. During execution, a paginated batch processing mode is used. After each batch of data is processed, the progress percentage and processing quantity status in the job progress record are updated. An interruptible checkpoint is set in the job execution process. Before each batch of data is processed, it is checked whether the job is marked as interrupted. If it is marked, the execution is terminated and the job status is updated to interrupted.

5. The enterprise-level unified templated data import and export method according to claim 1, characterized in that, In step S5, restoring the locale and timezone context of the front-end request during job execution specifically includes: Upon receiving a request from the front end, the Locale language environment and TimeZone time zone information are parsed from the request, encapsulated into a job context, and persistently stored together with the asynchronous job record; When an asynchronous thread executes a job, it reads the Locale and TimeZone from the job context and restores them to the LocaleContextHolder and the current execution thread, respectively. During import, it converts the time values ​​in the file into time objects of the server time zone according to the front-end time zone and persists them to the database. During export, it converts the time values ​​in the database into the time format corresponding to the front-end time zone and writes them to the export file.

6. The enterprise-level unified templated data import and export method according to claim 3, characterized in that, During the import operation, the business data object is validated using a three-tiered data validation system: The first level is basic format validation, which automatically checks the validity of number and date formats during the file parsing stage. Data with incorrect formats is marked as failed and format error feedback information is generated. The second level is field validity validation, which automatically parses and executes the validation rules defined by Bean Validation annotations for POJO format data, including field length, value range and required fields corresponding to @Size, @Max and @NotNull. Data that fails validation is marked as failed and filled with error messages. The third level is business logic verification. In the processData method, the business processor instance marks the data status and fills in the business verification feedback information by using the context.ignore or context.fail method.

7. The enterprise-level unified templated data import and export method according to claim 1, characterized in that, The method also includes numerical precision management: Define a global numerical precision configuration interface, load the global default decimal places (scale) and rounding mode (roundingMode) as the default configuration for all templates when the system starts; provide a single-template personalized numerical precision configuration extension point, where business processor instances define the numerical precision configuration of the current template by overriding the getBizValueSettings method; During the import or export process, the single template configuration is read first, and if the single template is not configured, it will fall back to the global default configuration.

8. The enterprise-level unified templated data import and export method according to claim 1, characterized in that, The method also includes internationalization management: Construct a three-level multilingual text management system: global general configuration corresponds to the import-template-0 level, template-level configuration corresponds to the import-template-{template number} level, and enumeration value configuration is for job status and error code; When generating import templates or export files, the corresponding level of internationalized text is read from the three-level management system based on the Locale of the current thread, and the column names, file names, and sheet names are replaced.

9. The enterprise-level unified templated data import and export method according to claim 1, characterized in that, The method also includes access control steps and report generation steps: Before performing import, export, or interrupt job operations, the global permission validator is automatically invoked to perform unified permission verification; when the business processor instance overrides the authentication method, the personalized authentication logic of the business processor instance is executed, otherwise the logic of the global permission validator is executed, and the single-template authentication has higher priority than the global configuration. When the export business implementation class also implements the CsvToJasperReportTransformer interface, the data queried by the business is first standardized and exported as a CSV temporary file during the export process. Then, the JasperReport template engine reads the CSV temporary file and generates a PDF or HTML report file based on the report parameters. The interface includes the getReportTemplate method that returns a report template instance and the resolveReportArguments method that defines the report parameters and resource paths.

10. An enterprise-level unified templated data import and export system, characterized in that, include: The access request module is used to receive import or export requests sent by the front end through the access layer, wherein the request contains a template number; The template routing module is used to search for the business processor instance corresponding to the template number from the global template registry. The configuration parsing and data adaptation module is used to parse the request using the business processor instance to obtain template configuration information and business data objects. The template configuration information includes at least the file format type. The file read / write abstraction module is used to perform corresponding file parsing or file generation operations on the business data object according to the file format type through a unified file read / write abstraction layer. The unified file read / write abstraction layer encapsulates the parsing logic and generation logic of various file formats, thereby decoupling the business logic from the specific file format. The asynchronous job scheduling and execution module is used to create asynchronous jobs for the operation through a unified asynchronous job engine, and perform paginated batch processing on the business data object based on the asynchronous job. The asynchronous job engine provides full lifecycle management of the job and restores the language environment and time zone context of the front-end request when executing the job.

Images