A component algorithm for fast synchronization of data warehouse files

Abstract

The present invention provides a component algorithm for fast synchronization of data warehouse files, aiming to improve the efficiency, reusability and scalability of DEL file synchronization, while enhancing the monitoring and exception handling capabilities of the synchronization process. The algorithm designs a data warehouse file model, records the results of each batch processing, and uses a template design pattern to decouple the general DEL file synchronization logic from specific business scenarios. The core method task is responsible for obtaining the DEL file path, parsing the file line by line, saving or updating business objects, and recording the synchronization status. Through Java's generics and reflection mechanisms, the data in the DEL file is quickly encapsulated as a business object, supporting multiple data type conversions. The abstract class WarehouseService of the component provides an extensible interface, and developers can implement specific business logic by inheriting this class. During the synchronization process, the system performs comprehensive monitoring and logging to ensure operability and maintainability. The algorithm is efficient, flexible, and reusable, and is suitable for large-scale data synchronization scenarios.

Description

Technical Field

[0001] The present invention relates to the field of data processing and information technology, and in particular to a component algorithm for fast synchronization of data warehouse files. Background Art

[0002] With the development of information technology and the growing demand for big data processing, data management and processing are facing complex challenges, especially in the field of financial institutions. At present, the system architecture of many financial institutions adopts microservice architecture to support their business processes. This architecture decomposes the system into multiple independent services, which can improve the scalability and flexibility of the system. However, the microservice architecture also introduces new problems in data processing, especially the business data is scattered in multiple independent services, forming a "data island" phenomenon. This phenomenon makes it difficult to directly share and integrate business data in cross-service scenarios, which brings great difficulties to the real-time synchronization and sharing of data.

[0003] Traditional solutions usually rely on API or RPC (remote procedure call) for cross-service data interaction. However, in some business scenarios, API or RPC calls cannot effectively meet data sharing needs due to performance bottlenecks, security considerations or system compatibility. To make up for this shortcoming, the data warehouse (DW) model is widely adopted. The data warehouse integrates and processes data from multiple microservices, converts it into a consistent format, and produces the processed data as a DEL file (DataExchangeLayer file) for the required services. This method effectively solves the problem of cross-service data sharing.

[0004] Although the data warehouse model solves the problem of cross-service data integration to a certain extent, how to quickly and efficiently synchronize the DEL files generated by these data warehouses in actual business is still a difficult problem that needs to be solved. Especially in large-scale business scenarios, with huge amounts of data and frequent file synchronization, how to ensure the efficiency, accuracy and operability of data synchronization becomes the key to system design.

[0005] The existing DEL file synchronization method usually has the following problems:

[0006] Low synchronization efficiency: When faced with large amounts of data, traditional synchronization methods take a lot of time to read, parse, process, and store files, affecting the overall performance of the system.

[0007] Poor reusability: In existing solutions, the synchronization logic of DEL files is usually closely integrated with business implementation, resulting in poor code reusability and the need for repeated development and maintenance in different business scenarios.

[0008] Weak monitorability: In the actual production environment, there is a lack of real-time monitoring and logging of the DEL file synchronization process, making it difficult to quickly locate and repair problems when anomalies occur.

[0009] Insufficient scalability: With the rapid development of financial services, new business needs continue to emerge. Existing solutions often require major modifications to synchronization processes when responding to business changes, and lack sufficient flexibility and scalability.

[0010] In order to solve these problems, the present invention proposes a component algorithm for fast synchronization of data warehouse files, which aims to improve the efficiency of DEL file synchronization, improve the reusability and scalability of the system, and provide comprehensive monitoring and operation and maintenance support for the synchronization process. By introducing template design patterns, abstract classes, and generic reflection mechanisms, the component algorithm can simplify the development and maintenance of the synchronization process, decouple synchronization logic from business rules, and reduce the complexity of the system. At the same time, with the help of monitoring points and exception handling mechanisms, the monitorability and operability of the file synchronization process are ensured, thereby greatly improving the overall efficiency and reliability of data warehouse file synchronization. Summary of the invention

[0011] The present invention relates to a component algorithm for fast synchronization of data warehouse files, aiming to solve the problems of low DEL file synchronization efficiency, poor reusability, weak monitorability and insufficient scalability in the prior art. The algorithm adopts template design mode, abstract class design, generics and reflection mechanism to achieve efficient synchronization of DEL files, decoupling of business logic and synchronization logic, and provides a comprehensive monitoring mechanism to ensure the efficiency, reliability and operability of the synchronization process.

[0012] Technical issues

[0013] The synchronization of traditional data warehouse files (DEL files) faces many challenges, especially in financial business scenarios, where business data often spans multiple microservices, which cannot directly share data through simple APIs or RPCs. The DEL files generated by the data warehouse are one of the main ways to share data. However, the existing DEL file synchronization methods have the following key problems:

[0014] Inefficient file synchronization: As the amount of data grows, the performance bottleneck of file parsing and storage becomes increasingly apparent, affecting the overall system performance and user experience.

[0015] Lack of versatility and reusability: The existing synchronization methods are tightly coupled with specific business logic. Different business scenarios require separate development of corresponding synchronization logic, which increases development and maintenance costs.

[0016] Insufficient monitoring and exception handling capabilities: In complex business scenarios, it is difficult to monitor the status in real time or record detailed logs during the file synchronization process. When an exception occurs, it takes a long time to troubleshoot and fix the problem, affecting business continuity.

[0017] Poor scalability: With the continuous emergence of new business scenarios, existing solutions are difficult to quickly adapt to changing needs and lack sufficient scalability and flexibility.

[0018] The purpose of the present invention is to provide a data warehouse file synchronization component algorithm that is efficient, highly reusable, scalable and has comprehensive monitoring capabilities to solve the above technical problems.

[0019] Technical Solution

[0020] In order to solve these problems, the present invention designs a component algorithm for fast synchronization of data warehouse files, and its overall technical solution includes the following steps:

[0021] DEL model design This invention provides a data structure foundation for the synchronization process by designing a data warehouse file model. The model is used to record the results of each batch processing, including key information such as batch number, file synchronization processing date, file path, business table name, and synchronization results. The main fields in the model include:

[0022] id: batch number, which uniquely identifies the batch of each file synchronization.

[0023] fileDt: The date when the file was synchronized.

[0024] filePath: storage path of DEL file.

[0025] tableName: the corresponding business table name.

[0026] flag: File synchronization result flag, including noFile (file not found), noValid (illegal file format), OK (successful parsing), Failed (failed parsing), etc.

[0027] createDt: record creation time.

[0028] updateDt: records the updated time.

[0029] In addition, the data warehouse file detailed object model is used to record the parsing and changes of each data in each DEL file. The fields of this model include:

[0030] batchId: the ID of the corresponding batch.

[0031] sqlData: specific business data.

[0032] key: primary key of business data.

[0033] operateType: business operation type, such as add or update.

[0034] flag: Parsing result, success or failure.

[0035] createDt and updateDt: record creation and update time.

[0036] In order to achieve universality and reusability, the present invention defines an abstract class WarehouseService, which contains the core program task and is responsible for executing the specific steps of DEL file synchronization. The main steps include:

[0037] Get the DEL file path: Use the getFile method to get the paths of the DEL file and Check file according to the specified synchronization date.

[0038] Update the data warehouse file model: Before synchronization starts, update the corresponding data warehouse file model record status to OK, indicating that synchronization has started.

[0039] Read the DEL file line by line: Read the contents of the DEL file line by line through the BufferedReader object.

[0040] Save or update business objects: For each row of records read, encapsulate it into a corresponding business object through the saveOrUpdate method, and save or update it according to the business logic.

[0041] Storage synchronization results: Store the parsed data in the data warehouse file model detail object.

[0042] Abnormal monitoring and processing: During the synchronization process, point-by-point monitoring is performed to record and process abnormal situations in a timely manner to ensure the stability and maintainability of the synchronization process.

[0043] Implementation of abstract methods Some methods in the WarehouseService class, such as getFile and saveOrUpdate, are abstract methods, and subclasses need to implement them according to specific business scenarios. By inheriting the WarehouseService class, developers can define independent subclasses for different business scenarios without rewriting the core synchronization logic, thus achieving decoupling of business logic and synchronization logic.

[0044] The getFile method is used to obtain the path of a specific DEL file. Usually, the path is specifically defined in conjunction with the configuration file and through the inherited class.

[0045] The saveOrUpdate method handles two key tasks: 1) saving or updating business objects; 2) feeding back detailed objects of the data warehouse file model for subsequent monitoring and storage.

[0046] Generic encapsulation and conversion of business objects In the saveOrUpdate method, the present invention introduces generics and reflection mechanisms, and converts the data read line by line from the DEL file into corresponding business objects through the FieldMatchUtil.getDaoClassIns method. This method maps and converts according to the field order in the DEL file and the field definition of the business object, thereby quickly completing the encapsulation of the business object. The specific steps include:

[0047] Split each line of data according to the data separator (such as #) in the DEL file to form a string array.

[0048] Combined with the field order definition of the business object, the segmented data is matched with the business object fields one by one.

[0049] Use Java's reflection mechanism to dynamically set the attribute values ​​of business objects.

[0050] According to the field type of the business object (such as byte, short integer, floating point, Boolean, etc.), the conversion from string to the corresponding data type is implemented through the Java wrapper class.

[0051] Template design pattern and business extension The present invention separates the general logic and business logic of DEL file synchronization through the template design pattern. When developers define new business scenarios, they only need to inherit the WarehouseService class and implement the necessary abstract methods to quickly create new DEL file synchronization logic without having to repeatedly write the core synchronization logic. This greatly improves the scalability and maintenance efficiency of the system.

[0052] Monitoring and operation and maintenance support In each link of file synchronization, the present invention records the loading status, parsing progress and abnormal situations of DEL files through point-of-use monitoring. The monitoring system can not only provide real-time status feedback during the file synchronization process, but also provide detailed logs and error information when an exception occurs, helping operation and maintenance personnel to locate problems and repair them in time. The key points of monitoring include file reading status, parsing errors, business data conversion errors, etc.

[0053] Beneficial effects of the invention:

[0054] The component algorithm of the present invention can achieve the following technical effects:

[0055] Efficient file synchronization: The present invention can significantly improve the synchronization efficiency of DEL files through optimized file parsing and business object encapsulation mechanism, and is suitable for business scenarios with large amounts of data.

[0056] Good reusability: By using template design patterns and abstract class design, business logic and core synchronization logic are completely decoupled. Developers only need to define a small amount of code to adapt to different business scenarios, reducing repeated development and maintenance costs.

[0057] Comprehensive monitoring and exception handling: Each key link in the synchronization process is monitored and logged to ensure rapid response and repair in abnormal situations, improving system reliability.

[0058] Strong scalability: Through abstract methods and generic encapsulation mechanisms, developers can easily expand and define new business scenarios. The components are highly flexible and scalable and can quickly adapt to changes in business needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only for the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0060] Figure 1 Schematic diagram of the DEL data warehouse file model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0061] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. At the same time, it is explained here that in order to make the embodiments more detailed, the following embodiments are the best and preferred embodiments, and those skilled in the art may also adopt other alternatives to implement some known technologies; and the accompanying drawings are only for more specific description of the embodiments, and are not intended to specifically limit the present invention.

[0062] It should be noted that the references to "one embodiment", "an embodiment", "an exemplary embodiment", "some embodiments" and the like in the specification indicate that the embodiments described may include specific features, structures or characteristics, but not every embodiment may include the specific features, structures or characteristics. In addition, when a specific feature, structure or characteristic is described in conjunction with an embodiment, it should be within the knowledge of a person skilled in the art to implement such feature, structure or characteristic in conjunction with other embodiments (whether or not explicitly described).

[0063] In general, a term can be understood, at least in part, from its use in context. For example, depending, at least in part, on the context, the term "one or more" as used herein can be used to describe any feature, structure, or characteristic in the singular sense, or can be used to describe a combination of features, structures, or characteristics in the plural sense. Additionally, the term "based on" can be understood as not necessarily intended to convey an exclusive set of factors, but can instead, depending, at least in part, on the context, allow for the presence of other factors that are not necessarily explicitly described.

[0064] Example 1

[0065] For a certain service, the data warehouse DEL successfully loads the public module within the service. The overall algorithm includes the following steps:

[0066] 1) DEL model design

[0067] First, the design of the DEL data warehouse file model provides a control basis for the component algorithm. Figure 1 As shown: The data warehouse file model is used to record each batch run and the conclusion of the batch run for storage. The specific field design is shown in the following table:

[0068]

[0069] The data warehouse file detailed object model is used to record every data change in each DEL file. The specific field design is shown in the following table:

[0070]

[0071]

[0072] Warehouse*** series represents the specific business record model table after the data warehouse file is synchronized to the disk. This table is a corresponding table for a type of data warehouse file DEL. It is defined according to the actual DEL file type. Example: WarehouseCredit, merchant credit data warehouse business data

[0073] WarehouseMcspSalesPerson, merchant sales representative loan amount business data

[0074] 1) DEL file synchronization algorithm

[0075] The main scenarios for synchronization of different DEL files are the same, including the data warehouse file model, data warehouse file detailed object model, and DEL file parsing process steps. However, the specific processing process of WareHouese** is different. Here, the DEL file component class is defined by combining JAVA's abstract class and abstract method with the template design pattern.

[0076] First, the DEL file component class is defined as the abstract class WarehouseService. The core program of this class is task, which provides the specific processing steps for data warehouse file synchronization:

[0077] 1. Get the DEL file path, getFile method, and get the DEL file and DEL's Check file path according to the specified synchronization date;

[0078] 2. First, update the corresponding data warehouse file model record to the OK status;

[0079] 3. Define the BufferedReader object to read the DEL file line by line;

[0080] 4. When the read row record is null, it means that the file parsing is complete; otherwise continue;

[0081] 5. Execute the saveOrUpdate method for each row record. This method is mainly encapsulated into the corresponding business object Warehouse*** for saving and updating, and returns the detailed object of the data warehouse file model.

[0082] 6. Store the data warehouse file model object;

[0083] 7. Repeat steps 4, 5, and 6;

[0084] Secondly, regarding the business characteristics in WarehousrService, the following methods are abstract and need to be implemented when inheriting the WarehousrService class.

[0085] The getFile method is abstracted, indicating that calling this method requires obtaining the specific path of the DEL file. Generally, in a specific business subclass, the definition of this method is usually defined in conjunction with the program's configuration file. The reference example is as follows:

[0086] @Override

[0087] protectedStringfindCheckFile(StringfileDt){

[0088] returnmerchSharRptFilePath+File.separator+fileDt+File.separator+merchSharRptCheckFile;

[0089] }

[0090] The saveOrUpdate method is abstracted to indicate that this method is called to handle two key points: 1. Complete the saving, storage and update of the Warehouse* object; 2. Encapsulate the detailed object according to the data warehouse file model and feedback the object entity reference.

[0091] Finally, in the saveOrUpdate step of WarehouseService, the encapsulation process of the business object is further rewritten. You only need to define the order of ORDER when defining the business object, and use the FieldMatchUtil.getDaoClassIns method to quickly complete the conversion of the business object from the data warehouse file detail object to the Warehouse**. The specific processing steps are as follows:

[0092] 1. Read row data from the data warehouse DEL file line by line, and split it into string array tests according to the separator "#";

[0093] 2. Combined with the generic characteristics of JAVA, define the getDaoClassIns method:

[0094] publicstatic <t>TgetDaoClassIns(String[]tests,String[]delOrder,intstart,Class <t>clazz); <t>It is the generic definition of the feedback business object; delOrder is the definition order of the business object fields, which needs to be consistent with the field order of the DEL file; start indicates the starting point of the deleOrder field; clazz is the class definition of the business object;

[0095] 3. Set the deleOrder group number, starting from the start element, according to

[0096] <key,value> -><delOrder[i+start],tests[i]> Format traversal generates Map object tempMap;

[0097] 4. Obtain the definition of all business attributes of the business object through the JAVA reflection method:

[0098] Field[]fields=clazz.getDeclaredFields();

[0099] 5. Perform traversal and value assignment processing on files; combine tempMap and wrapper class to complete the value assignment processing of business object entity object. Through the conversion of Java wrapper class, complete the conversion of DEL file character business value to wrapper class business value. The specific processing rules are as follows:

[0100] For byte type, use java.lang.Byte wrapper class for conversion, Byte.parseByte(value);

[0101] For short integer types, use the java.lang.Short wrapper class for conversion, Short.parseShort(value);

[0102] For integer types, use the java.lang.Integer wrapper class for conversion, Integer.valueOf(value);

[0103] For long integer types, use the java.lang.Long wrapper class for conversion, Long.parseLong(value);

[0104] For floating-point types, use the java.lang.Float wrapper class for conversion, Float.valueOf(value);

[0105] For double-precision floating-point types, use the "java.lang.Double wrapper class for conversion, Double.valueOf(value);

[0106] For Boolean type, use the java.lang.Boolean wrapper class for conversion, Boolean.parseBoolean(value);

[0107] For character types, use the java.lang.Character wrapper class for conversion, returnvalue.charAt(0);

[0108] For the specified precision type, use the BigDecimal wrapper class for conversion and return new BigDecimal(value);

[0109] For string types, no conversion is required;

[0110] For date types, use java.util.Date for conversion;

[0111] For timestamp types, use java.sql.Timestamp for conversion;

[0112] Comprehensively complete the core steps of the DEL data warehouse file synchronization component algorithm.

[0113] 2) The DEL file is loaded.

[0114] During the processing of DEL files, monitoring points are performed when the DEL files are waiting to be loaded, during loading, and after loading. Abnormal points in the processing process are monitored to ensure that the DEL file synchronization process can be monitored and maintained.

[0115] 3) Usage examples

[0116] Taking the synchronization of unified data warehouse files for merchant sales representatives as an example, the demo of using this component is described:

[0117] First, define the subclass WarehouseSalesService to inherit the WarehouseService class. Java syntax: publicclassWarehouseSalesServiceextendsWarehouseService;

[0118] Next, define the abstract method, getFile, and the core steps example:

[0119] returnsalesStatistcsFilePath+File.separator+fileDt+File.separat or+salesStatistcsFileName;

[0120] Third, define the abstract method, saveOrUpdate, and the core steps example:

[0121] WarehouseSaleswarehouseSales=FieldMatchUtil.getDaoClassIns(file ds,

[0122] WarehouseSales.ORDER,

[0123] WarehouseSales.START_NUM,

[0124] WarehouseSales.class);

[0125] Fourth, define the business object and define the field order;

[0126] / / File field order

[0127] publicstaticfinalString[]ORDER={

[0128] "statistcsDate","userId","stockNum","stockAmount","loanNum","loanAmount","overdueNum","overdueAmount"

[0129] };

[0130] Finally, in the batch running task, explicitly call the task method of the subclass object.

[0132] The present invention covers any substitution, modification, equivalent method and scheme made on the essence and scope of the present invention. In order to make the public have a thorough understanding of the present invention, specific details are described in detail in the following preferred embodiments of the present invention, but those skilled in the art can fully understand the present invention without the description of these details. In addition, in order to avoid unnecessary confusion about the essence of the present invention, well-known methods, processes, procedures, components and circuits are not described in detail.

[0133] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.< / t> < / t> < / t>

Claims

1. A component algorithm for fast synchronization of data warehouse files, characterized in that: The following steps are involved: Step 1: Design a DEL model to record the results of each batch process. The model includes fields such as batch number, file synchronization processing date, file path, business table name, and synchronization result. Step 2: DEL file synchronization algorithm. By defining an abstract class WarehouseService, whose core method is task, it is responsible for executing the specific steps of DEL file synchronization, including: a. obtaining the DEL file path; b. updating the data warehouse file model record; c. reading the DEL file line by line; d. saving or updating the business object; e. monitoring and storing the synchronization results; Step 3: DEL file parsing method: read the DEL file line by line through BufferedReader. When the file is finished reading, parse each record through the saveOrUpdate method and encapsulate it into a business object.

2. A component algorithm for fast synchronization of data warehouse files according to claim 1, characterized in that: The WarehouseService class defines a getFile method for obtaining a DEL file and a corresponding Check file path according to a specified synchronization date.

3. A component algorithm for fast synchronization of data warehouse files according to claim 1 or 2, characterized in that: The saveOrUpdate method encapsulates the steps of saving and updating business objects, and records each data change through the data warehouse file model detail object.

4. A component algorithm for fast synchronization of data warehouse files according to claim 3, characterized in that: The saveOrUpdate method converts the data parsed from the DEL file into a business object by calling the FieldMatchUtil.getDaoClassIns method using reflection and generic mechanisms.

5. A component algorithm for fast synchronization of data warehouse files according to claim 4, characterized in that: The FieldMatchUtil.getDaoClassIns method divides the data in the DEL file and converts it into corresponding business type objects according to the order of the business object fields.

6. A component algorithm for fast synchronization of data warehouse files according to any one of claims 1 to 5, characterized in that: When the DEL file is parsed, the Java packaging class mechanism is combined to convert the value according to the field type, including but not limited to byte type, short integer, long integer, floating point number, double precision floating point number, Boolean type, character type, specified precision type, etc.

7. A component algorithm for fast synchronization of data warehouse files according to claim 6, characterized in that: The type conversion step includes converting the string data into a Java wrapper class or object type according to the corresponding field type to ensure the integrity and consistency of the data.

8. A component algorithm for fast synchronization of data warehouse files according to any one of claims 1 to 7, characterized in that: The component algorithm supports the expansion of various business scenarios. By inheriting the WarehouseService class and implementing abstract methods, new business objects and synchronization logic can be quickly defined.

9. A component algorithm for fast synchronization of data warehouse files according to any one of claims 1 to 8, characterized in that: The components perform monitoring and tracking during the synchronization process to ensure exception handling and status monitoring during the synchronization process, thereby achieving the goals of monitorability and maintainability.

10. The component algorithm for fast synchronization of data warehouse files according to any one of claims 1 to 9, characterized in that: This component encapsulates the core synchronization logic through the template design pattern and concretizes the business logic through subclasses to improve the reusability and maintainability of the code.

Images