Data summarization integration optimization method and system

Abstract

The invention discloses a data summarization integration optimization method and system, and the method comprises the steps: setting server member configuration information, and storing the server member configuration information in a first format; constructing a database file at the server, traversing all meta configuration information, and establishing a corresponding data structure to obtain a first database; the client reads local table data, packages the local table data in a first format and packages the local table data into a message; the server receives the message, and operates the monitoring framework based on the first format to analyze the message to obtain field information; changing the first database based on different field information to obtain a second database; in response to the push command, the server pushes the data of the second database to the remote database, and constructs or updates the remote database according to the meta configuration information; codes do not need to be adjusted through meta-configuration information, and high expandability is achieved; cross-platform and cross-system integration of data can be realized; the method can be widely applied to various data management scenes, and occupied performance resources are smaller.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data aggregation, integration and optimization method and system. Background Art

[0002] With the implementation of the construction of the data analysis basic support platform, and the trend of continuously promoting the deep integration of business and technology, a large number of demand scenarios for data analysis and visual dashboards have emerged within power grid enterprises, striving to achieve scientific analysis and decision-making of production and operation activities by mining the distributed data of application systems in various business domains, which also puts higher requirements on the data supply link. In most cases, business personnel can directly obtain data and execute data analysis operation logic through the data middle platform, but when encountering some analysis scenarios of local personalized data, the existing data supply platform cannot fully meet business needs for the time being. The specific reasons are as follows: First, the data required for such personalized scenarios comes from the self-built systems of various units, which have not yet been connected to the data middle platform, and cannot be directly obtained and analyzed; second, although some data comes from the connected system, it is usually processed by multiple original tables and output to the front-end interface, and the data query conditions and calculation logic are complex; third, in addition to directly connecting to the business system to obtain online data, some personalized dashboard scenarios also require offline data, specifically to obtain reports manually filled out in computer office terminals and output them to the visual dashboard. For such personalized data analysis scenarios, business personnel usually use automation technologies such as RPA to automatically obtain the required data from the front-end page of the business system, including indicators, detailed data, etc., and store them in multiple computer terminal file systems in common office file formats. Due to collaboration needs, various data files need to be viewed and managed in a unified manner.

[0003] However, there is currently no method for reading, aggregating and centrally managing offline data that supports later expansion and is applicable to more scenarios, and is often limited to a specific file storage format. In addition, due to the variety of terminal data storage forms, directly using raw data will be detrimental to interface design and data exchange, and data updates cannot be refined to the field level. Currently, most application systems are based on the HTTP protocol, and there are a large number of open source HTTP development frameworks, but they are all too large for this application and are cumbersome to use.

[0004] Therefore, a data aggregation method is needed that can realize universal data reading, packaging, transmission, and updating in different scenarios. Summary of the invention

[0005] In view of the above existing problems, the present invention is proposed.

[0006] Therefore, the present invention provides a data aggregation, integration and optimization method and system to solve the problems that the current reading aggregation and centralized management methods are difficult to support later expansion and be applicable to more scenarios, data updates cannot be refined to the field level, and the data transmission process is relatively cumbersome.

[0007] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0008] In a first aspect, the present invention provides a data aggregation and integration optimization method, comprising:

[0009] Setting the server meta-configuration information to be stored in a first format;

[0010] Build a database file on the server side, traverse and read all meta-configuration information, establish a corresponding data structure in the database file, and obtain a first database;

[0011] The client reads the local table data, encapsulates it in the first format, and packages it into a message;

[0012] The server receives the message and runs the monitoring framework based on the first format to parse the message to obtain field information;

[0013] Modifying the first database based on different field information to obtain an updated second database;

[0014] In response to the push command, the server pushes the data of the second database to the remote database, and constructs a database file in the remote database according to the meta-configuration information to construct or update the remote database;

[0015] Data is exported and downloaded based on the second database.

[0016] As a preferred solution of the data aggregation and integration optimization method described in the present invention, wherein: the meta-configuration information is composed of table configurations, each table includes multiple configurations, and each configuration information can be dynamically adjusted according to actual business needs.

[0017] As a preferred solution of the data aggregation and integration optimization method described in the present invention, the encapsulation includes: encapsulating the table name corresponding to the local table name;

[0018] The dictionary list corresponds to the local table data, which is encapsulated in key-value pairs;

[0019] The primary key list is encapsulated in the table header dictionary corresponding to the local table;

[0020] The query conditions use key-value pairs to represent the filter conditions of multiple fields and are set according to the data update method;

[0021] Different encapsulation operations are performed on the dictionary list according to different data update methods.

[0022] As a preferred solution of the data aggregation and integration optimization method of the present invention, the server receives the message, and runs the monitoring framework based on the first format to parse the message to obtain field information, including:

[0023] Parse the request method, URL, request header and request body in sequence according to the HTTP message format;

[0024] Get the length of the request body through the content-length value in the request header, which is used to read the encapsulated format string;

[0025] Based on the obtained character string, parse the information of the five fields;

[0026] Based on the data update method information, call the corresponding program to update and maintain the data.

[0027] As a preferred solution of the data aggregation and integration optimization method described in the present invention, the first format includes five fields: table name, data update method, dictionary list, primary key list and query condition.

[0028] As a preferred solution of the data aggregation, integration and optimization method described in the present invention, wherein: the first format is JSON format.

[0029] As a preferred solution of the data aggregation and integration optimization method described in the present invention, the data update and maintenance includes: data addition, data query, data update and data deletion.

[0030] In a second aspect, the present invention provides a data aggregation integrated optimization system, comprising:

[0031] A configuration module, used for setting the server-side meta-configuration information, and storing it in a first format;

[0032] A construction module is used to construct a database file on the server side, traverse and read all meta-configuration information, establish a corresponding data structure in the database file, and obtain a first database;

[0033] An encapsulation module, used for the client to read the local table data, encapsulate it in the first format, and package it into a message;

[0034] A parsing module, configured to receive the message at the server end, and to run the monitoring framework based on the first format to parse the message to obtain field information;

[0035] A first updating module, used for modifying the first database based on different field information to obtain an updated second database;

[0036] A second update module, configured to, in response to a push command, cause the server to push the data of the second database to a remote database, and to construct a database file in the remote database according to the meta-configuration information, so as to construct or update the remote database;

[0037] A download module is used to export and download data based on the second database.

[0038] In a third aspect, the present invention provides a computing device, comprising:

[0039] Memory and processor;

[0040] The memory is used to store computer executable instructions, and the processor is used to execute the computer executable instructions. When the computer executable instructions are executed by the processor, the steps of the data aggregation and integration optimization method are implemented.

[0041] In a fourth aspect, the present invention provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the data aggregation and integration optimization method.

[0042] Compared with the prior art, the present invention has the following beneficial effects: the present invention defines the database, table, and field structure through meta-configuration information. When the demand is adjusted, only the configuration file needs to be modified without adjusting the code, and it has high scalability and ease of operation; by developing a lightweight HTTP framework, the message receiving, parsing, and processing processes are realized, and structured data on different computers can be collected and transmitted to the server for aggregation, thereby realizing cross-platform and cross-system data integration; the transmission format is consistent, and it can be integrated with other automation programs, and the scheduled tasks can also be manually set to execute the client program. It can be widely used in various data management scenarios, providing underlying technical support for big data analysis, and compared with other similar applications, it occupies less performance resources and is faster to deploy and implement. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0044] Figure 1 A schematic diagram of the overall process logic of the data aggregation, integration and optimization method according to an embodiment of the present invention;

[0045] Figure 2 A schematic diagram of the overall interaction structure in the data aggregation, integration and optimization method according to an embodiment of the present invention;

[0046] Figure 3 A schematic diagram of the encapsulation process in the data aggregation, integration and optimization method according to an embodiment of the present invention;

[0047] Figure 4 The present invention is a schematic diagram of a data query and update process in a data aggregation and integration optimization method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0048] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0049] Reference Figure 1-Figure 4 , as an embodiment of the present invention, provides a data aggregation and integration optimization method, comprising:

[0050] S100: Setting server-side meta-configuration information and storing it in a first format;

[0051] S200: constructing a database file on the server side, traversing and reading all meta-configuration information, establishing a corresponding data structure in the database file, and obtaining a first database;

[0052] Preferably, the database file can be a Sqlite database;

[0053] In another implementation, the database file may also be a lightweight database such as TDB (TinyDB) or SQLCipher.

[0054] For example, after the server is started, this step will first back up the current stored data, then create a new database file, traverse and read all meta-configuration information, establish the corresponding data structure in the database file, and all subsequent data addition, deletion, modification and query operations are completed based on this database file. When the meta-configuration information changes, just repeat the above operations without modifying any code.

[0055] S300: The client reads the local table data, encapsulates it in the first format, and packages it into a message;

[0056] S400: The server receives the message and runs the monitoring framework based on the first format to parse the message to obtain field information;

[0057] S500: modifying the first database based on different field information to obtain an updated second database;

[0058] S600: In response to the push command, the server pushes the data of the second database to the remote database, and constructs a database file in the remote database according to the meta-configuration information to construct or update the remote database;

[0059] S700: Export and download data based on the second database.

[0060] It should be noted that for data that needs to be transmitted and summarized through the network, it is usually stored in multiple computer terminal file systems in office document format, and the data needs to be updated to a remote database to facilitate business personnel to view the latest data. A data download function should also be provided, so it needs to be achieved through the optimization of the above steps S100-S700.

[0061] It should be noted that, for the server, the database, table, and field information are defined through meta-configuration information. After the service program is started, this information will be automatically read and the corresponding structure will be created in the database. A lightweight monitoring framework is used to gradually encapsulate and implement request monitoring, message parsing, data processing and other processes from the bottom layer. The framework can support HTTP and provide basic support for subsequent integration with other systems. At the same time, the database is used to store the data sent by the client. Due to the small amount of data collected, scalability and ease of development and deployment are taken into consideration. On the one hand, data can be directly written to the remote database to achieve integration with the database; on the other hand, based on the unified data transmission and exchange format, the HTTP interface is used to implement the add, delete, modify, and query interface to achieve integration with other business systems.

[0062] It should be noted that, for the client, through a unified data transmission exchange format and clear field information, it only needs to select the field name according to the actual scenario requirements, and read and parse the local structured and unstructured data through the client, and encapsulate it into a dictionary type, which can be integrated into this framework, fully ensuring scalability.

[0063] In the embodiment of the present application, the meta-configuration information in the above step S100 is composed of table configurations, each table includes multiple configurations, and each configuration information can be dynamically adjusted according to actual business needs.

[0064] Preferably, the multiple configurations may include four expressions: table name (tablename), field name (colname), primary key name (unique) and whether to integrate (isupload).

[0065] In another practicable manner, the specific configuration may also be modified or additionally expressed for the above four types based on actual needs.

[0066] It should be noted that each configuration information can be dynamically adjusted according to actual business needs to achieve consistency with the data structure stored in the terminal.

[0067] In the embodiment of the present application, the first format in the above step S100 is a JSON format.

[0068] It should be noted that the JSON format is conducive to integration with other systems. Currently, most system APIs are based on the JSON format, which can provide a basis for subsequent integration.

[0069] In another practicable manner, the first format may also be XML or YAML format, which is also compatible with the HTTP protocol.

[0070] In the embodiment of the present application, the first format in the above step S300 includes five fields: table name, data update method, dictionary list, primary key list and query condition.

[0071] Specifically, the data transmission exchange format based on the above JSON includes five fields: table, method, data, unique, and condition, where table is the table name; method is the data update method, which is divided into insert, query, update, and delete types, namely, add, query, modify, and delete operations; data is a dictionary list, representing actual business data, where each dictionary instance is composed of a key-value pair consisting of a field and a value, which can form a row of data in the database table; unique is a primary key list, which is used to identify the uniqueness of the data. When traversing the primary key list to compare two dictionary instances, if the corresponding values ​​are equal, it means that the two instances are equal. If they are not equal, it means that this instance has not been stored and a data insertion operation needs to be performed; condition is the query condition, which is also in a dictionary format. It uses a key-value pair to represent the screening conditions of multiple fields and needs to be set when method is query or delete.

[0072] In the embodiment of the present application, the encapsulation in the above step S300 includes the following steps A1-A5:

[0073] A1: The table name is encapsulated according to the local table name;

[0074] A2: The dictionary list corresponds to the local table data, which is encapsulated as key-value pairs;

[0075] A3: The primary key list is encapsulated in the table header dictionary of the local table.

[0076] A4: The query condition uses key-value pairs to represent the filter conditions of multiple fields and is set according to the data update method.

[0077] A5: Perform different encapsulation operations on the dictionary list according to different data update methods.

[0078] For example, Figure 3 As shown in the figure, it is an example of the Excel data encapsulation process, where the worksheet name is "Table A", which is a two-dimensional data table with 2 rows and 4 columns. The encapsulated JSON text is shown in the figure on the right. Its data field is a list composed of two dictionaries, corresponding to the 1st and 2nd rows of data in the source table respectively. Each dictionary contains the values ​​corresponding to "Field A", "Field B", "Field C", and "Field D". When the method is update, in order to distinguish between the two operations of adding data and updating data, the unique field is needed to describe the uniqueness of the data. In this example, "Field A" and "Field B" are used together to form a unique identifier; when there is another data "Field A" and "Field B" that are equal to the current data, it means that the two data are the same entity, and only "Field C" and "Field D" need to be updated. Otherwise, it means that the data are not equal, and an update operation needs to be performed at this time.

[0079] In the embodiment of the present application, in the above step S400, the server receives the message and runs the monitoring framework based on the first format to parse the message to obtain field information, including the following steps B1-B4:

[0080] B1: Parse the request method, URL, request header and request body in sequence according to the HTTP message format;

[0081] B2: Get the length of the request body through the content-length value in the request header, which is used to read the encapsulated format string;

[0082] B3: Based on the obtained string, parse the information of five fields;

[0083] B4: Based on the data update method information, call the corresponding program to update and maintain the data.

[0084] Preferably, the message format may be an HTTP POST message.

[0085] In another practicable manner, the message format may also be an HTTP PUT or HTTP PATCH message.

[0086] In the embodiment of the present application, the data update maintenance in the above step S400 includes: data addition, data query, data update and data deletion.

[0087] Specifically, the above four update maintenance operations correspond to the data update method (method) field enumeration values ​​insert, query, update, and delete respectively.

[0088] In the embodiment of the present application, in the above step S500, the first database is modified based on different field information to obtain an updated second database;

[0089] Specifically, the specific steps of the above data update and maintenance operation are as follows:

[0090] ① Add data (insert): extract the two fields table and data, that is, the table name and data, and check whether the table name exists in the meta configuration information. If it exists, directly insert the data into the database file. If it does not exist, an error is reported.

[0091] ②Data query (query): extract the two fields table and condition, that is, table name and query condition. Similarly, check whether the table name is in the meta configuration information. If it does not exist, an error is directly reported. If it exists, the condition is encapsulated as an SQL query condition.

[0092] For example, Figure 4 The figure shows a simple query conversion example, which specifically describes how a data query request is converted into an SQL statement. After the SQL statement is converted and executed to obtain the query result, the server encapsulates the query return data into a dictionary list through a similar encapsulation process and returns it to the client program.

[0093] ③Data update (update): extract the three fields of table, data, and unique, namely the table name, dictionary list, and primary key list. Similarly, check whether the table name is in the meta configuration information. If it does not exist, an error is directly reported. If it exists, retrieve the data in data and compare them with the data in the database one by one. The comparison process will be carried out through the primary key list set in unique, which consists of multiple fields and can be selected by the client program. These fields together constitute a unique identifier. When two pieces of data are compared, the corresponding values ​​of the fields are compared one by one. If all values ​​are equal, it means that the two pieces of data are the same entity, otherwise they are different entities;

[0094] When traversing the data dictionary list, if the same entity can be retrieved from the database, the dictionary will be marked as updated. If the same entity cannot be found, the dictionary will be marked as added. After the traversal is completed, the client program will complete the database record update and database record insertion operations according to the corresponding tag values.

[0095] ④Data deletion (delete): Extract the two fields of table and condition, namely the table name and query condition. Similar to the query handler, it first converts the condition into an SQL condition and then calls the database to complete the data deletion operation.

[0096] It should also be noted that for client programs, update and delete can already implement data aggregation functions. When a unique field is set, update messages can be used to insert new data and update existing data, and delete messages can be used to delete expired data. The insert and query messages are mainly used for integration with other systems and are not currently used in client programs.

[0097] In the embodiment of the present application, in response to the push command in the above step S600, the server pushes the data of the second database to the remote database, and constructs a database file in the remote database according to the meta-configuration information to construct or update the remote database;

[0098] Exemplarily, after the server backend completes data update and maintenance, it can call the psycopg2 library interface to update the data to the remote database in the above-mentioned S400-S500 manner. In order to adapt to the customized adjustment of the database table structure, the original data needs to be backed up and all tables need to be deleted before pushing the data. During the data pushing process, the library table is created in the remote database according to the set meta-configuration information, and then data insertion is performed.

[0099] The push command can be triggered by the client sending a specific request, where the host IP, port number, and database name are defined in the configuration file. When the meta configuration information is "upload" is "yes", it means that the table data needs to be pushed to the remote database; when it is "no", it means that the table data does not need to be pushed to the remote database and is only stored locally.

[0100] In the embodiment of the present application, the data export and download in the above step S700 can be implemented in the following way: the data export stage first traverses all data tables and writes the database records into an office file (such as Excel), where each data table corresponds to a worksheet in the office file (such as Excel); the data download directly writes the HTTP GET interface, and the specific process is to read the file binary data, set the value of Content-Length, and assemble it into an HTTP response message and send it to the client. Business personnel only need to use a browser to access the download link to obtain the latest data.

[0101] In summary, in order to improve scalability during the design and development process, the above methods all uniformly obtain data structures through meta-configuration information. Table names, field names, unique fields, etc. are all defined by meta-configuration information. When the data structure is adjusted due to changes in requirements, no code needs to be modified, but the set data structure can be flexibly adjusted; a lightweight database is used to store and update data, and HTTP message reception, parsing, and processing programs are encapsulated from the bottom layer. No third-party WEB framework is used, which is lighter than other similar products and only requires a small amount of system resources; the interface design follows a unified specification, and it is preferred to represent the request content and response content in JSON format, and the meaning of the field is clear; it can be implemented in pure Python language, and after being packaged by Pyinstaller, it forms an exe file, which can be deployed and run by double-clicking, and the deployment and implementation are convenient; the modular programming idea is adopted, and the coupling degree between the functional modules is low, and it can be further upgraded and optimized according to actual business needs in the future; the configuration modification is convenient and readable, and users do not need to have programming knowledge to easily modify the meta-configuration information; the entire process can use the open source Python programming language, and the development and maintenance costs are low, and it does not rely on any commercial software.

[0102] The above is a schematic scheme of a data aggregation and integration optimization method of this embodiment. It should be noted that the technical scheme of the data aggregation and integration optimization system and the technical scheme of the above-mentioned data aggregation and integration optimization method belong to the same concept. For details not described in detail in the technical scheme of the data aggregation and integration optimization system in this embodiment, please refer to the description of the technical scheme of the above-mentioned data aggregation and integration optimization method.

[0103] The data aggregation, integration and optimization system in this embodiment includes:

[0104] A configuration module, used for setting the server-side meta-configuration information, and storing it in a first format;

[0105] A construction module is used to construct a database file on the server side, traverse and read all meta-configuration information, establish a corresponding data structure in the database file, and obtain a first database;

[0106] The encapsulation module is used for the client to read the local table data, encapsulate it in the first format, and package it into a message;

[0107] A parsing module, used for receiving a message at the server end, and running the monitoring framework based on the first format to parse the message to obtain field information;

[0108] A first updating module, used for modifying the first database based on different field information to obtain an updated second database;

[0109] A second update module, configured to, in response to a push command, cause the server to push the data of the second database to the remote database, and to construct a database file in the remote database according to the meta-configuration information, so as to construct or update the remote database;

[0110] The download module is used to export and download data based on the second database.

[0111] This embodiment further provides a computing device suitable for data aggregation, integration and optimization, including:

[0112] Memory and processor; the memory is used to store computer executable instructions, and the processor is used to execute computer executable instructions to implement the data aggregation and integration optimization method proposed in the above embodiment.

[0113] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, the data aggregation and integration optimization method proposed in the above embodiment is implemented.

[0114] The storage medium proposed in this embodiment and the method for implementing data aggregation, integration and optimization proposed in the above embodiment belong to the same inventive concept. The technical details not fully described in this embodiment can be found in the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0115] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course can also be implemented by hardware. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ReadOnly, Memory, ROM), random access memory (Random Access Memory, RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform the methods of various embodiments of the present invention.

[0116] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A data aggregation and integration optimization method, characterized in that: include: Setting the server meta-configuration information to be stored in a first format; Build a database file on the server side, traverse and read all meta-configuration information, establish a corresponding data structure in the database file, and obtain a first database; The client reads the local table data, encapsulates it in the first format, and packages it into a message; The server receives the message and runs the monitoring framework based on the first format to parse the message to obtain field information; Modifying the first database based on different field information to obtain an updated second database; In response to the push command, the server pushes the data of the second database to the remote database, and constructs a database file in the remote database according to the meta-configuration information to construct or update the remote database; Data is exported and downloaded based on the second database.

2. The data aggregation and integration optimization method according to claim 1, characterized in that: The meta-configuration information is composed of table configurations, each table includes multiple configurations, and each configuration information can be dynamically adjusted according to actual business needs.

3. The data aggregation and integration optimization method according to claim 2, characterized in that: The encapsulation includes: encapsulating the table name corresponding to the local table name; The dictionary list corresponds to the local table data, which is encapsulated in key-value pairs; The primary key list is encapsulated corresponding to the header dictionary of the local table; The query conditions use key-value pairs to represent the filter conditions of multiple fields and are set according to the data update method; Different encapsulation operations are performed on the dictionary list according to different data update methods.

4. The data aggregation and integration optimization method according to claim 3, characterized in that: The server receives the message and runs the monitoring framework based on the first format to parse the message to obtain field information, including: Parse the request method, URL, request header and request body in sequence according to the HTTP message format; Get the length of the request body through the content-length value in the request header, which is used to read the encapsulated format string; Based on the obtained character string, parse the information of the five fields; Based on the data update method information, call the corresponding program to update and maintain the data.

5. The data aggregation and integration optimization method according to claim 4, characterized in that: The first format includes five fields: table name, data update method, dictionary list, primary key list and query condition.

6. The data aggregation and integration optimization method according to claim 1 or 5, characterized in that: The first format is JSON format.

7. The data aggregation and integration optimization method according to claim 6, characterized in that: The data update and maintenance includes: data addition, data query, data update and data deletion.

8. A system using the data aggregation and integration optimization method according to any one of claims 1 to 7, characterized in that: include: A configuration module, used for setting the server-side meta-configuration information, and storing it in a first format; A construction module is used to construct a database file on the server side, traverse and read all meta-configuration information, establish a corresponding data structure in the database file, and obtain a first database; An encapsulation module, used for the client to read the local table data, encapsulate it in the first format, and package it into a message; A parsing module, configured to receive the message at the server end, and to run the monitoring framework based on the first format to parse the message to obtain field information; A first updating module, used for modifying the first database based on different field information to obtain an updated second database; A second update module, configured to, in response to a push command, cause the server to push the data of the second database to a remote database, and to construct a database file in the remote database according to the meta-configuration information, so as to construct or update the remote database; A download module is used to export and download data based on the second database.

9. An electronic device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the data aggregation and integration optimization method described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the data aggregation and integration optimization method described in any one of claims 1 to 7.

Images