Real-time data processing system based on Web architecture and Canvas technology

Abstract

This real-time data processing system, based on Web architecture Canvas technology, belongs to the fields of automated report logic engine technology and web page table processing technology. The invention includes a data extraction module, a data parsing module, a data storage module, a real-time display module, and an online modification module. The data extraction module dynamically extracts business data from the business system to be analyzed. The data parsing module parses the extracted business data using a built-in parsing model and a custom parsing model. The data storage module stores the business data and the results of the data parsing. The online modification module performs real-time online modification and statistical analysis on the data to be displayed, while synchronizing the modified data to the data storage module. The real-time display module provides an interface with the operator. This invention is compatible with historical data from traditional business personnel and allows business personnel to manually configure and display data results in real time.

Description

Technical Field

[0001] This invention relates to a real-time data processing system based on Web architecture Canvas technology, belonging to the fields of automated report logic engine technology and web page table processing technology. Background Technology

[0002] In today's enterprise-level application systems, reporting systems are crucial for understanding and evaluating various aspects of a company's production and sales processes. System information integration often requires information to be presented in report form. Traditional evaluation models primarily rely on paper-based methods or simple Excel spreadsheets. These methods suffer from poor sharing and real-time performance when documents are circulated among different people. While some companies may have online evaluation systems, these typically only involve inputting paper data into the online system, failing to allow business personnel to input necessary formulas for customized analysis and evaluation models, or to perform online analysis and comparison. Furthermore, they lack the ability for users to edit and modify data in real time.

[0003] Existing reporting systems have the following defects and shortcomings:

[0004] Poor usability: Existing reporting engines are cumbersome to develop and configure, difficult to learn, and cannot extract complex data logic through simple drag-and-drop operations and configurations. Furthermore, they cannot automatically generate assessment models or dynamically display rankings.

[0005] Poor interactivity: The existing assessment system only displays some data and cannot allow users to modify inaccurate data in real time and submit it to the backend database.

[0006] Poor customizability: The existing assessment system is based on a specific set of public statements and assessment indicator models developed by the system developers, which makes it impossible for business personnel to modify the assessment model at any time and generate analytical and statistical data that conforms to the latest assessment indicators.

[0007] Poor compatibility: Since traditional business analysts use paper spreadsheets or offline Excel files for data analysis and recording, many existing assessment systems are not well compatible with these Excel documents. Business personnel have to manually enter all the data into the system one by one, resulting in a large amount of repetitive work and increasing the workload of users. Summary of the Invention

[0008] The technical problem solved by this invention is to overcome the shortcomings of the prior art and provide a real-time data processing system based on Web architecture Canvas technology, which is compatible with historical data of traditional business personnel and allows business personnel to manually configure and display data results in real time.

[0009] The technical solution of this invention is: a real-time data processing system based on Web architecture Canvas technology, including a data extraction module, a data parsing module, a data storage module, a real-time display module, and an online modification module;

[0010] The data extraction module is used to dynamically extract business data from the business system to be analyzed.

[0011] The data parsing module is used to parse the extracted business data using the built-in parsing model and the custom parsing model;

[0012] The data storage module is used to store business data and the results of data parsing.

[0013] The online modification module is used to modify and perform real-time statistical analysis on the data to be displayed, and simultaneously synchronize the modified data to the data storage module.

[0014] The real-time display module is used to provide an interface with the operator.

[0015] Furthermore, the business systems to be analyzed include various business systems that need to be analyzed, or historical EXCEL files used by business personnel, as well as other custom business format data.

[0016] Furthermore, the data extraction method of the data extraction module includes:

[0017] Bind data source, front-end and back-end binding and validation, and real-time rendering of the interface; or

[0018] The front-end and back-end establish a long-lived socket connection to obtain real-time data; or

[0019] Use a custom function formula to request specific data via AJAX.

[0020] Furthermore, the built-in parsing model includes: customized logic development and data maintenance based on user data to generate Excel reports; the backend uses GoExcel and EasyPoI to parse the defined logic and convert it into custom SQL; the backend uses a high-availability algorithm to implement logic parsing; the backend uses a REST service developed based on the microservice concept, using Spring MVC methods to perform data extraction in a synchronous / asynchronous manner; and the frontend uses JavaScript, static HTML pages, and HTML5 canvas technology to draw and render the Excel file.

[0021] Furthermore, the custom parsing module includes: using multi-threading to execute the underlying cells, using the scriptEngine or AviatorEvaluator formula engine to execute the JS or Java formulas in the cells, executing the cell formulas layer by layer. After the first layer of data is executed, the cells in the second layer of data that reference the first layer of data are replaced, and the formula engine is used again to execute the formulas in multi-threading to obtain the second layer of data, and so on to obtain all the cell data.

[0022] Furthermore, the data storage module includes a master database and a slave database. The slave database is a backup of the master database and uses a non-relational database and an in-memory database for storage expansion.

[0023] Furthermore, the storage method of the data storage module includes: data caching and write-back through the Redis mechanism, and data access and persistence through Spring Data JPA.

[0024] Furthermore, in the data storage module, the data storage method in the Excel file is cell-based: the data storage cells and required formulas are pre-configured, the data parsed in the previous step is stored in the database, and returned to the real-time display module. At the same time, the corresponding types are set for differentiation during display. The image file is transcoded, and the transcoded file is stored. The transcoding process uses a base64 control and is returned to the real-time display module.

[0025] Furthermore, the real-time display module includes a front-end and a back-end;

[0026] The front-end uses Vue's `watch` method to listen for data changes passed from outside the table within the corresponding preset table processing component. If a data change is detected, and Vue's `watch` method indicates asynchronous data loading, it checks if initial configuration data exists. If not, it performs initial configuration of the table by obtaining the cell using `sheet.getCell`, setting the cell's `value` method to indicate selection, and calling the encapsulated `setCellAttribute` method to reset the cell's data. When an asynchronous loading change is detected in the table body configuration data, it resets the properties of the `XSheetUtil` instance, sets the table data, the visibility of row filters, row styles, cell text styles, freeze column separators, adds a summary row at the bottom of the table, and sets the cells of the rows contained in the table body.

[0027] The backend employs a heartbeat message sending mechanism to determine the heartbeat period after receiving the device's response information and to send heartbeat messages to the long-connection server according to the heartbeat period. A second long-connection disconnection determination module determines that the long-connection communication link is disconnected and calls the retry module if no heartbeat response message is received within a second time period. The long-connection server generates data corresponding to the business data and sends the business data and corresponding synchronization point information to the long-connection service module for display via the long-connection communication link. After page rendering, the data is modeled using common methods.

[0028] Furthermore, the data is modeled using common methods, and the display formats include tables, graphs, reports, and pivot tables.

[0029] The advantages of this invention compared to the prior art are:

[0030] (1) High ease of use: This invention is based on the Web browser B / S architecture. Compared with the traditional office C / S architecture, this technology can be used out of the box. It can achieve complex logic data extraction through simple drag-and-drop operations and configuration, and can automatically generate various functions in any scenario.

[0031] (2) High interactivity: This invention allows business personnel to customize the formula content to be analyzed, and the generated analysis data can be modified in real time and saved to the database. Custom statistical analysis reports can also be generated using this system, which has strong interactivity.

[0032] (3) Strong compatibility: This invention solves the data storage problem of large-scale report business systems. This system can support business personnel to directly import Excel files, saving them the workload of manual data entry and reducing the probability of errors. It facilitates data maintenance, query and statistics, increases data security, and provides a good foundation for enterprise data processing and mining. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of the system framework of the present invention;

[0034] Figure 2 This is a schematic diagram of the business system to be analyzed in this invention. Detailed Implementation

[0035] To better understand the above technical solutions, the technical solutions of this application will be described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments of this application and the specific features in the embodiments are detailed descriptions of the technical solutions of this application, rather than limitations on the technical solutions of this application. In the absence of conflict, the embodiments of this application and the technical features in the embodiments can be combined with each other.

[0036] The following description, in conjunction with the accompanying drawings, provides a more detailed account of the real-time data processing system based on Web architecture Canvas technology provided in this application. Specific implementation methods may include (e.g.) Figure 1 , 2 (As shown): In response to a trigger command that processes a table contained in a webpage, the system obtains a preset atomic method based on the table control; it listens to whether the table configuration data of the table is loaded asynchronously; if the table configuration data is loaded asynchronously, it obtains the asynchronous table configuration data corresponding to the asynchronous loading, and calls the preset atomic method to draw the table based on the asynchronous table configuration data. The system architecture of this invention is as follows: Figure 1 It is divided into the following modules: data extraction module, data parsing module, data storage module, real-time display module, and online modification module. Specifically:

[0037] Data extraction module. The source files for dynamically extracted resource data are stored on the established FTP server (JSON format configuration files). Data extraction can be performed in three ways: First, binding to a data source, with front-end and back-end binding and validation for real-time interface rendering; second, establishing a long-lived socket connection between the front-end and back-end to obtain real-time data; third, supporting custom function formulas for AJAX requests of specific data.

[0038] The business systems to be analyzed can include various systems such as asset management systems, OA office systems, and audit management systems; or historical Excel files used by business personnel, which can be directly imported into this assessment model, reducing the workload of manual data entry and minimizing potential errors; and other custom business format data, such as JSON, XML, and TXT. Specific content that can be included is as follows: Figure 2 As shown.

[0039] The data parsing module has pre-built parsing models to perform routine parsing on the extracted business data, such as developing customized logic based on user data, maintaining data, and generating Excel reports. The backend uses technologies like GoExcel and EasyPoI to parse the defined logic and convert it into custom SQL. The backend employs high-availability algorithms to implement the logic parsing. The backend uses a REST service developed based on the microservice concept, employing Spring MVC technology and using synchronous / asynchronous methods to perform data extraction. The frontend uses JavaScript, static HTML pages, and HTML5 canvas technology to render the Excel file.

[0040] It also allows business personnel to input custom formulas according to actual business needs for custom model parsing. The backend uses multi-threading to execute the underlying cells, using the scriptEngine or AviatorEvaluator formula engine to execute the JS or Java formulas in the cells, executing the cell formulas layer by layer. After the first layer of data is executed, the cells in the second layer that reference the first layer of data are replaced, and the formula engine is used again to execute the formulas in multi-threading to obtain the second layer of data, and so on to obtain the data of all cells.

[0041] The data storage module utilizes Redis for data caching and write-back, and Spring Data JPA for data access and persistence. The database includes a master database and slave databases. The master database stores all data, while the slave databases employ both NoSQL and in-memory databases for storage expansion. When a write request is received, the data is written to the master database, and the write operation sequence is recorded in the master database's local log file. A first file offset, which is the file offset of the master database data corresponding to the local log file, is updated. The master database's local log file is read, and the data corresponding to the newly added write operation is synchronized to the slave database based on the first file offset. This eliminates the need to store separate log files for each slave database in the master database, reducing the space required for the distributed data storage system. Furthermore, data writing to the master database and data synchronization to the slave database can be performed simultaneously without blocking data writing to the master database during synchronization, thus improving data writing and synchronization efficiency, avoiding wasted storage space, and further enhancing data consistency and scalability between the master and slave databases.

[0042] Excel files store data in cells. Typically, the data storage cells and required formulas are configured in the configuration page. The parsed data (JSON format) from the previous step is directly stored in the database (the database, table, and data format are obtained from the JSON mentioned earlier), and simultaneously returned to the front end for later display. Appropriate data types are also set to differentiate data during display, allowing the system to uniformly store and manage the final data. For images, transcoding is required. After transcoding, the transcoded file is stored in the file database. Existing base64 controls can be used during transcoding, and the data is returned to the front end for later display.

[0043] Real-time display module.

[0044] The front-end uses Vue's `watch` method to listen for data changes passed to the table from outside the pre-defined table processing component. If a data change is detected, and the `watch` method indicates asynchronous data loading, it can then determine if initial configuration data exists. If not, it performs initial configuration for the table by retrieving the cell using `sheet.getCell` (i.e., getting the x-th column of row y), setting the cell's `value` method to indicate selection, and calling the encapsulated `setCellAttribute` method to reset the cell's data. When an asynchronous loading change is detected in the table body configuration data, the `XSheetUtil` instance's properties can be reset, and settings can be applied to the table data, row filter visibility, row styles, cell text styles, frozen column separators, and a summary row added to the bottom of the table, among other things, to configure the cells in the rows contained within the table body.

[0045] The backend employs a heartbeat message sending mechanism to determine the heartbeat period upon receiving the device's response information and to send heartbeat messages to the long-connection server according to the heartbeat period. A second long-connection disconnection determination module determines that the long-connection communication link is disconnected and calls the retry module if no heartbeat response message is received within a second time period. The long-connection server generates data corresponding to the business data and sends the business data and corresponding synchronization point information to the long-connection service module for display via the long-connection communication link. After page rendering, the data can be modeled in common ways, and the display formats include tables, graphs, reports, pivot tables, etc.

[0046] Online modification module. This system allows users to modify displayed data online, perform real-time statistical analysis, and perform in-memory MD5 verification after modification. The cached database is updated in real time, and the final modified data can be stored in a relational database in real time, greatly improving system efficiency, interactivity for business users, and data security.

[0047] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

[0048] The contents not described in detail in this specification are common knowledge to those skilled in the art.

Claims

1. A real-time data processing system based on Web architecture Canvas technology, characterized in that, It includes a data extraction module, a data parsing module, a data storage module, a real-time display module, and an online modification module; The data extraction module is used to dynamically extract business data from the business system to be analyzed. The data parsing module is used to parse the extracted business data using the built-in parsing module and the custom parsing module; The data storage module is used to store business data and the results of data parsing. The online modification module is used to modify the data to be displayed in real time and perform real-time statistical analysis, while synchronizing the modified data to the data storage module. The real-time display module is used to provide an interface with the operator; The built-in parsing module develops and maintains customized logic based on user data and generates Excel reports; the backend uses GoExcel and EasyPoI to parse the defined logic and convert it into custom SQL; the backend uses a high-availability algorithm to implement the parsing of the logic; the backend REST service, developed based on the microservice concept, uses Spring MVC methods and performs data extraction in a synchronous / asynchronous manner; the frontend uses JavaScript, static HTML pages, and HTML5 canvas technology to draw and render the Excel file. The custom parsing module uses multi-threading to execute the underlying cells, and uses the scriptEngine or AviatorEvaluator formula engine to execute the JS or Java formulas in the cells, executing the cell formulas layer by layer. After the first layer of data is executed, the cells in the second layer of data that reference the first layer of data are replaced, and the formula engine is used again to execute the formulas in multi-threading to obtain the second layer of data, and so on to obtain all the cell data. The real-time display module includes a front-end and a back-end; The front-end uses Vue's `watch` method to listen for data changes passed from outside the table within the corresponding preset table processing component. If a data change is detected, and the `watch` method indicates asynchronous data loading, it checks if initial configuration data exists. If not, it performs initial configuration of the table by obtaining the cell using `sheet.getCell`, setting the cell's `value` method to indicate selection, and calling the encapsulated `setCellAttribute` method to reset the cell's data. When an asynchronous loading change is detected in the table body configuration data, it resets the properties of the `XSheetUtil` instance, sets the table data, the visibility of row filters, row styles, cell text styles, freeze column separators, adds a summary row at the bottom of the table, and sets the cells of the rows contained in the table body. The backend uses a heartbeat message sending mechanism to determine the heartbeat cycle after receiving the device response information, and sends heartbeat messages to the long connection server according to the heartbeat cycle. The second long connection disconnection determination module is used to determine that the long connection communication link is disconnected and to call the retry module when no heartbeat response message is received in response to the heartbeat message within the second time period. The long-connection server generates the data corresponding to the business data and sends the business data and the corresponding synchronization point information to the long-connection service module for display through the long-connection communication link; After the page is rendered, the data is modeled using common methods.

2. The real-time data processing system based on Web architecture Canvas technology according to claim 1, characterized in that, The business systems to be analyzed include various business systems that need to be analyzed, or historical EXCEL files used by business personnel, as well as other custom business format data.

3. The real-time data processing system based on Web architecture Canvas technology according to claim 1, characterized in that, The data extraction method of the data extraction module includes: Bind data source, front-end and back-end binding and validation, and real-time rendering of the interface; or The front-end and back-end establish a long-lived socket connection to obtain real-time data; or Use a custom function formula to request specific data via AJAX.

4. The real-time data processing system based on Web architecture Canvas technology according to claim 1, characterized in that, The data storage module includes a master database and a slave database. The slave database is a backup of the master database. The slave database uses a non-relational database and an in-memory database for storage expansion.

5. The real-time data processing system based on Web architecture Canvas technology according to claim 4, characterized in that, The storage methods of the data storage module include: data caching and write-back through the Redis mechanism, and data access and persistence through Spring Data JPA.

6. The real-time data processing system based on Web architecture Canvas technology according to claim 5, characterized in that, In the data storage module, the data storage method in the Excel file is cell-based: the data storage cells and the required formulas are pre-configured, the data parsed in the previous step is stored in the database, and returned to the real-time display module. At the same time, the corresponding types are set for differentiation during display. The image file is transcoded, and the transcoded file is stored. The transcoding process uses a base64 control and is returned to the real-time display module.

7. The real-time data processing system based on Web architecture Canvas technology according to claim 1, characterized in that, The data is modeled using common methods, and the display formats include tables, graphs, reports, and pivot tables.

Images