Data application method and system based on agile construction

Abstract

The invention discloses a data application method and system based on agile construction, which support one-stop rapid acquisition, processing, analysis, storage and display of data through combination of an agile development mode and a low-code technology, and realize whole-process agile management of the data. Comprising the following steps: configuring a data source, mapping a data set into a global variable of a low-code platform, binding the data source and a front-end component, processing data, converting and optimizing the data, displaying the data, and internally arranging a monitoring tool to ensure the security of the data; according to the method, the development efficiency, agility, safety and flexibility of data application can be effectively improved, and the development period is shortened.

Description

Technical Field

[0001] The present invention relates to a data application method and system, and in particular to a data application method and system based on agile construction. Background Art

[0002] With the rapid development of information technology, the importance of data in various fields has become increasingly prominent. How to use data efficiently has become a major challenge facing enterprises and organizations. The traditional data application development process still uses management processes such as security protection and third-party testing of digital projects, resulting in high management costs, long development cycles, and difficulty in promotion, which cannot meet the market's demand for agility and flexibility. Summary of the invention

[0003] Purpose of the invention: The purpose of the invention is to provide a data application method based on agile construction to improve the agility, security and flexibility of data application, and on the other hand, to provide a data application system based on agile construction.

[0004] Technical solution: The data application method based on agile construction described in the present invention comprises the following steps:

[0005] (1) Apply to import a data source. The management platform will review and approve the application. After approval, the configuration is completed based on the information provided by the data source.

[0006] (2) Select a database from the data source, form a data set through SQL query, and map the data set to a global variable of the agile construction platform;

[0007] (3) Bind the data source and front-end components through the wizard tool, and different components obtain the required data by accessing global variables;

[0008] (4) Use node configuration and data processing modules to process data, display the processing process through a visual interface, and customize the data processing flow according to needs;

[0009] (5) Convert data sets according to requirements and preview the converted data effects in real time;

[0010] (6) By dragging and dropping components, the data is configured into data reports according to the components or templates provided by the component material R&D platform, and the data is published on the page after selecting the publishing mode according to the needs. The status and changes of the data are monitored in real time through monitoring tools.

[0011] Preferably, the data sources in step 1 include relational database MySQL, relational database PostgreSQL, non-relational database MongoDB, big data platform Hadoop, big data platform Spark, and real-time streaming data Kafka.

[0012] Preferably, the data source importing method described in step 1 includes API data usage, static file import and data set method.

[0013] Preferably, step 2 also includes operations of cleaning, converting and validating the data set.

[0014] Preferably, the data processing module in step 4 includes data aggregation, data filtering, data grouping, data sorting and data calculation.

[0015] Preferably, step 5 also includes performing custom conversion on the data set via JavaScript on the page.

[0016] The data application system based on agile construction described in the present invention includes:

[0017] The management platform is used to manage data source related affairs, including receiving data source import applications, approving them according to preset approval rules, completing data source configuration after approval, recording approval opinions and personnel information. The management platform is also responsible for managing user information in the system.

[0018] Storage platform, used to store data source configuration information and data sets;

[0019] Agile construction platform, used to provide an operation interface, supports building data reports by dragging and dropping components, and has a data set mapping function. It can map the data set obtained and processed from the data source to a global variable, and use the wizard tool to bind the data source to the front-end component, so that the front-end component can access the global variable to obtain the required data and publish the configured data report to the page. It also provides data processing flow customization function, uses node configuration and data processing modules to process data, and displays the processing process through a visual interface;

[0020] Component material R&D platform, used to provide component resources and function expansion and meet the personalized needs of large-screen reports. The produced components can be exported to the agile construction platform for use;

[0021] Front-end components include interface components, layout components, data display components and controller components. The interface components are basic UI elements, the layout components are used to organize other interface elements, and the controller components are used to perform personalized control on components.

[0022] Monitoring tools, used to monitor the status and changes of data in real time;

[0023] Debugging platform, used for functional debugging and performance debugging.

[0024] Beneficial effects: Compared with the prior art, the present invention has the following advantages: 1. Reduce the complexity and cost of data application development; 2. Improve data application development efficiency and shorten the development cycle through agile construction methods; 3. Improve the flexibility of data applications; 4. Improve the security of data applications through permission control and approval, logging and monitoring. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 This is a schematic diagram of the agile construction platform of the present invention;

[0026] Figure 2 This is a schematic diagram of the management platform of the present invention;

[0027] Figure 3 Schematic diagram of the controller assembly of the present invention. DETAILED DESCRIPTION

[0028] The technical solution of the present invention is described in detail below in conjunction with the accompanying drawings.

[0029] The present invention can be applied to the energy field, especially the equipment status monitoring and maintenance in the power industry. For example, taking the power transformer as an example, the aging degree and remaining life of its key components are predicted, including collecting the operating parameters of the power transformer under different operating states, extracting representative characteristic indicators, and being able to automatically detect changes in the equipment status and update data; repeating the data update and model iteration process, continuously improving the prediction accuracy, and realizing accurate prediction of the aging status of the power transformer, especially the severe aging status, so as to provide strong support for the maintenance and management of power equipment and ensure the stable operation of the power system. The specific steps are as follows:

[0030] Taking the data application development of an energy enterprise as an example, the energy production data and energy consumption data within the enterprise are imported into the system for analysis and display. First, the developer submits an application for importing the data source through the data source management interface of the agile construction platform, selects the data source type as the relational database MySQL, fills in the data source name as "energy_data_source", and the connection address is the IP address and port number of the internal database server of the enterprise, and provides authentication information such as the database user name and password. After the application is submitted, according to the preset approval process, the department manager logs in to the platform to view the application, checks the integrity and rationality of the data source information, and passes the preliminary review after confirmation. Then the system administrator conducts the final review. In addition to checking the data source information again, the security detection tool of the system is used to evaluate the security of the data source connection, and the application is approved after ensuring that there are no security vulnerabilities; after the data source is approved, the system automatically completes the configuration according to the information provided. During the configuration process, the system tests the connection with the data source to ensure that the connection can be successfully established and obtain basic information of the database, such as the database name, table structure, etc. At the same time, the system generates the corresponding data source configuration object internally according to the data source type and connection information, and stores it in the configuration file so that subsequent data operations can accurately access the data source.

[0031] After the developer completes the data source configuration, they enter the dataset generation phase. In the dataset management interface of the agile build platform, select the configured "energy_data_source" data source, then select the "production_data" table containing energy production data and the "consumption_data" table containing energy consumption data from the database, build a dataset through SQL query statements, and extract the energy production data for the whole year of 2023. After the construction is complete, the dataset is named "2023_energy_production_dataset" and mapped to the global variable "energyProductionData" of the low-code platform. The system then automatically identifies the data type in the dataset, such as integers, floating point numbers, strings, dates, etc., and performs necessary data type conversions according to the data requirements of the component. At the same time, a unique identifier "energyProductionData_2023" is set for the global variable, and the access level is set according to the user's permissions to ensure that only authorized users can access the global variable.

[0032] Developers use the wizard tool of the agile construction platform to bind the data source to the front-end components. In the report design interface, drag a bar chart component and a line chart component from the component library to the canvas. Through the wizard tool, select the "energyProductionData" global variable as the data source, bind the "production_date" field in the energy production data to the horizontal axis of the bar chart as the time axis, and bind the "production_quantity" field to the vertical axis of the bar chart to represent energy production; bind the "consumption_date" field in the energy consumption data to the horizontal axis of the line chart, and bind the "consumption_amount" field to the vertical axis of the line chart to represent energy consumption. During the binding process, the wizard tool automatically detects the compatibility of the data format. When the date format is inconsistent, it prompts the developer to convert the format. The developer selects the automatic conversion function, and the system converts the date format in the consumption data to the same format as the production data. After the binding is completed, the developer adjusts the position and size of the component by dragging and dropping, and sets the style of the component, such as the color and font size of the bar chart, the line color and marker style of the line chart, etc. At the same time, using the layout function of the platform, the two components are reasonably arranged on the report page, and elements such as titles and legends are added to complete the preliminary design of the report.

[0033] In order to further analyze the energy data, developers use the data processing module to process the data set. For example, add a data cleaning node, configure the filtering conditions, and remove the outliers in the energy production data; add a data aggregation node to aggregate the energy production data and consumption data by month, and calculate the total output and total consumption per month; add a data calculation node to calculate the energy production efficiency. By dragging and dropping these nodes, the data processing flow is constructed, and the processing process is displayed through a visual interface. Developers can clearly see the flow and conversion of data between various nodes. During the node configuration process, developers set the outlier judgment threshold of the data cleaning node, the aggregation method and time granularity of the data aggregation node, and the calculation formula of the data calculation node. After processing the data, each node outputs the processing results and detailed log information to the console, which is convenient for developers to track and debug. For example, the data cleaning node outputs the number and specific data of outliers filtered out, the data aggregation node outputs the results of monthly aggregation, and the data calculation node outputs the calculated energy production efficiency value. If the processing results are found to be not as expected, developers can quickly locate the problem node based on the log information and adjust the node parameters or processing logic.

[0034] According to business needs, developers need to convert datasets to a specific format in order to interact with other systems. In the dataset conversion interface of the agile construction platform, select the "2023_energy_production_dataset" dataset, perform the data format conversion operation, and convert it to JSON format. During the conversion process, select the fields to be included, sort the data, and group them. Developers can preview the effect of the converted data in real time, and check whether the data format, field content, sorting, and grouping results meet expectations by comparing the data before and after the conversion. For example, the data is in table form before conversion and is displayed in JSON format after conversion. Developers can check whether the JSON structure is correct and whether the field values ​​in each data item are accurate. At the same time, the system automatically records the detailed steps and parameter settings of the conversion operation as a script. Developers can save the script so that they can directly reuse it when the same or similar datasets need to be converted in the future, thereby improving work efficiency.

[0035] After the report design and data processing are completed, the developer selects the publishing mode as immediate publishing and publishes the report to the energy data monitoring page within the enterprise. During the publishing process, the system interacts with the deployment environment on the server side and uploads the report-related files to the specified server directory. After the publishing is completed, the system records detailed information such as the publishing time, publishing status, and publishing target page in the publishing log. After the report is published, the data monitoring module begins to monitor the status and changes of the data in real time. Monitoring indicators include data accuracy, completeness, timeliness, data flow, and data fluctuations. For example, if it is found that the energy production on a certain day suddenly drops sharply and exceeds the normal fluctuation range, the monitoring module will notify the developer and relevant managers through emails and text messages. At the same time, the system supports statistical analysis of historical data status and changes, and generates data trend charts, report access statistics and other charts to help managers understand the long-term trend of data changes and the use of reports so as to make corresponding decisions.

[0036] System administrators regularly check and maintain configured data sources. For example, data source connections are tested weekly to ensure the stability of the data source. If an abnormality is found in the data source connection, such as connection timeout, inability to obtain data, etc., the administrator promptly investigates the cause of the problem. Possible causes include network failure, data source server failure, data source configuration information change, etc. If it is a network failure, the administrator checks the operating status of the network equipment and repairs the network connection; if it is a data source server failure, contact the data source provider for assistance; if it is a data source configuration information change, the administrator updates the data source configuration in the system according to the new information. As the business of the enterprise develops, it may be necessary to add new data sources or modify the configuration of existing data sources. For example, the enterprise has newly connected an energy IoT device, and its data is stored in a new non-relational database. The administrator follows the data source import process, adds a new data source type, configures the connection information, and after approval, the system can include the data source in the management scope, and developers can use the data of the data source in data application development.

[0037] As user needs continue to change, the front-end component library needs to be constantly updated and expanded. For example, users reported that the existing chart component was not intuitive enough when displaying complex energy data relationships, so the development team decided to develop a new interactive chart component. Developers use the component development tools provided by the agile construction platform to develop new components based on technologies such as React and Echarts. After the development is completed, the new component is uploaded to the component library through the component management interface of the platform, and the component description information and configuration parameters are added and reviewed. After the review is passed, the new component can be used by users in report design. At the same time, existing components are optimized and updated. For example, when it is found that the performance of a component degrades under large data volumes, the developer optimizes the component code and improves the component performance by adopting data caching strategies, optimizing rendering algorithms, and other methods. The updated component is released to the component library after testing. When users use the report design function, the system will prompt that the component has been updated, and users can choose to update the component for a better user experience.

[0038] As the amount of data increases and the number of user visits grows, system performance may be affected. In order to optimize system performance, the database is optimized regularly, such as rebuilding indexes, optimizing query statements, cleaning up useless data, etc., to improve data query and storage efficiency. For example, by analyzing the slow query logs of the database, it is found that some complex queries take a long time to execute. Developers optimize these queries, adjust query conditions, add appropriate indexes, etc., to significantly improve the query speed. At the application level, optimize the algorithms and processes of the data processing module. For example, optimize the algorithms of the data aggregation nodes, use parallel computing technology, and process the aggregation operations of multiple data groups on a multi-core processor at the same time to reduce data processing time. At the same time, optimize the loading speed of the front-end page, and improve the user experience by compressing pictures, merging scripts and style files, and enabling browser caching. In addition, stress test the system to simulate the situation of a large number of users accessing concurrently, and adjust the server configuration and system parameters according to the test results to ensure that the system can still run stably under high load.

Claims

1. A data application method based on agile construction, characterized in that: The following steps are involved: (1) Apply to import a data source. The management platform will review and approve the application. After approval, the configuration is completed based on the information provided by the data source. (2) Select a database from the data source, form a data set through SQL query, and map the data set to a global variable of the agile construction platform; (3) Bind the data source and front-end components through the wizard tool, and different components obtain the required data by accessing global variables; (4) Use node configuration and data processing modules to process data, display the processing process through a visual interface, and customize the data processing flow according to needs; (5) Convert data sets according to requirements and preview the converted data effects in real time; (6) By dragging and dropping components, the data is configured into data reports according to the components or templates provided by the component material R&D platform, and the data is published on the page after selecting the publishing mode according to the needs. The status and changes of the data are monitored in real time through monitoring tools.

2. The data application method based on agile construction according to claim 1 is characterized in that: The data sources in step 1 include relational database MySQL, relational database PostgreSQL, non-relational database MongoDB, big data platform Hadoop, big data platform Spark and real-time streaming data Kafka.

3. The data application method based on agile construction according to claim 1 is characterized in that: The data source importing methods described in step 1 include API data usage, static file import and data set methods.

4. The data application method based on agile construction according to claim 1 is characterized in that: Step 2 also includes operations to clean, transform, and validate the dataset.

5. The data application method based on agile construction according to claim 1, characterized in that: The front-end components described in step 3 include interface components, layout components, data display components and controller components.

6. The data application method based on agile construction according to claim 1, characterized in that: The data processing module described in step 4 includes data aggregation, data filtering, data grouping, data sorting and data calculation.

7. The data application method based on agile construction according to claim 1 is characterized in that: In step 5, data conversion includes data format conversion, data encoding conversion, and data unit conversion, and conversion parameters can be adjusted during real-time preview.

8. The data application method based on agile construction according to claim 1, characterized in that: Step 5 also includes custom transformations of the dataset via JavaScript on the page.

9. The data application method based on agile construction according to claim 1, characterized in that: The publishing modes described in step 6 include scheduled publishing, manual publishing, and event-triggered publishing.

10. A data application system based on agile construction, characterized in that: include: The management platform is used to manage data source related affairs, including receiving data source import applications, approving them according to preset approval rules, completing data source configuration after approval, recording approval opinions and personnel information. The management platform is also responsible for managing user information in the system. Storage platform, used to store data source configuration information and data sets; Agile construction platform, used to provide an operation interface, supports building data reports by dragging and dropping components, and has a data set mapping function. It can map the data set obtained and processed from the data source to a global variable, and use the wizard tool to bind the data source to the front-end component, so that the front-end component can access the global variable to obtain the required data and publish the configured data report to the page. It also provides data processing flow customization function, uses node configuration and data processing modules to process data, and displays the processing process through a visual interface; Component material R&D platform, used to provide component resources and function expansion and meet the personalized needs of large-screen reports. The produced components can be exported to the agile construction platform for use; Front-end components include interface components, layout components, data display components and controller components. The interface components are basic UI elements, the layout components are used to organize other interface elements, and the controller components are used to perform personalized control on components. Monitoring tools, used to monitor the status and changes of data in real time; Debugging platform, used for functional debugging and performance debugging.

Images