System for realizing rapid data construction and arrangement based on visual application development method

Abstract

The invention discloses a system for realizing rapid data construction and arrangement based on a visualization application development method, and particularly relates to the technical field of data arrangement, and the system comprises a data acquisition module, a data analysis module, a model construction module and a signal generation module. The method comprises the following steps of: extracting data quality information and extraction efficiency information from a database, analyzing the data quality information and the extraction efficiency information, and constructing a data analysis model by using a data quality scoring coefficient, a data conversion efficiency coefficient and a data extraction rate instability coefficient in the data quality information and the extraction efficiency information; according to the method, the performance of the constructed and arranged data in different time intervals is judged, the signal evaluation coefficient of the constructed and arranged data system is determined, signals of different grades are generated, the system performance is optimized, the data quality is ensured, related personnel are notified to process in time when problems occur, and the influence of system faults on services is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of data arrangement, and more specifically, to a system for realizing rapid construction and arrangement of data based on a visual application development method. Background Art

[0002] Colleges and universities usually have a variety of data sources, which may come from student management systems, financial systems, scientific research management systems, etc. These data are usually stored in different databases, using different data formats and structures, and may even be supported by different suppliers. Therefore, integrating these data into a visualization application may require facing data silos and integration challenges, and need to deal with data interaction and consistency issues between different systems. In addition, it is not easy to achieve real-time monitoring and alarm of the data integration process, resulting in slow integration speed and poor results.

[0003] In order to solve the above defects, a technical solution is now provided. Summary of the invention

[0004] In order to overcome the above-mentioned defects of the prior art, the embodiments of the present invention provide a system for quickly building and arranging data based on a visualization application development method, thereby solving the problems raised in the above-mentioned background technology.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] A system for quickly building and arranging data based on a visual application development method, including a data acquisition module, a data analysis module, a model building module, and a signal generation module, with signal connections between modules;

[0007] Data acquisition module, used to monitor the information generated during the development of visualization applications, data quality information extracted from the database, and extraction efficiency information;

[0008] A data analysis module is used to analyze the data quality information and the extraction efficiency information, obtain the data quality scoring coefficient in the data quality information, and obtain the data conversion efficiency coefficient and the data extraction rate stability coefficient in the extraction efficiency information;

[0009] The model building module is used to build a data analysis model with data quality scoring coefficients, data conversion efficiency coefficients, and data extraction rate instability coefficients, generate performance evaluation coefficients, and judge the performance of building and arranging data within a time interval;

[0010] The signal generation module is used to determine the signal evaluation coefficient of the construction and arrangement data system according to the performance evaluation coefficient in different time intervals and the pre-set performance evaluation coefficient threshold, and generate signals of different levels.

[0011] In a preferred embodiment, analyzing the data quality information includes:

[0012] The data quality information is represented by a data quality scoring coefficient. The logic for obtaining the data quality scoring coefficient is: obtaining the ratio of the number of integrated data records to the number of original data records within a time interval, and marking the ratio of the number of integrated data records to the number of original data records as data integrity. The calculation formula for data integrity is: Among them, SJ jl is the number of data records after integration, SJ ys is the number of original data records;

[0013] For each key field, compare whether the value of the field in the integrated data within the time interval is consistent, and calculate the ratio of the number of records with consistent field values ​​to the total number of records as an indicator of field consistency. The calculation formula for data consistency is: Among them, SJ zjl is the total number of records, SJ yz The number of records with consistent field values;

[0014] Determine the rules or standards for data validation, use programming language or query language to write validation rules, and apply these rules to verify the accuracy of the data. Count the number of records that pass the validation rules within a time interval and calculate the data accuracy. The calculation formula is: Among them, SJ tg The number of records that have passed verification;

[0015] Calculate the data quality score coefficient, the calculation formula is: Among them, ZL pf is the data quality scoring coefficient.

[0016] In a preferred embodiment, extracting efficiency information for analysis includes:

[0017] The extraction efficiency information is represented by the data conversion efficiency coefficient and the data extraction rate stability coefficient. The acquisition logic of the data extraction rate instability coefficient is: obtain the rate of extracting data from the database within the time interval, and mark the rate of extracting data within the time interval as: SL i , where i=1, 2, 3...I, I is a positive integer, and i is the number of the average extraction rate of a unit time period in the time interval;

[0018] Calculate the mean and standard deviation of the rate of extracting data within the time interval, and mark the mean and standard deviation of the rate of extracting data within the time interval as: SL avg and SL bzc ,in,

[0019] Calculate the coefficient of variation of the rate of extracting data within the time interval, the calculation formula is:

[0020] Calculate the data extraction rate stability coefficient, the calculation formula is: WD sl =BY×e BY+1 ; Among them, WD sl is the data extraction rate stability coefficient;

[0021] The acquisition logic of the data conversion efficiency coefficient is: within the time interval, the data size completed from the database at different times is obtained, and the data size completed from the database at different times is marked as: ZH n , where n=1, 2, 3...N, N is a positive integer, n represents the size of data converted at different times within the time interval, and the functional relationship between data conversion and time is constructed, and the functional relationship between data conversion and time is marked as: Z(t);

[0022] Set the conversion threshold, compare the size of data converted from the database at different times with the conversion threshold, and calculate the data conversion efficiency coefficient. The calculation formula is: Among them, XL zh is the data conversion efficiency coefficient, t h ~t g is the time period during which data can be converted from the database normally, t w ~t q The time period during which data conversion from the database cannot be completed normally.

[0023] In a preferred embodiment, the data quality scoring coefficient, the data conversion efficiency coefficient and the data extraction rate instability coefficient are used to construct a data analysis model to generate a performance evaluation coefficient, including:

[0024] The data quality information and extraction efficiency information are comprehensively analyzed, and the data quality scoring coefficient, data conversion efficiency coefficient and data extraction rate instability coefficient are used to construct a data analysis model to generate a performance evaluation coefficient. The calculation formula of the performance evaluation coefficient is: Among them, pg xn is the performance evaluation coefficient, α 1 , α 2 , α 3 is the proportional coefficient of data quality score coefficient, data conversion efficiency coefficient, and data extraction rate instability coefficient, α 1 , α 2 , α 3 Greater than 0.

[0025] In a preferred embodiment, determining the signal evaluation coefficients of the construction and arrangement data system includes:

[0026] Monitor the system that builds and arranges data, generate several performance evaluation coefficients to establish a data analysis set, and mark the data analysis set as: D, where D = {pg m}, m = 1, 2, 3...M, M is a positive integer, m is the number of several performance evaluation coefficients, pg m is the mth performance evaluation coefficient in the data analysis set;

[0027] Set the performance evaluation coefficient threshold and mark the performance evaluation coefficient threshold as: pg yz , compare the performance evaluation coefficient in the data analysis set with the performance evaluation coefficient threshold, and mark the performance evaluation coefficient less than the performance evaluation coefficient threshold as: pg k , where k = 1, 2, 3 ... K, K is a positive integer, and k is the number of the performance evaluation coefficient in the data analysis set that is less than the performance evaluation coefficient threshold;

[0028] Calculate the signal evaluation coefficient of the construction and arrangement data system. The calculation formula of the signal evaluation coefficient is: Where XH is the signal evaluation coefficient for constructing and arranging the data system.

[0029] In a preferred embodiment, different levels of signals are generated, including:

[0030] Compare the signal evaluation coefficient generated in the data analysis set with the set signal evaluation coefficient threshold Sum signal evaluation coefficient threshold In contrast, Less than Generates the following situation:

[0031] If XH is greater than An interrupt signal is generated;

[0032] If XH is greater than And XH is less than An early warning signal is generated, and the construction and orchestration data system can continue to work. Professional staff need to check the construction and orchestration data system;

[0033] If XH is less than A working signal is generated to continue working.

[0034] Technical effects and advantages of the present invention:

[0035] The present invention obtains data quality information and extraction efficiency information in different time periods by monitoring the process of constructing and arranging data, comprehensively analyzes the data quality information and extraction efficiency information, determines the performance of the system for constructing and arranging data in different time periods, and continuously monitors the system for constructing and arranging data over a long period of time to determine the possibility of hidden dangers in the system for constructing and arranging data, and generates different signals, which helps to optimize system performance and ensure data quality, ensure system stability and reliability, and promptly notify relevant personnel to handle problems when they occur, thereby minimizing the impact of system failures on business. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to facilitate understanding by those skilled in the art, the present invention is further described below in conjunction with the accompanying drawings;

[0037] Figure 1 The schematic diagram of the structure of the system for rapidly constructing and arranging data based on the visualization application development method of the present invention. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] Example 1

[0040] like Figure 1 A schematic diagram of the structure of a system for rapidly building and arranging data based on a visualization application development method of the present invention is given, which includes a data acquisition module, a data analysis module, a model building module, and a signal generation module;

[0041] Data acquisition module, used to monitor the information generated during the development of visualization applications, data quality information extracted from the database, and extraction efficiency information;

[0042] A data analysis module is used to analyze the data quality information and the extraction efficiency information, obtain the data quality scoring coefficient in the data quality information, and obtain the data conversion efficiency coefficient and the data extraction rate stability coefficient in the extraction efficiency information;

[0043] The model building module is used to build a data analysis model with data quality scoring coefficients, data conversion efficiency coefficients, and data extraction rate instability coefficients, generate performance evaluation coefficients, and judge the performance of building and arranging data within a time interval;

[0044] The signal generation module is used to determine the signal evaluation coefficient of the construction and arrangement data system according to the performance evaluation coefficient in different time intervals and the pre-set performance evaluation coefficient threshold, and generate signals of different levels.

[0045] Visualization-based application development methods are of great significance in the field of universities, helping universities to quickly build and organize data systems, improve the efficiency and quality of data processing and analysis, and thus better support the teaching, management and scientific research work of universities;

[0046] In the field of universities, the use of visualization-based application development methods to quickly build and orchestrate data systems is important in the following ways:

[0047] Data presentation and analysis needs: Colleges and universities involve a wide variety of data, including student information, course data, scientific research results, etc., and these data usually need to be processed, presented and analyzed to help education administrators, teachers and students make decisions. Visualization-based applications can display data more intuitively and provide interactive data analysis functions to help users better understand data and discover information.

[0048] Multi-source data integration needs: College data usually comes from multiple different systems and platforms, such as student management systems, financial systems, scientific research management systems, etc. These data sources may have differences in format, structure, standards, etc. Visual application development methods can help achieve rapid integration and arrangement of these data sources, allowing users to access and analyze data from different data sources on the same platform.

[0049] Quickly respond to changing needs: The needs in a university environment usually change frequently, including teaching needs, management needs, scientific research needs, etc. The visualization-based application development method is flexible and has the characteristics of rapid iteration. It can respond to changes in user needs more quickly and provide universities with more adaptable data processing and analysis solutions.

[0050] For data involving colleges and universities, select appropriate index types according to the query type and data characteristics, such as single-column index, combined index, full-text index, etc., and index key fields that are frequently used for queries, such as primary keys, foreign keys, and fields that frequently appear in the WHERE clause or connection conditions.

[0051] For the indexed data, ensure the accuracy and availability of the data through data conversion and cleaning. Data cleaning includes missing value processing, duplicate value processing, and outlier processing, and data conversion includes feature extraction, feature scaling, and feature encoding.

[0052] Missing value processing uses statistical methods or visualization tools to identify missing values ​​in the data, and selects appropriate missing value processing strategies based on the characteristics of the data and the needs of analysis, such as deleting missing values, filling missing values, etc.

[0053] Duplicate value processing compares data rows to identify duplicate values ​​in the data. Depending on the needs of the analysis, you can choose to keep one duplicate value or delete all duplicate values.

[0054] Outlier processing uses statistical methods, visualization tools or domain knowledge to identify outliers in the data and select appropriate processing methods based on the causes and impacts of the outliers, such as correcting or deleting outliers.

[0055] Feature extraction is to extract or create new features from raw data. Common feature extraction methods include bag-of-words model of text, TF-IDF vectorization, color histogram of image, edge detection, etc.

[0056] Feature scaling is to scale data features to ensure that they have similar scales and avoid certain features from having too much influence on the model. Common feature scaling methods include standardization (Z-score standardization) and normalization (Min-Max standardization).

[0057] Feature encoding is the process of converting non-numeric data into numerical data. Common feature encoding methods include One-Hot Encoding and Label Encoding.

[0058] Data quality information includes data integrity, data consistency, and data accuracy. If data integrity, data consistency, and data accuracy are not guaranteed in visualization-based application development, the following problems may occur:

[0059] Misleading data presentation: Incomplete, inconsistent, or inaccurate data may lead to misleading data presentation and analysis results. Users may make wrong decisions based on incorrect data, which may have a negative impact on the business.

[0060] Decreased quality of student services: The school’s student services may be affected, such as course registration, grade inquiry, financial settlement, etc. If the data is inaccurate or incomplete, students may not be able to obtain the required services in a timely manner, affecting their learning experience and satisfaction.

[0061] School management chaos: Inaccurate student information, course schedules or financial data may lead to school management chaos. For example, incorrect student course selection information may lead to chaotic course scheduling, waste of classroom resources, or affect students' academic progress.

[0062] The data quality information is represented by a data quality scoring coefficient. The logic for obtaining the data quality scoring coefficient is: obtaining the ratio of the number of integrated data records to the number of original data records within a time interval, and marking the ratio of the number of integrated data records to the number of original data records as data integrity. The calculation formula for data integrity is: Among them, SJ jl is the number of data records after integration, SJ ys is the number of original data records;

[0063] For each key field, compare whether the value of the field in the integrated data within the time interval is consistent, and calculate the ratio of the number of records with consistent field values ​​to the total number of records as an indicator of field consistency. The calculation formula for data consistency is: Among them, SJ zjl is the total number of records, SJ yz The number of records with consistent field values;

[0064] Determine the rules or standards for data validation, including rules on data types, ranges, formats, etc., such as date formats, numerical ranges, etc. Use programming languages ​​or query languages ​​to write validation rules and apply these rules to verify the accuracy of the data. Count the number of records that pass the validation rules within a time interval and calculate the data accuracy. The calculation formula is: Among them, SJ tg The number of records that have passed verification;

[0065] Calculate the data quality score coefficient, the calculation formula is: Among them, ZL pf is the data quality scoring coefficient.

[0066] It can be seen from the formula that the larger the data quality scoring coefficient is, the higher the integrity, consistency and accuracy of the data is, and the higher the quality of the data is. Conversely, the smaller the data quality scoring coefficient is, the lower the integrity, consistency and accuracy of the data is, and the lower the quality of the data is.

[0067] The extraction efficiency information is represented by the data conversion efficiency coefficient and the data extraction rate stability coefficient. If you do not pay attention to the stability of the data conversion efficiency and the data extraction rate, the following problems may occur:

[0068] Delayed decision-making: University administrators may not be able to obtain the required data analysis results in a timely manner, thus delaying the decision-making process. For example, if the data extraction rate is unstable during the course selection period, the school may not be able to understand the student course selection situation in a timely manner, and thus cannot make corresponding teaching resource allocation decisions;

[0069] Impact on teaching and student services: If the data conversion efficiency is low or the data extraction rate is unstable, it may affect the quality of teaching and student services. For example, teachers may not be able to obtain students' homework submission status in a timely manner, and students may not be able to obtain course grades and other information in a timely manner;

[0070] Increased workload and costs: If data conversion is inefficient, more manpower and resources may be required to manually process or fix data quality issues. This not only increases the workload, but may also increase related costs;

[0071] Impact on data quality: If the data extraction rate is unstable, it may lead to data loss or errors. This may affect the accuracy and completeness of the data, thereby reducing the data quality and affecting the subsequent data analysis and decision-making process.

[0072] The logic for obtaining the data extraction rate instability coefficient is as follows: obtaining the rate of extracting data from the database within a time interval, and marking the rate of extracting data within the time interval as: SL i , where i=1, 2, 3...I, I is a positive integer, and i is the number of the average extraction rate of a unit time period in the time interval;

[0073] It should be noted that the rate of extracting data from the database is not only limited by the network bandwidth, but also related to the system's ability to process data, including processor performance, disk I / O speed, and the algorithm for extracting data.

[0074] Calculate the mean and standard deviation of the rate of extracting data within the time interval, and mark the mean and standard deviation of the rate of extracting data within the time interval as: SL avg and SL bzc ,in,

[0075] Calculate the coefficient of variation of the rate of extracting data within the time interval, the calculation formula is:

[0076] It should be noted that the smaller the coefficient of variation of the rate of extracting data within a time interval, the smaller the fluctuation of the data extraction rate, and the rate is relatively stable; conversely, the larger the coefficient of variation, the greater the fluctuation of the extraction rate, and the rate is not stable enough.

[0077] Calculate the data extraction rate stability coefficient, the calculation formula is: WD sl =BY×e BY+1 ; Among them, WD sl is the data extraction rate stability coefficient.

[0078] It can be seen from the formula that the larger the data extraction rate instability coefficient is, the larger the variation range of the data extraction rate is within the time interval, indicating that there may be problems with the algorithm or processing logic used in the data extraction process, resulting in fluctuations in the data extraction rate.

[0079] The acquisition logic of the data conversion efficiency coefficient is: within the time interval, the data size completed from the database at different times is obtained, and the data size completed from the database at different times is marked as: ZH n , where n=1, 2, 3...N, N is a positive integer, n represents the size of data converted at different times within the time interval, and the functional relationship between data conversion and time is constructed, and the functional relationship between data conversion and time is marked as: Z(t);

[0080] It should be noted that the functional relationship between data conversion and time is completed by fitting the data points, and a suitable mathematical model or function form, such as linear function, exponential function, polynomial function, etc., is selected to fit the data points.

[0081] Set a conversion threshold, and compare the size of data converted from the database at different times with the conversion threshold. If the size of data converted from the database at different times is less than the conversion threshold, it means that the data conversion is abnormal. If the size of data converted from the database at different times is greater than the conversion threshold, it means that the data conversion is normal.

[0082] Calculate the data conversion efficiency coefficient, the calculation formula is: Among them, XL zh is the data conversion efficiency coefficient, t h ~t g is the time period during which data can be converted from the database normally, t w ~t q The time period during which data conversion from the database cannot be completed normally.

[0083] It can be seen from the formula that the larger the data conversion efficiency coefficient is, the more data is converted within the time interval and the better the data conversion performance is. Conversely, the smaller the data conversion efficiency coefficient is, the less data is converted within the time interval and the worse the data conversion performance is.

[0084] The data quality information and extraction efficiency information are comprehensively analyzed, and the data quality scoring coefficient, data conversion efficiency coefficient and data extraction rate instability coefficient are used to construct a data analysis model to generate a performance evaluation coefficient. The calculation formula of the performance evaluation coefficient is: Among them, pg xn is the performance evaluation coefficient, α 1 , α 2 , α 3is the proportional coefficient of data quality score coefficient, data conversion efficiency coefficient, and data extraction rate instability coefficient, α 1 , α 2 , α 3 Greater than 0.

[0085] It can be seen from the formula that if the data conversion efficiency coefficient is smaller and the data quality scoring coefficient and the data extraction rate instability coefficient are larger, the performance evaluation coefficient is larger, which means that the effect of data construction and arrangement in the university database is better. Conversely, if the data conversion efficiency coefficient is larger and the data quality scoring coefficient and the data extraction rate instability coefficient are smaller, the performance evaluation coefficient is smaller, which means that the effect of data construction and arrangement in the university database is worse.

[0086] It should be noted that the time interval is a relatively short period of time during which the required information of the system is collected. Usually, the time interval is set by staff in professional fields.

[0087] Monitor the system that builds and arranges data, generate several performance evaluation coefficients to establish a data analysis set, and mark the data analysis set as: D, where D = {pg m}, m = 1, 2, 3...M, M is a positive integer, m is the number of several performance evaluation coefficients, pg m is the mth performance evaluation coefficient in the data analysis set;

[0088] Set the performance evaluation coefficient threshold and mark the performance evaluation coefficient threshold as: pg yz , compare the performance evaluation coefficient in the data analysis set with the performance evaluation coefficient threshold, and mark the performance evaluation coefficient less than the performance evaluation coefficient threshold as: pg k , where k = 1, 2, 3 ... K, K is a positive integer, and k is the number of the performance evaluation coefficient in the data analysis set that is less than the performance evaluation coefficient threshold;

[0089] Calculate the signal evaluation coefficient of the construction and arrangement data system. The calculation formula of the signal evaluation coefficient is: Among them, XH is the signal evaluation coefficient for building and arranging data systems;

[0090] It can be seen from the formula that the larger the signal evaluation coefficient generated in the data analysis set, the higher the hidden danger of building and arranging the data system, and vice versa, the lower the hidden danger of building and arranging the data system;

[0091] Compare the signal evaluation coefficient generated in the data analysis set with the set signal evaluation coefficient threshold Sum signal evaluation coefficient threshold In contrast, Less than Generates the following situation:

[0092] If XH is greater than An interrupt signal is generated;

[0093] If XH is greater than And XH is less than An early warning signal is generated, and the construction and orchestration data system can continue to work. Professional staff need to check the construction and orchestration data system;

[0094] If XH is less than A working signal is generated to continue working.

[0095] This embodiment monitors the process of constructing and orchestrating data to obtain data quality information and extraction efficiency information in different time periods, comprehensively analyzes the data quality information and extraction efficiency information, determines the performance of the system for constructing and orchestrating data in different time periods, and continuously monitors the system for constructing and orchestrating data over a long period of time to determine the possibility of hidden dangers in the system for constructing and orchestrating data, and generates different signals, which helps to optimize system performance and ensure data quality, ensure system stability and reliability, and promptly notify relevant personnel to handle problems when they occur, thereby minimizing the impact of system failures on business.

[0096] The above formulas are all dimensionless and numerical calculations. The formula is a formula for the most recent real situation obtained by collecting a large amount of data and performing software simulation. The preset parameters in the formula are set by technicians in this field according to actual conditions.

[0097] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0098] It should be understood that in the various embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0099] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0100] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0101] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0102] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.

[0103] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A system for quickly building and arranging data based on a visual application development method, characterized by: It includes a data acquisition module, a data analysis module, a model building module and a signal generation module, and the signals between the modules are connected; Data acquisition module, used to monitor the information generated during the development of visualization applications, data quality information extracted from the database, and extraction efficiency information; A data analysis module is used to analyze the data quality information and the extraction efficiency information, obtain the data quality scoring coefficient in the data quality information, and obtain the data conversion efficiency coefficient and the data extraction rate stability coefficient in the extraction efficiency information; The model building module is used to construct a data analysis model using data quality scoring coefficients, data conversion efficiency coefficients, and data extraction rate instability coefficients, generate performance evaluation coefficients, and determine the performance of building and arranging data within a time interval; The signal generation module is used to determine the signal evaluation coefficient of the construction and arrangement data system according to the performance evaluation coefficient in different time intervals and the pre-set performance evaluation coefficient threshold, and generate signals of different levels.

2. The system for rapidly building and arranging data based on the visualization-based application development method according to claim 1, characterized in that: Analyze data quality information, including: The data quality information is represented by a data quality scoring coefficient. The logic for obtaining the data quality scoring coefficient is: obtaining the ratio of the number of integrated data records to the number of original data records within a time interval, and marking the ratio of the number of integrated data records to the number of original data records as data integrity. The calculation formula for data integrity is: Among them, SJ jl is the number of data records after integration, SJ ys is the number of original data records; For each key field, compare whether the value of the field in the integrated data within the time interval is consistent, and calculate the ratio of the number of records with consistent field values ​​to the total number of records as an indicator of field consistency. The calculation formula for data consistency is: Among them, SJ zjl is the total number of records, SJ yz The number of records with consistent field values; Determine the rules or standards for data validation, use programming language or query language to write validation rules, and apply these rules to verify the accuracy of the data. Count the number of records that pass the validation rules within a time interval and calculate the data accuracy. The calculation formula is: Among them, SJ tg The number of records that have passed verification; Calculate the data quality score coefficient, the calculation formula is: Among them, ZL pf is the data quality scoring coefficient.

3. The system for rapidly building and arranging data based on the visualization-based application development method according to claim 1, characterized in that: Extract efficiency information for analysis, including: The extraction efficiency information is represented by the data conversion efficiency coefficient and the data extraction rate stability coefficient. The acquisition logic of the data extraction rate instability coefficient is: obtain the rate of extracting data from the database within the time interval, and mark the rate of extracting data within the time interval as: SL i , where i=1, 2, 3...I, I is a positive integer, and i is the number of the average extraction rate of a unit time period in the time interval; Calculate the mean and standard deviation of the rate of extracting data within the time interval, and mark the mean and standard deviation of the rate of extracting data within the time interval as: SL avg and SL bzc ,in, Calculate the coefficient of variation of the rate of extracting data within the time interval, the calculation formula is: Calculate the data extraction rate stability coefficient, the calculation formula is: WD sl =BY×e BY+1 ; Among them, WD sl is the data extraction rate stability coefficient; The acquisition logic of the data conversion efficiency coefficient is: within the time interval, the data size completed from the database at different times is obtained, and the data size completed from the database at different times is marked as: ZH n , where n=1, 2, 3...N, N is a positive integer, n represents the size of data converted at different times within the time interval, and the functional relationship between data conversion and time is constructed, and the functional relationship between data conversion and time is marked as: Z(t); Set the conversion threshold, compare the size of data converted from the database at different times with the conversion threshold, and calculate the data conversion efficiency coefficient. The calculation formula is: Among them, XL zh is the data conversion efficiency coefficient, t h ~t g is the time period during which data can be converted from the database normally, t w ~t q The time period during which data conversion from the database cannot be completed normally.

4. The system for rapidly building and arranging data based on the visualization-based application development method according to claim 1, characterized in that: The data quality scoring coefficient, data conversion efficiency coefficient and data extraction rate instability coefficient are used to construct a data analysis model to generate performance evaluation coefficients, including: The data quality information and extraction efficiency information are comprehensively analyzed, and the data quality scoring coefficient, data conversion efficiency coefficient and data extraction rate instability coefficient are used to construct a data analysis model to generate a performance evaluation coefficient. The calculation formula of the performance evaluation coefficient is: Among them, pg xn is the performance evaluation coefficient, α1, α2, and α3 are the proportional coefficients of the data quality score coefficient, the data conversion efficiency coefficient, and the data extraction rate instability coefficient, and α1, α2, and α3 are greater than 0.

5. The system for rapidly building and arranging data based on the visualization-based application development method according to claim 4 is characterized in that: Determine signal evaluation factors for building and orchestrating data systems, including: Monitor the system that builds and arranges data, generate several performance evaluation coefficients to establish a data analysis set, and mark the data analysis set as: D, where D = {pg m }, m = 1, 2, 3...M, M is a positive integer, m is the number of several performance evaluation coefficients, pg m is the mth performance evaluation coefficient in the data analysis set; Set the performance evaluation coefficient threshold and mark the performance evaluation coefficient threshold as: pg yz , compare the performance evaluation coefficient in the data analysis set with the performance evaluation coefficient threshold, and mark the performance evaluation coefficient less than the performance evaluation coefficient threshold as: pg k , where k = 1, 2, 3 ... K, K is a positive integer, and k is the number of the performance evaluation coefficient in the data analysis set that is less than the performance evaluation coefficient threshold; Calculate the signal evaluation coefficient of the construction and arrangement data system. The calculation formula of the signal evaluation coefficient is: Where XH is the signal evaluation coefficient for constructing and arranging the data system.

6. The system for rapidly building and arranging data based on the visualization-based application development method according to claim 5, characterized in that: Generates signals at different levels, including: Compare the signal evaluation coefficient generated in the data analysis set with the set signal evaluation coefficient threshold Sum signal evaluation coefficient threshold In contrast, Less than Generates the following situation: If XH is greater than An interrupt signal is generated; If XH is greater than And XH is less than An early warning signal is generated, and the construction and orchestration data system can continue to work. Professional staff need to check the construction and orchestration data system; If XH is less than A working signal is generated to continue working.

Images