A lightweight real-time data system

Abstract

The present invention provides a lightweight real-time data system, which relates to the technical field of data processing. A system management module, a data governance module, a data acquisition module, a data service module and a data asset module constitute the entire system structure. A data quality assessment model ensures the accuracy, integrity and consistency of data. An anomaly detection model can timely discover abnormal values. The data acquisition module efficiently manages data sources. The data service module provides safe and reliable data access. The data asset module reasonably allocates resources, provides a basis for decision-making and regularly updates and maintains. A lightweight design is adopted, and general data acquisition, calculation and processing, data management and data service functions are integrated to reduce repetitive related code writing, realize flexible configuration and rapid deployment, and innovate business application construction modes. Lightweight processing enables the data system to adapt to global tourism management platforms and scenic area management platforms of different scales and types.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a lightweight real-time data system. Background Art

[0002] At present, in the field of smart cultural and tourism management software, it is divided into two types: global tourism management platform and scenic spot management platform. The core of both is data access, processing and display. The global tourism management platform mainly needs to access the operating data of cultural and tourism units such as scenic spots and cultural and museum venues within the jurisdiction, and access various basic data of the cultural and tourism industry by filling in and reporting, and access data from government public security, transportation, statistics, market supervision and other departments, as well as external Internet, operators, financial payment and other data. The scenic spot management platform mainly needs to access various intelligent system data such as ticketing, security, parking, passenger flow, environmental monitoring, etc. built in the scenic area, as well as external Internet, operators and other data. According to the traditional development model, the back-end development of the platform is centered on the database, and corresponding data tables are established for each type of data. First, it is necessary to develop access collection programs according to the API of the docking system to realize data storage, and then calculate and process the data, and then output the API in the form of web services for front-end program calls, especially using Java specification layered development. Each type of data needs to write a series of codes such as DAO, DTO, Service, Implement, Controller, etc., which involves a lot of repetitive work, large workload, high coupling, and easy errors.

[0003] According to a tourism evaluation and analysis system based on a scenic spot tourism platform disclosed in China with a publication number of "CN115204678A", the tourism evaluation and analysis system based on the scenic spot tourism platform includes a scenic spot project passenger flow monitoring module, a scenic spot project passenger flow analysis and guidance module, a scenic spot project heat analysis module, a scenic spot project safety analysis module, a scenic spot comprehensive evaluation display module, a scenic spot safety early warning terminal and a database; the present invention obtains the game behavior safety index corresponding to each tourist in each play project in the target scenic spot, the maintenance evaluation coefficient of each play project and the service life evaluation coefficient, and comprehensively analyzes to obtain the equipment safety index corresponding to each play project in the target scenic spot, analyzes the equipment safety of each play project in the target scenic spot through multiple dimensions, improves the representativeness and reference of the equipment safety detection results, avoids accidents in the scenic spot, and further provides data support for the subsequent maintenance of each play project in the target scenic spot.

[0004] The above patent documents and prior art have the following technical problems when used:

[0005] Problem 1: In the construction of the global tourism management platform and the scenic area management platform, the existing data management systems are often customized for specific projects. The data access, processing and display methods between different projects vary greatly, resulting in a large amount of repetitive code writing work, which not only wastes development resources, but also makes system maintenance and upgrades difficult. The system is not highly integrated when processing data and is not adaptable enough to different scenarios.

[0006] Problem 2: Due to the differences in the scale and region of scenic spots, the data sources are extensive and complex, the data quality of different data sources is uneven, and the data cannot be collected in combination with the characteristics of the data source, resulting in untimely data collection and incomplete subsequent analysis;

[0007] Problem three: There is a lack of effective quality control mechanisms in data processing. Single processing of data accuracy cannot meet the needs, resulting in single data evaluation and a lack of comprehensive evaluation of data quality and anomaly detection mechanisms. Summary of the invention

[0008] Technical issues solved:

[0009] In view of the shortcomings of the prior art, the present invention provides a lightweight real-time data system to solve the following problems:

[0010] 1. The system has complicated steps for data processing, is difficult to operate, has insufficient integration, and cannot adapt to data processing in multiple scenarios;

[0011] 2. Lack of comprehensive evaluation and anomaly detection mechanisms for data collection quality. The problem of not being able to collect data in accordance with the characteristics of the data source during data collection;

[0012] 3. The data evaluation is single and lacks comprehensive evaluation of data quality and anomaly detection mechanism.

[0013] Technical solution:

[0014] To achieve the above objectives, the present invention is implemented through the following technical solutions: a lightweight real-time data system, the system includes a system management module, a data governance module, a data acquisition module, a data service module and a data asset module, wherein:

[0015] The data governance module is used to configure basic data governance operations, including data modeling, data query and data cleaning. According to business needs and data sources, the entities, attributes and relationships of the data are determined, and a data model is established. During data collection and storage, data is organized and managed according to the data model. Users input query conditions, and the system quickly retrieves qualified data according to the data model and index, processes the collected data with a data cleaning algorithm, trains and learns the data using a machine learning algorithm, establishes a data quality assessment model and anomaly detection model, and monitors data quality in real time during data processing, and promptly issues an alarm when anomalies are found;

[0016] The data collection module is responsible for acquiring data from various data sources and managing the collection process, including data source management, template management, automatic collection and manual collection. It registers and manages various data sources, determines the data collection method and frequency, and designs corresponding data collection templates according to different data sources and data types. When collecting data, users select corresponding templates to reduce configuration workload. Intelligent scheduling algorithms are used to set the execution time and sequence of tasks according to the characteristics of data sources and collection requirements. Data capture technology is used to quickly capture data from data sources, and preliminary processing and verification are performed. When automatic collection cannot meet the needs, users collect data by manually inputting or importing data.

[0017] The data service module provides functions related to data services, including API management, operation and maintenance monitoring and data statistics, designs and develops APIs for data services, provides external data access interfaces, performs identity authentication and permission control during API calls, uses real-time monitoring algorithms to monitor server performance, network traffic, data access volume, etc. in real time, and promptly issues alarms and takes corresponding measures to handle abnormal situations, performs statistical analysis on the use of data services, generates statistical reports, understands user needs and behaviors through statistical data analysis, provides a basis for optimizing services, and uses real-time monitoring algorithms and data analysis technologies to ensure service quality and data accuracy;

[0018] The data asset module manages data assets, including the creation and maintenance of asset catalogs, the processing of assets to be allocated, and the statistical summary of assets. According to the characteristics and classification standards of data assets, a data asset catalog is created, data assets are classified and described in the asset catalog, and the asset catalog is regularly maintained and updated. Assets that have not yet been allocated are registered and reviewed, the value and use of assets are determined, and allocation and management are carried out according to the characteristics and needs of assets. Data asset evaluation algorithms and asset classification technologies are used to statistically summarize data assets and generate asset statistical reports, including asset quantity, value, usage, etc., to provide a basis for asset management and decision-making;

[0019] The system management module configures and manages the global information of the system, including organizational structure, user information, role permissions and application configuration. It assigns corresponding permissions to users based on their roles and responsibilities, verifies their permissions when they log in to the system, and ensures that they can only access and operate authorized functions and data. It continuously optimizes the organizational structure based on system usage and business needs to ensure the normal operation of the system and the collaborative work between its parts.

[0020] Preferably, the data governance module uses a machine learning algorithm to establish a data quality assessment model for evaluating the accuracy, completeness, and consistency of the data. data samples, each with Features, using represents a data set, where Indicates The feature vector of samples defines the quality assessment function:

[0021] ;

[0022] in:

[0023] Represents the quality assessment result of the entire data set, which is obtained by summing the quality scores of each sample;

[0024] Represents a single sample The quality score is calculated by comprehensively calculating one or more indicators of accuracy, completeness and consistency. The calculation of accuracy indicator is as follows;

[0025] The accuracy index is measured by calculating the error between the sample and the true value, and the accuracy index is expressed as mean square error:

[0026] ;

[0027] in:

[0028] Indicates The true value of samples;

[0029] Indicates The predicted value of a sample, that is, the result obtained through model prediction, that is, the result obtained by predicting the sample through the established prediction model. The prediction model can be selected according to specific business needs and data characteristics. The model analyzes and predicts various characteristics of the sample to obtain the predicted value. With the training and optimization of the model, the predicted value will continue to approach the true value, thereby improving the accuracy of the data;

[0030] Represents the mean square error, which is used to measure the difference between the predicted value and the true value. It represents the square of the difference between the predicted value and the true value. The sum of all samples is divided by the total number of samples to get the average square error value.

[0031] when The smaller the time, the higher the accuracy, and the quality score increases accordingly. The completeness is measured by the proportion of missing values ​​in the statistical sample, and the consistency is measured by comparing whether the values ​​of the same feature between different samples are consistent.

[0032] Preferably, an anomaly detection model is established in the data governance module to detect outliers in the data. data samples, using Represents a data set, each sample has features, and use a distance-based anomaly detection method to calculate the Euclidean distance between each sample and other samples:

[0033] ;

[0034] in, Indicates the data set Samples and The Euclidean distance between samples is Indicates The sample eigenvalues, Indicates The sample eigenvalues;

[0035] Define anomaly score function to represent samples The total distance to other samples is:

[0036] ;

[0037] Set a threshold ,when , it is considered that the sample is an outlier, when , it is considered that the sample is the normal value.

[0038] Preferably, the intelligent scheduling algorithm in the data acquisition module arranges the execution time and order of the acquisition tasks according to the characteristics of the data source and the acquisition requirements. Suppose the data source set is , each data source has different collection frequency requirements and data size ;

[0039] Define the task priority function:

[0040] ;

[0041] in:

[0042] and is the weight coefficient, which is adjusted according to the actual situation.

[0043] Indicates The collection frequency requirements of each data source;

[0044] Indicates The amount of data in each data source is measured in bytes and records.

[0045] Indicates The task priority of each data source;

[0046] Indicates The reciprocal of the collection frequency of a data source. The higher the collection frequency, the smaller this value is, and the greater the contribution to the task priority is.

[0047] Indicates The ratio of the data volume of a data source to the total data volume of all data sources. The larger the data volume, the higher this ratio is, and the greater the contribution to the task priority is.

[0048] The higher the collection frequency and the larger the data volume, the higher the priority of the data source task.

[0049] Preferably, the real-time monitoring algorithm in the data service module is used to monitor server performance, network traffic, data access volume, etc. in real time, and detect abnormal situations in time. The server performance indicator set is set as , It is the internal performance index of the server, including CPU utilization, memory utilization, network traffic Indicated by data access volume Use to indicate;

[0050] Define the anomaly detection function:

[0051] ;

[0052] in, Indicates the indicator The anomaly detection function sets a normal range for CPU utilization ,when When the CPU utilization is abnormal, ,when When the CPU utilization is normal, , Indicates the anomaly detection parameters for network traffic. Expressed as the amount of data transmitted per unit time, such as bytes / second, set a normal range. When it is not within this range, the network traffic is considered abnormal, and e(F)=0. When the network traffic F is within this range, the network traffic is considered normal, and e(F)=1. Represents an anomaly detection function for data access volume. It is expressed as the number of data accesses per unit time, setting a normal range. When it is not within this range, the data access volume is considered abnormal, and e(A)=0; when the data access volume When it is within this range, the data access volume is considered normal, and e(A)=1.

[0053] Preferably, the data asset module evaluates the data assets through a data asset evaluation algorithm to determine the value of the assets. Suppose the data asset set is ,Each data asset has multiple evaluation indicators, including data accuracy, completeness, timeliness, etc.;

[0054] Define the asset value function:

[0055] ;

[0056] in:

[0057] Indicates The value score of each data asset;

[0058] It is The weight coefficient of each evaluation indicator is used to adjust the importance of different evaluation indicators in the calculation of asset value;

[0059] It is a data asset In the The scores on the evaluation indicators are calculated by evaluating the accuracy, completeness and timeliness of the data.

[0060] For the data accuracy indicator, the score is calculated based on the error between the data and the true value, assuming the true value is , the predicted value is , the accuracy score formula is as follows:

[0061] ;

[0062] in, represents the maximum mean square error of all data assets on the accuracy metric, ;

[0063] For the completeness indicator, the score is calculated by calculating the proportion of missing values ​​in the data asset. The total number of records is , where the number of records with missing values ​​is , then the completeness score can be defined as:

[0064] ;

[0065] For the timeliness index, the score is calculated based on the difference between the data update time and the current time. Suppose the data update time is , the current time is , set a timeliness threshold ,then , think that the data is timely and has a higher score; when , the data is considered to be of poor timeliness and the score is low.

[0066] Preferably, the asset classification technology in the data asset module classifies data assets, manages and uses them, and uses a clustering algorithm to classify data assets. Suppose the data asset set is , each data asset is represented by a feature vector, , the K-Means algorithm is used. The goal is to divide the data assets into clusters so that the sum of the distances from each data asset to the center of its cluster is minimized. The objective function formula is defined as follows:

[0067] ;

[0068] in:

[0069] It represents the objective function of clustering, which is used to measure the quality of clustering. The goal is to minimize the value of this function.

[0070] Representing data assets Is it a cluster? , if it belongs to ;otherwise, ;

[0071] It is clustering The center of is a vector composed of data asset feature vectors;

[0072] Indicates Data assets and The square of the Euclidean distance between the cluster centers is used to measure the distance between the data asset and the cluster center.

[0073] Preferably, when creating a data asset catalog in the data asset module, techniques such as classification and coding are used to classify and encode data assets, an asset catalog update mechanism is established, and the asset catalog is updated and maintained regularly.

[0074] Preferably, the data acquisition module provides data verification and validation functions in the method of manual input or importing of data to ensure that the manually collected data meets the data quality requirements, establishes a data review mechanism, reviews and approves the manually collected data, and ensures the accuracy and reliability of the data.

[0075] Beneficial effects:

[0076] The present invention provides a lightweight real-time data system. It has the following beneficial effects:

[0077] 1. The present invention adopts a system management module, a data governance module, a data acquisition module, a data service module and a data asset module to form the entire system structure, so that the data quality assessment model established by the system using the machine learning algorithm ensures the accuracy, integrity and consistency of the data, the anomaly detection model can timely discover outliers and improve data quality, the data acquisition module is used to efficiently manage the data source, reduce the configuration workload, intelligently schedule the acquisition task, and when the automatic acquisition cannot meet the needs, manual acquisition and ensure data quality can be performed, the data service module is used to provide safe and reliable data access, and the service quality is guaranteed in real time. It can also provide a basis for optimizing services through statistical analysis, and the data asset module is used to effectively manage data assets, reasonably allocate resources, provide decision-making basis and regularly update and maintain. The system management module ensures system security and optimizes the organizational structure to ensure the normal operation and collaborative work of the system. The entire system adopts a lightweight design, integrates general data acquisition, computing and processing, data management and data service functions, reduces repetitive related code writing, realizes flexible configuration and rapid deployment, and innovates the business application construction model. Lightweight processing enables the data system to adapt to global tourism management platforms and scenic area management platforms of different scales and types. Whether it is a small scenic spot or a large tourist area, it can be flexibly configured and deployed according to actual needs.

[0078] 2. The present invention adopts data source management technology in the system data acquisition module. By registering and managing various data sources and determining the acquisition method and frequency, it realizes the effect of effective control over different types of data sources. It uses intelligent scheduling algorithms to arrange the execution time and sequence of acquisition tasks according to the characteristics of the data source and the acquisition requirements, ensuring that data source tasks with high acquisition frequency and large data volume have higher priority, improving the timeliness and comprehensiveness of data acquisition, and realizing the effect of efficiently and accurately acquiring data from various data sources. It adopts lightweight algorithms and technologies with low cost and faster data processing speed. The system can respond to various requests more quickly, whether it is data acquisition, governance or service modules, and can complete the task in a shorter time, reducing user waiting time.

[0079] 3. The present invention adopts data modeling technology in the system data governance module. By determining data entities, attributes and relationships and establishing data models, it achieves the effect of organizing and managing data in an orderly manner during data collection and storage. It uses machine learning algorithms to establish data quality assessment models. By evaluating the accuracy, completeness and consistency of data, it achieves the effect of comprehensively grasping the data quality status, timely discovering data quality problems, and providing a reliable basis for further data processing and analysis. It establishes anomaly detection models and adopts distance-based anomaly detection methods. By real-time monitoring of data quality and timely alarming when abnormal situations are found, it achieves the effect of quickly identifying outliers in the data, which helps to take timely measures to avoid abnormal data from having adverse effects on subsequent analysis and decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] Figure 1 is a system structure diagram of the present invention;

[0081] Figure 2 It is a cloud diagram of the internal data connection of the system of the present invention;

[0082] Figure 3 It is the asset processing flow chart of the present invention;

[0083] Figure 4 A data source priority visualization chart of the present invention;

[0084] Figure 5 This is a visualization chart of CPU anomaly detection in the data service module of the present invention. DETAILED DESCRIPTION

[0085] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. Specific embodiment one:

[0087] like Figure 1-5 As shown, a lightweight real-time data system includes a system management module, a data governance module, a data acquisition module, a data service module and a data asset module, wherein:

[0088] The data collection module is responsible for acquiring data from various data sources and managing the collection process, including data source management, template management, automatic collection and manual collection. It registers and manages various data sources, determines the data collection method and frequency, and designs corresponding data collection templates according to different data sources and data types. When collecting data, users select the corresponding template to reduce the configuration workload. The intelligent scheduling algorithm is used to set the execution time and sequence of tasks according to the characteristics of the data source and the collection requirements. The data capture technology is used to quickly capture data from the data source and perform preliminary processing and verification. When automatic collection cannot meet the needs, users collect data by manually inputting or importing data. The intelligent scheduling algorithm in the data collection module arranges the execution time and sequence of collection tasks according to the characteristics of the data source and the collection requirements. Suppose the data source set is , each data source has different collection frequency requirements and data size ;

[0089] Define the task priority function:

[0090] ;

[0091] in:

[0092] and is the weight coefficient, which is adjusted according to the actual situation.

[0093] Indicates The collection frequency requirements of each data source;

[0094] Indicates The amount of data in each data source is measured in bytes and records.

[0095] Indicates The task priority of each data source;

[0096] Indicates The reciprocal of the collection frequency of a data source. The higher the collection frequency, the smaller this value is, and the greater the contribution to the task priority is.

[0097] Indicates The ratio of the data volume of a data source to the total data volume of all data sources. The larger the data volume, the higher this ratio is, and the greater the contribution to the task priority is.

[0098] The higher the collection frequency and the larger the data volume, the higher the priority of the data source task.

[0099] When the data acquisition module is used for intelligent scheduling of data sources, it collects information about the data source, including the collection frequency requirements and data volume, calculates the task priority of each data source according to the task priority function, and arranges the execution time and sequence of the collection tasks in order from high to low priority. When executing the collection task, it dynamically adjusts the task priority according to the actual situation. For example, when the data volume of a certain data source suddenly increases, its task priority is increased. The data source management technology is used to register and manage various data sources and determine the collection method and frequency, so as to achieve effective control of different types of data sources, ensure that data can be stably obtained from multiple data sources, meet the data needs of the system, and use template management technology. By designing corresponding data collection templates according to different data sources and data types, users can quickly select appropriate templates when collecting data, reduce configuration workload, and improve the efficiency and convenience of data collection. By adopting intelligent scheduling algorithm technology, by setting the task execution time and sequence according to the characteristics of the data source and collection requirements, it can reasonably arrange the collection tasks and improve the collection efficiency. It can make full use of system resources and ensure the timeliness and accuracy of data collection. By using data capture technology, by quickly capturing data from the data source and performing preliminary processing and verification, it can achieve the effect of efficiently acquiring data and ensuring the initial quality of data, laying a good foundation for subsequent data governance and storage.

[0100] The data governance module is used to configure basic data governance operations, including data modeling, data query, and data cleaning. According to business needs and data sources, the entities, attributes, and relationships of the data are determined, and a data model is established. During data collection and storage, data is organized and managed according to the data model. Users input query conditions, and the system quickly retrieves eligible data based on the data model and index. The collected data is processed using a data cleaning algorithm. Machine learning algorithms are used to train and learn the data, and data quality assessment models and anomaly detection models are established. During data processing, data quality is monitored in real time, and anomalies are detected and reported in a timely manner. The data governance module uses machine learning algorithms to establish a data quality assessment model to evaluate the accuracy, completeness, and consistency of the data, and sets the data set to have data samples, each with Features, using represents a data set, where Indicates The feature vector of samples defines the quality assessment function:

[0101] ;

[0102] in:

[0103] Represents the quality assessment result of the entire data set, which is obtained by summing the quality scores of each sample;

[0104] Represents a single sample The quality score is calculated by combining one or more of the following indicators: accuracy, completeness, and consistency. The accuracy indicator is calculated as follows:

[0105] The accuracy index is measured by calculating the error between the sample and the true value, and the accuracy index is expressed as mean square error:

[0106] ;

[0107] in:

[0108] Indicates The true value of samples;

[0109] Indicates The predicted value of a sample, that is, the result obtained through model prediction, that is, the result obtained by predicting the sample through the established prediction model. The prediction model can be selected according to specific business needs and data characteristics. The model analyzes and predicts various characteristics of the sample to obtain the predicted value. With the training and optimization of the model, the predicted value will continue to approach the true value, thereby improving the accuracy of the data;

[0110] Represents the mean square error, which is used to measure the difference between the predicted value and the true value. It represents the square of the difference between the predicted value and the true value. The sum of all samples is divided by the total number of samples to get the average square error value.

[0111] when The smaller the time, the higher the accuracy, and the higher the quality score.

[0112] The completeness index is measured by the proportion of missing values ​​in the statistical sample, defining the data set The total number of eigenvalues ​​in is:

[0113] ;

[0114] set up Represents the total number of missing values ​​for all features in the dataset;

[0115] For numeric data, if a feature value is NaN or a predefined specific value indicating a missing value, such as -9999;

[0116] For categorical data, if a feature value is an empty string or a predefined category label indicating missing, the value is considered missing, and the data set is traversed to count the number of such missing values.

[0117] The missing value ratio is:

[0118] ;

[0119] pass Indicates the integrity score and sets the interval for calculating the integrity score:

[0120] when , that is, when the missing value ratio is less than 10%, ;

[0121] when , that is, when the missing value ratio is between 10% and 30%, ;

[0122] when , that is, when the missing value ratio is greater than 30%, .

[0123] The consistency index is measured by comparing whether the values ​​of the same feature between different samples are consistent;

[0124] pass Represents the consistency score, which can be determined based on the ratio of the number of samples that violate the consistency rule to the total number of samples. data set, the proportion of samples that violate the consistency rule is:

[0125] ;

[0126] when , that is, when the proportion of samples violating the consistency rule is less than 5%, ;

[0127] when , that is, when the proportion of samples violating the consistency rule is between 5% and 15%, ;

[0128] when , that is, when the proportion of samples violating the consistency rule is greater than 15%, ;

[0129] According to business needs and the importance of data, different weights are assigned to the accuracy, completeness and consistency assessment scores, and the comprehensive score of data quality is calculated using the weighted summation method. When the data quality assessment model is actually used, the data is first collected and preprocessed, including noise removal, missing value processing, etc., and one or more quality assessment indicators of accuracy, completeness and consistency are selected. For each indicator, a corresponding calculation method is designed, such as mean square error, missing value ratio, feature consistency comparison, etc. Different weights are assigned according to the importance of each indicator, and the quality score of the sample is comprehensively calculated. The quality of the entire data set is evaluated to obtain an overall picture of the data quality.

[0130] Anomaly detection model is established in the data governance module to detect outliers in the data. data samples, using Represents a data set, each sample has features, and use a distance-based anomaly detection method to calculate the Euclidean distance between each sample and other samples:

[0131] ;

[0132] in, Indicates the data set Samples and The Euclidean distance between samples is Indicates The sample eigenvalues, Indicates The sample eigenvalues;

[0133] Define anomaly score function to represent samples The total distance to other samples is:

[0134] ;

[0135] Set a threshold ,when , it is considered that the sample is an outlier, when , it is considered that the sample is the normal value;

[0136] The data governance module adopts data modeling technology. By determining data entities, attributes and relationships and establishing data models, it can organize and manage data in an orderly manner during data collection and storage. This makes the data storage structure clearer and facilitates subsequent data query and analysis operations. By using data query technology, the system can quickly retrieve data based on data models and indexes after the user enters the query conditions, so that users can efficiently obtain data that meets specific needs, greatly saving the time of users looking for data and improving work efficiency. By using data cleaning technology, the method of processing the collected data using data cleaning algorithms can achieve the effect of removing noise, errors and duplicate data in the data, thereby improving the accuracy and reliability of the data. The data quality assessment model is established using machine learning algorithms. By evaluating the accuracy, completeness and consistency of the data, the effect of fully grasping the data quality status is achieved, and data quality problems are discovered in a timely manner, providing a reliable basis for further data processing and analysis. Anomaly detection models are established, and distance-based anomaly detection methods are adopted. By real-time monitoring of data quality and timely alarming when abnormal situations are found, the effect of quickly identifying abnormal values ​​in the data is achieved, which helps to take timely measures to avoid the adverse effects of abnormal data on subsequent analysis and decision-making.

[0137] The data service module provides functions related to data services, including API management, operation and maintenance monitoring and data statistics. It designs and develops APIs for data services, provides data access interfaces to the outside world, performs identity authentication and permission control during API calls, uses real-time monitoring algorithms to monitor server performance, network traffic, data access volume, etc. in real time, and promptly issues alarms and takes appropriate measures to handle abnormal situations. It conducts statistical analysis on the use of data services and generates statistical reports. By analyzing statistical data, it understands user needs and behaviors, provides a basis for optimizing services, and uses real-time monitoring algorithms and data analysis techniques to ensure service quality and data accuracy. The real-time monitoring algorithm in the data service module is used to monitor server performance, network traffic, data access volume, etc. in real time, promptly detect abnormal situations, and set the server performance indicator set to , It is the internal performance index of the server, including CPU utilization, memory utilization, network traffic Indicated by data access volume Use to indicate;

[0138] Define the anomaly detection function:

[0139] ;

[0140] in, Indicates the indicator The anomaly detection function sets a normal range for CPU utilization ,when When the CPU utilization is abnormal, ,when When the CPU utilization is normal, , Indicates the anomaly detection parameters for network traffic. Expressed as the amount of data transmitted per unit time, such as bytes / second, set a normal range. When it is not within this range, the network traffic is considered abnormal, and e(F)=0. When the network traffic F is within this range, the network traffic is considered normal, and e(F)=1. Represents an anomaly detection function for data access volume. It is expressed as the number of data accesses per unit time, setting a normal range. When it is not within this range, the data access volume is considered abnormal, and e(A)=0; when the data access volume When it is within this range, the data access volume is considered normal, and e(A)=1.

[0141] The data service module adopts API management technology. By designing and developing the API of data services and providing data access interfaces to the outside world, it makes it easy for external applications to obtain and use system data, promotes data sharing and integration, and provides data support for different application scenarios. It uses operation and maintenance monitoring technology to monitor server performance, network traffic, and data access volume in real time, so as to timely discover problems in system operation and ensure service quality, respond to and solve problems quickly, and ensure the stability and reliability of data services. It adopts data statistics technology to conduct statistical analysis on the use of data services and generate statistical reports, so as to understand user needs and behaviors and provide a basis for optimizing services, which helps to improve services in a targeted manner and increase user satisfaction.

[0142] The data asset module manages data assets, including the creation and maintenance of asset catalogs, the processing of assets to be allocated, and the statistical summary of assets. According to the characteristics and classification standards of data assets, a data asset catalog is created. Data assets are classified and described in the asset catalog, and the asset catalog is regularly maintained and updated. Classification and coding and other technologies are used to classify and code data assets. An asset catalog update mechanism is established to regularly update and maintain the asset catalog. Unallocated assets are registered and reviewed to determine the value and purpose of assets. According to the characteristics and needs of assets, they are allocated and managed. Data asset evaluation algorithms and asset classification technologies are used to summarize data assets and generate asset statistical reports, including asset quantity, value, usage, etc., to provide a basis for asset management and decision-making. In the data asset module, data assets are evaluated by data asset evaluation algorithms to determine the value of assets. Let the data asset set be ,Each data asset has multiple evaluation indicators, including data accuracy, completeness, timeliness, etc.;

[0143] Define the asset value function:

[0144] ;

[0145] in:

[0146] Indicates The value score of each data asset;

[0147] It is The weight coefficient of each evaluation indicator is used to adjust the importance of different evaluation indicators in the calculation of asset value;

[0148] It is a data asset In the The scores on the evaluation indicators are calculated by evaluating the accuracy, completeness and timeliness of the data.

[0149] For the data accuracy indicator, the score is calculated based on the error between the data and the true value, assuming the true value is , the predicted value is , the accuracy score formula is as follows:

[0150] ;

[0151] in, represents the maximum mean square error of all data assets on the accuracy metric, ;

[0152] For the completeness indicator, the score is calculated by calculating the proportion of missing values ​​in the data asset. The total number of records is , where the number of records with missing values ​​is , then the completeness score can be defined as:

[0153] ;

[0154] For the timeliness index, the score is calculated based on the difference between the data update time and the current time. Suppose the data update time is , the current time is , set a timeliness threshold ,then , think that the data is timely and has a higher score; when When the data is considered to be of poor timeliness, the score is low.

[0155] The asset classification technology in the data asset module classifies data assets, manages and uses them, and uses clustering algorithms to classify data assets. Suppose the data asset set is , each data asset is represented by a feature vector, , the K-Means algorithm is used. The goal is to divide the data assets into clusters so that the sum of the distances from each data asset to the center of its cluster is minimized. The objective function formula is defined as follows:

[0156] ;

[0157] in:

[0158] It represents the objective function of clustering, which is used to measure the quality of clustering. The goal is to minimize the value of this function.

[0159] Representing data assets Is it a cluster? , if it belongs to ;otherwise, ;

[0160] It is clustering The center of is a vector composed of data asset feature vectors;

[0161] Indicates Data assets and The square of the Euclidean distance between cluster centers is used to measure the distance between data assets and cluster centers;

[0162] The data asset module adopts data asset evaluation algorithm technology. By performing multi-indicator evaluation on data assets, it can determine the value of data assets, provide important reference for data asset management and decision-making, and help to reasonably allocate resources. It uses asset classification technology and classifies data assets using clustering algorithms to facilitate the management and use of data assets, making data assets more orderly and improving the efficiency of finding and using data assets. By creating and maintaining data asset catalogs, processing assets to be allocated, and performing statistical summaries, it can achieve the effect of comprehensive management and effective utilization of data assets, thereby improving the management level and value of data assets.

[0163] The system management module configures and manages the global information of the system, including organizational structure, user information, role permissions and application configuration. It assigns corresponding permissions to users according to their roles and responsibilities. When users log in to the system, their permissions are verified to ensure that they can only access and operate authorized functions and data. According to the usage of the system and business needs, the organizational structure is continuously optimized to ensure the normal operation of the system and the coordination between various parts. The permission management technology is used to assign corresponding permissions to users according to their roles and responsibilities, so as to ensure that users can only access and operate authorized functions and data, thereby ensuring the security and confidentiality of data. The organizational structure optimization algorithm technology is used to continuously optimize the organizational structure to adapt to the usage of the system and business needs, so as to ensure the normal operation of the system and the coordination between various parts, thereby improving the overall efficiency and management level of the system. Specific embodiment 2:

[0165] like Figure 1-5 As shown, according to the contents in the above specific embodiments, the following contents are further disclosed:

[0166] The system management module further includes the following:

[0167] Organizational structure management: not only manage departments and institutions, but also further refine them to project groups, teams and other levels to adapt to more complex business scenarios, clarify the reporting relationships and collaboration methods between different levels, and improve the flexibility and responsiveness of the organization;

[0168] User information management: In addition to basic user names, passwords and contact information, it also records the user's professional background, work experience and other information in order to better assign tasks and permissions to users. At the same time, it provides user self-service functions, such as changing passwords and updating personal information.

[0169] Role and authority management: In addition to traditional roles such as administrator and operator, more specific roles are created according to specific business needs, such as data analyst and data quality manager. Each role has different operation permissions and data access scope to ensure data security and confidentiality.

[0170] Application configuration management: not only configure system parameters, but also customize configurations for different application scenarios. For example, for different business departments, set different data collection frequencies, data cleaning rules, and data service interfaces to meet their respective business needs.

[0171] The data governance module further includes the following:

[0172] Data modeling: When determining the entities, attributes, and relationships of data, tools such as entity relationship diagrams (ERDs) are used for visual modeling to make the data model more intuitive and easier to understand. At the same time, a data version management mechanism is introduced to enable effective tracking and management when the data model changes;

[0173] Data query: In addition to users entering query conditions for retrieval, it provides intelligent query function, automatically recommends relevant data query results based on the user's historical query records and behavior patterns, and supports complex query statements and aggregation functions to meet the data analysis needs of advanced users;

[0174] Data cleaning: Remove noise, errors and duplicate data from the collected data to improve data quality. The specific steps include:

[0175] Sp1: De-duplication processing: For structured data, the key fields of each record, such as unique identifiers and primary keys, are compared to determine whether there are duplicates. If duplicate records are found, one of the records is retained and the other duplicate records are deleted. For unstructured data, text similarity algorithms, such as cosine similarity and Jaccard similarity, are used to determine the similarity of text content, and the content with higher similarity is treated as duplicate data for processing;

[0176] Sp2: Missing value processing: For numerical data, use statistical values ​​such as mean, median, mode, etc. to fill in the missing values. For example, if most of the data values ​​in a field are within a certain range, use the mean value within the range to fill in the missing values; for categorical data, use the mode to fill in, that is, select the category with the highest frequency in the field as the filling value for the missing values;

[0177] Sp3: Outlier processing:

[0178] Based on statistical methods: calculate the mean and standard deviation of the data, and treat data that exceeds a certain multiple of the standard deviation from the mean as outliers. For example, for normally distributed data, data that exceeds ±3 times the standard deviation from the mean can be treated as outliers for processing;

[0179] Based on the box plot: Use the interquartile range (IQR) of the box plot to determine outliers. ,in is the lower quartile, is the upper quartile, which will be less than or greater than The data are considered as outliers;

[0180] Data quality assessment model: In addition to accuracy, completeness and consistency, it also adds other evaluation indicators such as data timeliness and uniqueness. At the same time, it uses advanced machine learning algorithms such as deep learning to improve the accuracy and robustness of data quality assessment;

[0181] Anomaly detection model: In the anomaly detection method, time series analysis and other technologies are combined to monitor abnormal changes in data in real time. At the same time, an abnormal event library is established to classify and record common abnormal situations so that problems can be quickly located and corresponding measures can be taken when an abnormality occurs.

[0182] The data acquisition module further includes the following:

[0183] Data source management: In addition to registering and managing traditional data sources such as databases and file systems, it supports emerging data source types such as IoT devices and social media. At the same time, it establishes a data source monitoring mechanism to monitor the status and changes of data sources in real time to ensure the stability and reliability of data collection;

[0184] Template management: When designing data collection templates, we use technologies such as template engines to achieve dynamic generation and customization of templates. At the same time, we establish a template sharing mechanism to allow users to share and reuse excellent collection templates, thereby improving collection efficiency and quality.

[0185] Automatic collection: In the intelligent scheduling algorithm, a prediction model is introduced to predict future data collection needs based on historical data and trends, and collection tasks are arranged in advance to improve the response speed and efficiency of the system. At the same time, distributed collection technology is used to distribute collection tasks to multiple nodes for parallel execution, thereby improving the throughput and speed of collection.

[0186] Manual collection: In the method of manual input or import of data, data verification and validation functions are provided to ensure that the manually collected data meets the data quality requirements. At the same time, a data review mechanism is established to review and approve the manually collected data to ensure the accuracy and reliability of the data.

[0187] The data service module further includes the following:

[0188] API management: When designing and developing data service APIs, standard specifications such as RESTful APIs are adopted to improve the versatility and scalability of APIs. At the same time, API documentation and testing tools are provided to facilitate external application developers to use and integrate data services.

[0189] Operation and maintenance monitoring: In the real-time monitoring algorithm, add monitoring of data service performance indicators, such as response time, throughput, etc., and establish an early warning mechanism. When performance problems or abnormal situations occur in data services, relevant personnel are notified in a timely manner for processing;

[0190] Data statistics: When conducting statistical analysis on the usage of data services, data visualization technology is used to display the statistical results in the form of charts, etc., so that users can intuitively understand the usage and trends of data services, establish data analysis models, conduct in-depth analysis of statistical data, explore user needs and behavior patterns, and provide a basis for optimizing data services.

[0191] The data asset module further includes the following:

[0192] Creation and maintenance of asset catalogs: When creating data asset catalogs, use classification and coding techniques to classify and code data assets to make the asset catalogs more standardized and easier to manage. Establish an asset catalog update mechanism to regularly update and maintain the asset catalogs to ensure the accuracy and completeness of the asset catalogs.

[0193] Processing of assets to be distributed: When registering and reviewing assets to be distributed, establish an asset valuation model, evaluate and predict the value and use of assets, and distribute assets through auctions, bidding, etc. to improve the utilization efficiency and value of assets.

[0194] Asset statistics summary: When summarizing the statistics of data assets, add indicators such as risk assessment of assets, such as data leakage risk, data loss risk, etc., establish an asset reporting mechanism, generate asset statistics reports regularly, and report the management status and value changes of data assets to management and relevant personnel. Specific embodiment three:

[0196] like Figure 1-5 As shown, according to the contents in the above specific embodiments, the following contents are further disclosed:

[0197] The data capture technology in the data governance module is mainly used to quickly and accurately obtain data from various data sources. The specific contents are as follows:

[0198] Web crawler technology: For web page data, use a web crawler program to automatically traverse web page links and grab the required data. Web crawlers usually include steps such as page downloading, page parsing, and data extraction. In the page downloading stage, use HTTP requests to obtain web page content; in the page parsing stage, use HTML parsers such as BeautifulSoup, lxml, etc. to parse the web page content into structured data; in the data extraction stage, extract the required data from the parsed web page according to specific rules and patterns.

[0199] Database connection technology: For database data sources, use database connection drivers, such as JDBC, ODBC, etc., to establish a connection with the database, and then obtain data through SQL query statements. Select the appropriate database connection method and query statement according to the type and characteristics of the data source to improve the efficiency and accuracy of data capture.

[0200] API calling technology: For data sources that provide API interfaces, use the corresponding API calling methods to obtain data. First, you need to understand the API documentation of the data source, determine the API request parameters, return format, and calling method, and then use the HTTP client library in the programming language, such as Python's requests library, Java's HttpClient, etc., to send API requests, obtain data, and parse and process it.

[0201] The design of the data collection template in the data collection module should be customized according to different data sources and data types to improve collection efficiency and reduce configuration workload. The following is the specific content of the design of the data collection template as shown in Table 1:

[0202]

[0203] Table 1

[0204] When the data service module detects an abnormal situation, the following measures in Table 2 can be taken to handle it:

[0205]

[0206] Table 2 Specific embodiment four:

[0208] like Figure 1-5 As shown, according to the contents in the above specific embodiments, the following contents are further disclosed:

[0209] The operation steps of the entire system are as follows:

[0210] Sp1: system initialization;

[0211] The Sp1.1 system management module first performs global configuration and management of the system, including creating organizational structure, adding user information, assigning role permissions, and performing application configuration;

[0212] Sp1.2 Data governance module determines the entities, attributes and relationships of data according to business needs and data sources, establishes data models, and creates data quality assessment models and anomaly detection models;

[0213] Sp1.3 data collection module registers and manages various data sources, determines the data collection method and frequency, and designs corresponding data collection templates;

[0214] Sp1.4 Data service module designs and develops data service APIs, and establishes real-time monitoring algorithms and data statistics mechanisms;

[0215] Sp1.5 Data asset module creates a data asset catalog based on the characteristics and classification standards of data assets, and establishes data asset evaluation algorithms and asset classification technology;

[0216] Sp2: Data collection: The data collection module arranges the execution time and sequence of collection tasks according to the intelligent scheduling algorithm. For automatic collection tasks, it uses data capture technology to quickly capture data from the data source and conducts preliminary processing and verification. If automatic collection cannot meet the needs, users can collect data by manually inputting or importing data;

[0217] Sp3: Data governance: The data governance module cleans the collected data, removes noise, errors and duplicate data, uses the data quality assessment model to evaluate the accuracy, completeness and consistency of the data, and uses the anomaly detection model to detect outliers in the data. If data quality problems or abnormal situations are found, timely alarms are issued and corresponding measures are taken to deal with them;

[0218] Sp4: Data storage and management: Governed data is organized and managed according to the data model and stored in the corresponding database or file system. The data asset module manages data assets, including creating asset catalogs, registering assets to be allocated, and summarizing asset statistics;

[0219] Sp5: Data service: The data service module provides external data access interfaces through API management. External applications can obtain and use data by calling APIs. Real-time monitoring algorithms are used to monitor server performance, network traffic, and data access volume in real time to ensure the quality and stability of data services. Statistical analysis is performed on the use of data services to understand user needs and behaviors, providing a basis for optimizing services.

[0220] Sp6: System optimization and maintenance:

[0221] The Sp6.1 system management module continuously optimizes the organizational structure and application configuration according to the system usage and business needs to ensure the normal operation of the system and the coordination between various parts;

[0222] Sp6.2 data governance module continuously optimizes data cleaning algorithms and models to improve data quality based on data quality assessment and anomaly detection results;

[0223] The Sp6.3 data collection module continuously optimizes the intelligent scheduling algorithm and data collection template according to the changes in data sources and collection requirements to improve collection efficiency;

[0224] The Sp6.4 data service module continuously optimizes API management and real-time monitoring algorithms to improve service quality based on user needs and statistical analysis results;

[0225] The Sp6.5 data asset module continuously optimizes the asset catalog and evaluation algorithm based on asset value changes and management needs, thereby improving asset utilization efficiency and value.

[0226] The entire system architecture adopts data modeling, query and cleaning technologies, and achieves the effect of improving data quality and timely discovering data anomalies by establishing data quality assessment and anomaly detection models. It adopts data source management, template management and intelligent scheduling algorithm technologies, and achieves the effect of efficiently and accurately obtaining data from various data sources by reasonably arranging the execution time and sequence of collection tasks and data crawling methods. It adopts API management, operation and maintenance monitoring and data statistics technologies, and achieves the effect of ensuring data service quality and understanding user needs to optimize services by real-time monitoring of server performance, network traffic and data access volume and analyzing statistical data. It adopts data asset assessment algorithms and asset classification technologies, and achieves the effect of clarifying the value and purpose of data assets and improving asset utilization efficiency by effectively managing and summarizing data assets. It adopts permission management and organizational structure optimization algorithm technologies, and achieves the effect of ensuring the normal operation of the system and the coordinated work of various parts by assigning permissions to users and optimizing the organizational structure. Specific embodiment five:

[0228] like Figure 1-5 As shown, according to the contents in the above specific embodiments, the following contents are further disclosed:

[0229] With respect to the system content in the above specific embodiment, in actual application, it can be applied to data processing in the global tourism management platform and the scenic area tourism management platform. Taking the data access, processing and display of the global tourism management platform as an example, the content is as follows:

[0230] Data access to the global tourism management platform: Data sources include statistical data from local tourism bureaus, user evaluation data from online tourism platforms, ticket sales data from scenic spots, etc. These data sources are registered and managed through the data source management function of the data collection module, and the collection method and frequency of different data sources are determined. For example, the statistical data from the tourism bureau may be updated once a month, and regular automatic collection is adopted; the user evaluation data from online tourism platforms has high real-time requirements, and intelligent scheduling algorithms are used to increase the collection frequency;

[0231] Design corresponding data collection templates according to different data sources and data types. For example, for the statistical report of the Tourism Bureau, a special template can be designed to automatically identify the key data items in the table and collect them. When collecting data, users can select the corresponding template to reduce the configuration workload;

[0232] Data processing of the global tourism management platform: The data governance module processes the collected data. First, data cleaning is performed to remove noise and erroneous data. For example, obviously unreasonable user evaluation data, such as duplicate content and malicious evaluation, are removed. The data quality assessment model is used to assess the accuracy, completeness and consistency of the data. Taking the tourist number data counted by the Tourism Bureau as an example, the accuracy is assessed by comparing it with data from other data sources; the data is checked for missing values ​​and the completeness is assessed; the tourist number data of the same time period counted by tourism bureaus in different regions is compared to assess consistency, and an anomaly detection model is established to detect abnormal data in a timely manner. For example, if the number of tourists in a certain area fluctuates greatly in a short period of time, it may be a data anomaly. The system will issue an alarm in time for further inspection and processing;

[0233] Data display of the global tourism management platform: Through the data API provided by the data service module, the processed data is displayed on the leadership cockpit interface. For example, the tourism popularity of various places is displayed in the form of a map, and different colors are used to indicate the number of tourists; the growth trend of tourism income is displayed in the form of a chart, etc. The data statistics function is used to generate statistical reports to provide a basis for leadership decision-making, such as displaying the development and utilization of tourism resources in different regions, tourist satisfaction and other statistical data;

[0234] Because the entire system uses lightweight algorithms and technologies, data processing is faster and the system can respond to various requests more quickly. Whether it is data collection, governance or service modules, tasks can be completed in a shorter time, reducing user waiting time. For example, in the data collection module, the intelligent scheduling algorithm can quickly determine the execution time and sequence of tasks, and the data capture technology can quickly obtain data from the data source and perform preliminary processing, so that the data can enter the system in time for subsequent processing.

[0235] Lightweight processing reduces the system's demand for computing resources, storage resources, and network resources, which is very important for resource-limited environments or large-scale deployment scenarios. It can reduce hardware costs and energy consumption. For example, in the data service module, the real-time monitoring algorithm uses lightweight monitoring indicators and algorithms to reduce the impact on server performance, allowing the server to provide services for other tasks more efficiently.

[0236] The lightweight design makes system development simpler and more efficient. Developers can understand and implement the functions of each module more quickly, reducing development time and costs. For example, the data cleaning algorithm, quality assessment model, and anomaly detection model in the data governance module use simple and effective methods. Developers can implement these functions more easily and can quickly debug and optimize.

[0237] The lightweight system structure is clearer, the code is more concise, and it is easy to maintain and modify. At the same time, due to the small system resource usage, the impact on the existing system is small when expanding functions. For example, the asset classification technology and evaluation algorithm in the data asset module are relatively simple. When it is necessary to add new evaluation indicators or adjust the classification method, developers can modify and expand them more easily without causing a major impact on the entire system.

[0238] Lightweight processing enables the data system to adapt to global tourism management platforms and scenic area management platforms of different sizes and types. Whether it is a small scenic area or a large tourist area, it can be flexibly configured and deployed according to actual needs. For example, for scenic spots with a small amount of data, the data acquisition module can adopt a lower acquisition frequency and a simple data processing method. For scenic spots with large data volumes and high real-time requirements, the system can improve the efficiency of acquisition and processing through intelligent scheduling algorithms. Due to the low system resource requirements and simple development process, lightweight data systems can be deployed and put online faster, which is very important for scenarios that require rapid response to market demands or emergencies. For example, in new scenic spot projects, the data system can be quickly deployed to provide timely data support for scenic spot management.

[0239] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the statement "comprising a reference structure" do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0240] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and alterations may be made to the embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A lightweight real-time data system, characterized by: The system includes a system management module, a data governance module, a data acquisition module, a data service module and a data asset module, wherein: The data governance module is used to configure basic data governance operations, including data modeling, data query and data cleaning. According to business needs and data sources, the entities, attributes and relationships of the data are determined, and a data model is established. During data collection and storage, the data is organized and managed according to the data model. The user inputs query conditions, and the system quickly retrieves qualified data according to the data model and index, processes the collected data with a data cleaning algorithm, trains and learns the data using a machine learning algorithm, establishes a data quality assessment model and anomaly detection model, monitors data quality in real time during data processing, and promptly issues an alarm when anomalies are discovered. The data governance module uses a machine learning algorithm to establish a data quality assessment model to assess the accuracy, completeness and consistency of the data, and sets the data set to have data samples, each with Features, using represents a data set, where Indicates The feature vector of samples defines the quality assessment function: ； in: Represents the quality assessment result of the entire data set, which is obtained by summing the quality scores of each sample; Represents a single sample The quality score is calculated by combining one or more of the following indicators: accuracy, completeness, and consistency. The accuracy indicator is calculated as follows: The accuracy index is measured by calculating the error between the sample and the true value, and the accuracy index is expressed as mean square error: ； in: Indicates The true value of samples; Indicates The predicted value of a sample, that is, the result obtained by model prediction; Represents the mean square error, which is used to measure the difference between the predicted value and the true value. It represents the square of the difference between the predicted value and the true value. The sum of all samples is divided by the total number of samples to get the average square error value. when The smaller the time, the higher the accuracy, and the quality score increases accordingly. The completeness is measured by the proportion of missing values ​​in the statistical sample, and the consistency is measured by comparing whether the values ​​of the same feature between different samples are consistent; The data governance module establishes an anomaly detection model to detect outliers in the data. data samples, using Represents a data set, each sample has features, and use a distance-based anomaly detection method to calculate the Euclidean distance between each sample and other samples: ； in, Indicates the data set Samples and The Euclidean distance between samples is Indicates The sample eigenvalues, Indicates The sample eigenvalues; Define anomaly score function to represent samples The total distance to other samples is: ； Set a threshold ,when , it is considered that the sample is an outlier, when , it is considered that the sample is the normal value; The data acquisition module is responsible for acquiring data from various data sources and managing the acquisition process, including data source management, template management, automatic acquisition and manual acquisition. It registers and manages various data sources, determines the data acquisition method and frequency, and designs corresponding data acquisition templates according to different data sources and data types. When collecting data, users select corresponding templates to reduce configuration workload. Intelligent scheduling algorithms are used to set the execution time and sequence of tasks according to the characteristics of data sources and acquisition requirements. Data capture technology is used to quickly capture data from data sources, and preliminary processing and verification are performed. When automatic acquisition cannot meet the needs, users collect data by manually inputting or importing data. The intelligent scheduling algorithm in the data acquisition module arranges the execution time and sequence of acquisition tasks according to the characteristics of data sources and acquisition requirements. Suppose the data source set is , each data source has different collection frequency requirements and data size ; Define the task priority function: ； in: and is the weight coefficient, which is adjusted according to the actual situation. Indicates The collection frequency requirements of each data source; Indicates The amount of data in each data source is measured in bytes and records. Indicates The task priority of each data source; Indicates The reciprocal of the collection frequency of a data source. The higher the collection frequency, the smaller this value is, and the greater the contribution to the task priority is. Indicates The ratio of the data volume of a data source to the total data volume of all data sources. The larger the data volume, the higher this ratio is, and the greater the contribution to the task priority is. The higher the collection frequency and the larger the data volume, the higher the priority of the data source task; The data service module provides functions related to data services, including API management, operation and maintenance monitoring and data statistics, designs and develops APIs for data services, provides data access interfaces to the outside world, performs identity authentication and permission control during API calls, uses real-time monitoring algorithms to monitor server performance, network traffic, data access volume, etc. in real time, and promptly issues alarms and takes corresponding measures to handle abnormal situations, performs statistical analysis on the use of data services, generates statistical reports, and understands user needs and behaviors through analysis of statistical data, providing a basis for optimizing services. Real-time monitoring algorithms and data analysis technologies are used to ensure service quality and data accuracy. The real-time monitoring algorithm in the data service module is used to monitor server performance, network traffic, data access volume, etc. in real time, promptly detect abnormal situations, and set a server performance indicator set as , It is the internal performance index of the server, including CPU utilization, memory utilization, network traffic Indicated by data access volume Use to indicate; Define the anomaly detection function: ； in, Indicates the indicator The anomaly detection function sets a normal range for CPU utilization ,when When the CPU utilization is abnormal, ,when When the CPU utilization is normal, ; The data asset module manages data assets, including the creation and maintenance of asset catalogs, the processing of assets to be allocated, and the statistical summary of assets. According to the characteristics and classification standards of data assets, a data asset catalog is created, data assets are classified and described in the asset catalog, and the asset catalog is regularly maintained and updated. Assets that have not yet been allocated are registered and reviewed to determine the value and purpose of the assets. According to the characteristics and needs of the assets, they are allocated and managed. Data asset evaluation algorithms and asset classification techniques are used to statistically summarize data assets and generate asset statistical reports, including asset quantity, value, usage, etc., to provide a basis for asset management and decision-making. In the data asset module, data assets are evaluated by a data asset evaluation algorithm to determine the value of assets. Let the data asset set be ,Each data asset has multiple evaluation indicators, including data accuracy, completeness, timeliness, etc.; Define the asset value function: ； in: Indicates The value score of each data asset; It is The weight coefficient of each evaluation indicator is used to adjust the importance of different evaluation indicators in the calculation of asset value; It is a data asset In the The scores on the evaluation indicators; For the data accuracy indicator, the score is calculated based on the error between the data and the true value, assuming the true value is , the predicted value is , the accuracy score formula is as follows: ； in, Represents the maximum mean square error of all data assets on the accuracy metric; The asset classification technology in the data asset module classifies data assets, manages and uses them, and uses clustering algorithms to classify data assets. Suppose the data asset set is , each data asset is represented by a feature vector, , the K-Means algorithm is used. The goal is to divide the data assets into J clusters so that the sum of the distances from each data asset to the center of its cluster is minimized. The objective function formula is defined as follows: ； in: Represents the objective function of clustering, which is used to measure the quality of clustering. The goal is to minimize the value of this function; Representing data assets Is it a cluster? , if it belongs to ;otherwise, ; It is clustering The center of is a vector composed of data asset feature vectors; Indicates Data assets and The square of the Euclidean distance between cluster centers is used to measure the distance between data assets and cluster centers; The system management module configures and manages the global information of the system, including organizational structure, user information, role permissions and application configuration. It assigns corresponding permissions to users based on their roles and responsibilities, verifies their permissions when they log in to the system, and ensures that they can only access and operate authorized functions and data. It continuously optimizes the organizational structure based on system usage and business needs to ensure the normal operation of the system and the collaborative work between its parts.

2. A lightweight real-time data system according to claim 1, characterized in that: When creating a data asset catalog in the data asset module, classification and coding techniques are used to classify and encode data assets, establish an asset catalog update mechanism, and regularly update and maintain the asset catalog.

3. A lightweight real-time data system according to claim 1, characterized in that: The data collection module provides data verification and validation functions in the manual input or import of data to ensure that the manually collected data meets the data quality requirements, establishes a data review mechanism, reviews and approves the manually collected data, and ensures the accuracy and reliability of the data.

Images