Full-cycle data management method for digital transformation of small and medium-sized enterprises

Abstract

The invention discloses a full-cycle data management method for digital transformation of small and medium-sized enterprises, and the method comprises the following steps: obtaining enterprise characteristic data, and building a data quality monitoring model; collecting enterprise related data related to the business target and digital transformation of the enterprise, wherein the enterprise related data comprises enterprise internal business data, production data and enterprise external data; and checking the data quality of the enterprise related data by using the data quality monitoring model to obtain effective data. According to the method provided by the invention, the data acquisition quality can be effectively ensured by utilizing the established data quality monitoring model, so that powerful data support is provided for subsequent analysis of the bottleneck and risk of the whole business process in enterprise digital transformation.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a full-cycle data management method for digital transformation of small and medium-sized enterprises. Background Art

[0002] In today's digital age, small and medium-sized enterprises face many challenges and opportunities. Digital transformation has become a key way for small and medium-sized enterprises to enhance their competitiveness and achieve sustainable development. As the core element of digital transformation, data management is crucial.

[0003] In today's digital age, small and medium-sized enterprises face many challenges and opportunities. Enterprise digital transformation has become an unstoppable trend and presents various forms. Digital transformation has become a key way for small and medium-sized enterprises to enhance their competitiveness and achieve sustainable development. Many enterprises have integrated internal business processes, achieved centralized management and sharing of data, and improved operational efficiency by introducing advanced information technology systems, such as enterprise resource planning (ERP) and customer relationship management (CRM) software. At the same time, emerging technologies such as big data analysis, artificial intelligence, and the Internet of Things have gradually been integrated into various aspects of enterprise production, sales, and services, helping enterprises make accurate decisions, optimize products and services, and innovate business models.

[0004] However, enterprises are facing many current situations and problems in the process of digital transformation. On the one hand, the amount of data is growing explosively, but the data quality is uneven. A large amount of data contains noise, missing values ​​and erroneous information, which seriously affects the subsequent data analysis and application. Small and medium-sized enterprises often do not have enough resources to control data quality, so a third-party general management method is needed to manage data for small and medium-sized enterprises. Summary of the invention

[0005] The purpose of the present invention is to provide a full-cycle data management method for the digital transformation of small and medium-sized enterprises, so as to at least solve the problem that small and medium-sized enterprises have difficulty in controlling data quality.

[0006] In a first aspect, the present invention provides a full-cycle data management method for the digital transformation of small and medium-sized enterprises, comprising the following steps: acquiring enterprise-specific data and establishing a data quality monitoring model; collecting enterprise-related data related to the enterprise's business objectives and digital transformation, wherein the enterprise-related data includes internal business data, production data, and external data of the enterprise; and using the data quality monitoring model to verify the data quality of the enterprise-related data to obtain valid data.

[0007] In some embodiments, the establishment of a data quality monitoring model includes: establishing an enterprise data governance framework and organizational structure based on the enterprise characteristic data of the digital transformation, including defining the goals, scope, roles and responsibilities of data governance; using machine learning algorithms to analyze the enterprise's existing data standards and business needs to obtain new data standards that meet the specific needs of the enterprise; and according to the enterprise's new data standards, matching appropriate data standards for data fields when creating tables in the database, and matching appropriate data standards for existing data; the new data standards include field standards, code value standards and dictionary management standards; based on the new data standards, establishing data quality inspection models of multiple dimensions, and the data quality inspection models of multiple dimensions constitute a data quality monitoring model; wherein the data quality inspection models of multiple dimensions include: accuracy inspection model, compliance inspection model, completeness inspection model, timeliness inspection model, consistency inspection model, and availability inspection model.

[0008] In some embodiments, the use of machine learning algorithms to analyze the existing data standards and business needs of an enterprise includes: obtaining the historical data of the enterprise; introducing polynomial terms and interaction terms to construct a regularized logistic regression model; using the enterprise-related data to train the regularized logistic regression model to predict the category labels to which the data features of the historical data belong; using kernel techniques and introducing soft intervals and regularization parameters to construct a soft interval support vector machine; based on the nonlinear data of the enterprise-related data, training the soft interval support vector machine to find the best separating hyperplane between different categories to determine the category labels of the data features of the historical data.

[0009] In some embodiments, the collection of enterprise-related data related to the enterprise's business goals and digital transformation includes: determining collection targets based on the enterprise's customer needs, market trends, and product feedback; collecting data based on the collection targets through one or more of the following methods: directly acquiring data from the data source; collecting the enterprise's on-site environmental data through various enterprise sensing devices; and using a distributed crawler system to crawl and collect external network data.

[0010] In some embodiments, the enterprise-related data is processed according to the data characteristics and business needs of the enterprise to obtain valid enterprise data, including: cleaning and processing the data, including: outlier detection and processing, data standardization and normalization, data missing value processing; using artificial intelligence algorithms to extract and select features from text data, image data, and structured data; establishing a data quality assessment model, and using the data quality monitoring model to monitor and feedback data.

[0011] Through the above scheme provided by the present invention, the beneficial effects of the present invention are as follows: by establishing a data quality monitoring model and a data quality inspection model in multiple dimensions, through data collection, cleaning, and processing processes, the quality of enterprise data can be effectively monitored, and data quality issues can be responded to in a timely manner to ensure data reliability and business continuity. The problem of uneven data quality is effectively solved, and the accuracy, completeness, consistency, and timeliness of data are significantly improved, providing a reliable data foundation for enterprises. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The present invention will be further described below in conjunction with the accompanying drawings.

[0013] Figure 1 It is an exemplary flow chart of a full-cycle data management method for digital transformation of small and medium-sized enterprises provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0014] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0015] Figure 1 It is an exemplary flow chart of a full-cycle data management method 100 for digital transformation of small and medium-sized enterprises provided in an embodiment of the present invention.

[0016] like Figure 1 As shown, the method 100 includes step S101: obtaining enterprise characteristic data and establishing a data quality monitoring model.

[0017] In one embodiment, the step of establishing a data quality monitoring model includes: First, based on the enterprise-specific data of the digital transformation, an enterprise data governance framework and organizational structure are established, including defining the goals, scope, roles and responsibilities of data governance, so as to ensure the effective implementation of data governance work.

[0018] Then, using machine learning algorithms, analyze the existing data standards and business needs of the enterprise to obtain new data standards that meet the specific needs of the enterprise; and according to the new data standards of the enterprise, match the appropriate data standards for the data fields when creating tables in the database, and match the appropriate data standards for the existing data. The new data standards include field standards, code value standards, and dictionary management standards. The field standards may include data types, lengths, or formats, etc. The dictionary management standards may include standard rules for maintaining and updating data dictionaries.

[0019] Then, based on the new data standard, a multi-dimensional data quality inspection model is established, and the multi-dimensional data quality inspection model constitutes a data quality monitoring model. Among them, the multi-dimensional data quality inspection model includes: accuracy inspection model, compliance inspection model, completeness inspection model, timeliness inspection model, consistency inspection model, and availability inspection model.

[0020] It can be understood that a data quality monitoring model is formulated to perform quality checks on enterprise data, wherein the quality check dimensions may include accuracy, compliance, completeness, timeliness, consistency, availability, integrity, etc. The data quality monitoring model includes configuration functions, alarm management functions, etc.

[0021] In the process of enterprise digital transformation, data quality monitoring model is the core of data quality management, which is used to evaluate the quality and integrity of data. By building a rich and extensible data quality monitoring standard library, different business scenarios and data types can be covered. It can also provide the function of customizing data quality monitoring model and perform syntax checking to ensure the correctness and effectiveness of the standard.

[0022] The configuration function in the data quality monitoring model allows users to configure monitoring standards through the interface, including selecting data sources, databases, tables or fields, and configuring specific monitoring standards. Based on the search results, a variety of template standards can be built in, supporting standard types such as library level, table level, and field level. Monitoring standards can be quickly configured based on these templates, or you can customize standards to meet specific business needs. The alarm management function is used to notify relevant personnel in a timely manner when problems are found during the data quality monitoring process. Supports multiple alarm methods, including email notifications, SMS notifications, DingTalk group robots, etc. You can set alarm levels, such as fuse phone alarms, email alarms, etc., and automatically trigger alarms based on monitoring results.

[0023] Specifically, the data quality inspection model may include the following multiple dimensions: Accuracy check model: Data accuracy check can be used to check whether the data is correct, for example, by comparing the source data and the target data to verify the accuracy of the data. The formula can be: .

[0024] Compliance Check Model: Compliance dimension checks can ensure that data complies with specific regulations and business standards, such as data masking and privacy protection.

[0025] Completeness check model: The completeness check can be used to check whether the data is comprehensive and has no missing values. For example, the completeness of the data can be evaluated by counting the number of records in the table.

[0026] Timeliness check model: Data timeliness check can ensure the timeliness of data, for example, by checking the time slice to evaluate whether the data is updated in time. The formula can be: .

[0027] Usability check model: Usability checks can ensure that data can be used correctly, for example by checking the accessibility and readability of the data.

[0028] Consistency check model: Data consistency check can be used to check the consistency of data at different sources or time points. For example, consistency can be evaluated by comparing historical data with current data. The formula can be: .

[0029] The data integrity check can be used to check whether the data is complete.

[0030] The formula can be: .

[0031] Then, a data security model can be developed, which includes data desensitization, data security classification, and data security monitoring to ensure that data is not accessed or leaked without authorization. Data desensitization is designed to protect sensitive information and private data from unauthorized access or leakage. It includes static data desensitization, which desensitizes data stored in the database for safe use in non-production environments. Dynamic data desensitization: Dynamically desensitize sensitive data when querying or accessing data to ensure that the original data can only be accessed in an authorized environment.

[0032] The data access permission scoring formula can be: .

[0033] Data security classification can be based on national standards or enterprise-defined classification standards. A basic framework of three levels, general data, important data, and core data, is established through a data classification framework, and classification is carried out in combination with industry data classification and grading rules or organizational production practices.

[0034] Data security monitoring is the monitoring and auditing of the data operation plane, such as monitoring of access behavior of databases and data storage, monitoring and auditing of operation behavior. Specific implementations can include real-time monitoring of data access and operations, timely detection of abnormal behaviors and potential security threats; recording and auditing data access logs to track and analyze security incidents when they occur; building a multi-level alarm system for big data network security to provide early warnings and alarms for security risks at different levels; and implementing alarm management functions, including configuration of alarm rules, alarm notifications, and alarm responses.

[0035] Then, unified data modeling can be developed, which includes business system modeling and model management to support data governance. Business system modeling involves understanding and analyzing the business processes and data requirements of the enterprise. It includes identifying the data sources in the business system and modeling the existing database through methods such as reverse engineering to form a source data model, which can quickly understand the business and accurately analyze business requirements; in the modeling process, the data dictionary is completed, the data items with the same name but different meanings are uniformly defined, and the data items with the same meaning but different names are merged to form an enterprise-level dictionary, so that data item standardization can be achieved.

[0036] Model management is the process of unified storage, management and maintenance of data models, including metadata management, unified storage of data models, monitoring and management of models, etc.

[0037] Finally, a data governance effectiveness verification and evaluation model can also be developed, including the maximization of data assets, the integrity and traceability of data lineage. Data lineage refers to the relationship between data formed throughout its life cycle.

[0038] The formula for evaluating the value of data assets can be: .

[0039] By effectively verifying and evaluating the effectiveness of data governance, we ensure that the value of data assets is maximized, while ensuring the integrity and traceability of data lineage, providing a solid foundation for enterprise data governance.

[0040] Through the method provided by the above embodiment of the present invention, the quality of enterprise data can be effectively monitored, and data quality issues can be responded to in a timely manner to ensure data reliability and business continuity.

[0041] In one embodiment, the machine learning algorithm is used to analyze the existing data standards and business needs of the enterprise. The machine learning algorithm provided in the embodiment of the present invention will be adjusted based on multiple factors, including field standards, code value standards, and dictionary management standards, and will match appropriate data standards for data fields when creating tables in the database, as well as matching appropriate data standards for existing data. Specifically, the following steps may be included: First, obtain the historical data of the enterprise. Preprocess the historical data of the enterprise to ensure data quality. Specific preprocessing may include cleaning, standardization and feature extraction, so as to obtain the characteristic data set of the enterprise.

[0042] Secondly, polynomial terms and interaction terms are introduced to construct a regularized logistic regression model. The model can be used to obtain the probability or category label of each sample belonging to a specific category.

[0043] In order to capture the interaction effect between features in the input feature data set, polynomial terms and interaction terms can be introduced into the model. For example, for two sample feature vectors of the feature data set and , you can also add , and As a new feature.

[0044] Then, the regularized logistic regression model is trained using the enterprise-related data to predict the category label to which the data features of the historical data belong. Specifically, in order to prevent overfitting, an L1 or L2 regularization term can be added to the loss function to construct a regularized logistic regression model, which can be expressed as a log-likelihood function. The formula is: ; in, , represents the probability of the model predicting the positive class (e.g., class 1); is the label of the i-th feature sample, which is the information used for supervision during model training. are input features, is the weight parameter. is the intercept term, is the feature corresponding to The parameters determine the degree of response of the model to the input features, that is, the influence of the features on the output results (whether they meet the data standards). is the regularization parameter, which controls the strength of regularization. The larger the value, the stronger the regularization effect. will be penalized more to reduce the risk of overfitting.

[0045] By introducing polynomial terms and interaction terms, the model can capture nonlinear relationships and interaction effects between features. For example, if there is an interaction effect between the length and data type of a field, this effect can be captured by adding the product term of length and type as a new feature. Therefore, in data governance, if multiple attributes of a field (such as data type, length, format) jointly affect whether it meets data standards, the model's predictive ability can be improved by introducing polynomial and interaction terms.

[0046] A soft margin support vector machine can also be constructed by using kernel techniques and introducing soft margins and regularization parameters. The soft margin support vector machine is trained according to the nonlinear data of the enterprise-related data to find the best separation hyperplane between different categories to determine the category labels of the data features of the historical data.

[0047] Specifically, kernel techniques can be used to map data to a high-dimensional space, so that nonlinear problems can be transformed into linear problems. Common kernel functions include linear kernels, polynomial kernels, and radial basis function kernels (RBF). Among them, the linear kernel is suitable for linearly separable data. The polynomial kernel is suitable for situations where there are polynomial relationships in the data. The RBF kernel is suitable for situations where there are nonlinear relationships in the data and the relationships are complex. In data governance, if there are complex nonlinear relationships between the attributes of the fields, the RBF kernel can be selected to improve the classification ability of the model.

[0048] Then, soft margin and regularization parameter C are introduced into the soft margin support vector machine to allow some data points to violate the margin rule while controlling the width of the margin. The formula can be expressed as: , in, is a slack variable used to handle data points that violate the interval rule. w is a weight vector that determines the importance of different features in the classification decision. b is a bias term that allows the decision boundary to move away from the origin. is the label of the i-th feature sample, is the feature vector of the i-th sample.

[0049] The regularization parameter C can control the strength of regularization. A smaller C value indicates a greater regularization strength, which will make the model parameters closer to 0, thereby reducing the complexity of the model and avoiding overfitting. A larger C value will weaken the regularization effect, allowing the model to be more complex, which may increase the risk of overfitting. In data governance, if you find that the model performs well on the training set but poorly on the validation set, you may need to reduce the C value to enhance regularization, thereby increasing the generalization ability of the model by increasing the interval width. Therefore, if there are some abnormal data points in the analysis data, or there is noise in the data set, you can adjust the C value to balance the model's sensitivity to these abnormal points and improve the robustness of the model.

[0050] From the above description, it can be understood that the decision boundary of data field classification is determined by the w and b parameters, while the C parameter can achieve a trade-off between the complexity of the model and the degree of fit to the training data.

[0051] For multi-class classification problems, one-to-one or one-to-many strategies can be used. For example, for a three-class problem, three-to-two classifiers can be trained, each of which distinguishes one class from all other classes.

[0052] Through the regularized logistic regression model and soft margin support vector machine provided by the present invention, the output category is predicted given the input features, so as to better adapt to the complex scenarios in data governance and improve the generalization ability and prediction accuracy of the model. Through the method of the above embodiment, not only the nonlinear relationship and feature interaction of the features are taken into account, but also the robustness of the model is improved through regularization and soft margin.

[0053] In one embodiment, the steps of formulating unified data modeling may include: First, create a panoramic data asset map to achieve enterprise data asset management. The panoramic data asset map can be used to quickly find assets and analyze assets, achieve full asset coverage, and support global search and precise positioning of target assets.

[0054] Secondly, customize unified enterprise data standards, including fields, code values, and data dictionary management, to ensure unified standards for business data and middle-office data.

[0055] Then, create a data quality system before, during, and after the event to support the configuration and alarm management functions of the data quality monitoring model.

[0056] Finally, according to the enterprise data assets, configure data security desensitization, security classification and monitoring to achieve data security classification management of enterprise data assets; and conduct data access security behavior monitoring to identify access risks.

[0057] Unified data modeling can not only improve the efficiency and ease of use of data, but also support the company's data governance work to ensure the security, reliability and maximum value of data.

[0058] Further, the process proceeds to step S102: collecting enterprise-related data related to the enterprise's business goals and digital transformation, including constructing a specific association map between business goals and required data based on the enterprise's specific business indicators and sub-goals, and setting the data type, data source, and collection time node and frequency through the specific association map; wherein the enterprise-related data includes internal business data, production data, and external data of the enterprise; the data source of the external data of the enterprise includes but is not limited to one or more of the following: data released by industry associations, reports from market research institutions, government public data, social media platform data, and partner shared data.

[0059] In one embodiment, the business goals of the enterprise are deeply analyzed and broken down into specific business indicators and sub-goals. For example, if the business goal of the enterprise is to increase market share, it can be further broken down into sub-goals such as increasing product sales, expanding new customer groups, and optimizing customer retention. For each sub-goal, the data requirements closely related to it are sorted out. For example, increasing product sales may involve product sales data, marketing effect data, competitor product pricing data, etc. In this way, a detailed correlation map between business goals and required data is constructed, and the data categories that are valuable for digital transformation are accurately located.

[0060] Develop a comprehensive data collection plan based on business goals and data correlation analysis results. Determine the data types and data sources that are prioritized for collection, as well as the time nodes and frequency of collection. For data that is directly related to core business processes and has a greater impact on real-time decision-making, such as real-time order data of e-commerce companies, set a high-frequency collection plan to ensure the timeliness of the data; for some data that is used for long-term trend analysis and has a lower update frequency, such as industry annual report data, regular, relatively low-frequency collection tasks can be arranged.

[0061] In one embodiment, enterprise-related data related to the business objectives and digital transformation of the enterprise is collected, including: Determine the collection target based on the company's customer needs, market trends, product feedback, etc., so as to formulate more accurate business strategies and decisions. When determining the data collection target, the comprehensiveness, accuracy and real-time nature of the data should be considered.

[0062] According to the collection target, data is collected through one or more of the following methods: directly obtaining data from the data source; collecting enterprise on-site environmental data through various enterprise sensor devices; using distributed crawler systems to crawl and collect external network data.

[0063] For internal business data of an enterprise, collection includes database extraction: extracting business-related data information from the database of the relevant system within the enterprise, converting it into a unified format, and loading it into the enterprise data warehouse; interface collection: obtaining data by calling the API interfaces of various business systems within the enterprise and performing processing and analysis; file collection: extracting useful information from various files within the enterprise by collecting, parsing and processing them; tracking technology: adding tracking codes to the enterprise's applications to record user behaviors and operations in the applications. By collecting this data, we can gain an in-depth understanding of user behaviors and usage habits.

[0064] For production data collection, including sensor collection: by installing various sensor devices, collect real-time data such as on-site environmental data, and upload it to the company's internal data system for analysis and processing. Real-time data collection system: accurately collect, organize and deeply analyze various types of data in workshop production equipment, process flow and production activities, providing solid data support for the company's efficient operation.

[0065] External data sources of enterprises include but are not limited to data released by industry associations, reports from market research institutions, government public data, social media platform data, partner shared data, etc. When collecting external data sources, focus on the authority, relevance, coverage and cost of use of the data. For example, the annual industry statistics released by industry associations are highly authoritative and can be used to understand the overall development trend of the industry, but you may need to pay to obtain them; social media platform data can reflect consumer opinions and market hotspots in real time, but the data volume is large and messy, and needs to be effectively screened and sorted.

[0066] Further, the process proceeds to step S103: using the data quality monitoring model to check the data quality of the enterprise-related data to obtain valid data, including embedding data quality verification links in real time during the data collection process and regularly conducting data quality assessments.

[0067] In one embodiment, first, the enterprise-related data is cleaned and processed, including: outlier detection and processing, data standardization and normalization, data missing value processing, etc., so as to improve the integrity, accuracy and consistency of the data and provide high-quality data input for subsequent analysis.

[0068] Statistical methods can be used for outlier detection and processing, such as calculating the mean, standard deviation and other statistics of the data to identify data points that are significantly deviated from the normal range. For example, for numerical data, if a data point is beyond the range of the mean plus or minus several times the standard deviation, it may be an outlier. Clustering algorithms, such as K-Means clustering, can also be used to divide data points into different clusters. Data points that are far away from the main clusters may be outliers. For text data, text classification algorithms can be used to mark text that does not conform to normal text patterns or semantic logic as abnormal. Once an outlier is detected, it can be corrected, deleted, or individually marked for subsequent analysis according to business needs.

[0069] In the process of data standardization and normalization, for numerical data, standardization methods are used, such as converting the data into a distribution with a mean of 0 and a standard deviation of 1. Normalization maps the data to a specific interval, such as [0,1].

[0070] For missing values, you can use model-based methods, such as using machine learning algorithms (such as random forests, decision trees, etc.) to predict missing values. Taking random forest as an example, the data features containing missing values ​​are used as target variables, and other complete features are used as input variables to train the model, and then the trained model is used to predict missing values. You can also use the mean, median, or mode filling method. For numerical data, choose an appropriate filling value based on the data distribution. For example, if the data is approximately normally distributed, you can use the mean to fill; if the data is skewed, the median may be more appropriate. For categorical data, use the mode to fill.

[0071] Secondly, artificial intelligence algorithms are used to extract and select features from text data, image data, and structured data. This may include: For text data, word vector models in natural language processing technology, such as Word2Vec or GloVe, can be used to convert text into numerical vector representations. For example, Word2Vec can learn the distributed representation of words based on a text corpus, map each word to a low-dimensional vector space, and thus extract the semantic features of the text. For image data, a convolutional neural network (CNN) can be used for feature extraction. The convolution layer of CNN can automatically learn local features in an image, such as edges and textures, and convert the original image data into representative feature vectors through multi-layer convolution and pooling operations. For structured data, dimensionality reduction techniques such as principal component analysis (PCA) can be used to extract the main features. PCA decomposes the data covariance matrix to find the main component direction of the data, projects the high-dimensional data into a low-dimensional space, reduces the data dimension while retaining most of the information, and removes redundant features.

[0072] When selecting features, you can use correlation analysis methods, such as calculating the Pearson correlation coefficient, to measure the correlation between features and target variables (such as indicators related to corporate business goals). Delete features with low correlation and retain features that have a significant impact on the target variable. Use model-based feature selection methods, such as recursive feature elimination (RFE) combined with classification or regression models such as support vector machines (SVM). RFE starts with all features and gradually eliminates features that have the least impact on model performance until the preset number of features is reached or the model performance is no longer improved. In this way, the most valuable feature combination for the business needs of the enterprise is selected to obtain effective enterprise data.

[0073] Finally, a data quality assessment model is established, and the data quality monitoring model is used for data monitoring and feedback. This includes using a neural network or other machine learning algorithm to build a data quality assessment model. The cleaned, preprocessed, and feature-engineered data is used as input, and the data quality assessment indicators (such as accuracy, completeness, consistency, etc.) are used as output labels for training. For example, a multi-layer perceptron neural network can be used to input various feature statistics of the data (such as the mean, variance, missing value ratio, etc. of the data) and the relationship indicators between the features, and output a comprehensive score of the data quality.

[0074] In the process of data collection, data quality verification can be embedded in real time. For numerical data, set a reasonable range of values, such as the sales amount of enterprises cannot be negative; for date data, verify the correctness of its format and logical order to ensure the continuity of time series data; for text data, perform spelling checks, keyword matching and other verifications to ensure the accuracy of information. Use the verification rule library and automated verification scripts to conduct instant verification of each piece of collected data. Once a data quality problem is found, immediately mark and trigger the corresponding error correction mechanism, such as re-collection, data repair or notify the data source provider to make corrections.

[0075] Set a time period, conduct data quality assessment work regularly, and use sampling statistics, data comparison and other methods to comprehensively evaluate the accuracy, completeness, consistency and timeliness of collected data. For example, extract a certain percentage of sales data and compare it with financial accounting data every month to verify the consistency of the data; determine whether its timeliness meets business needs through the update frequency of statistical data. Based on the evaluation results, continuously optimize the data collection process and verification mechanism to continuously improve data quality.

[0076] Deploy the data quality assessment model into the enterprise data processing process to monitor the quality of newly collected and processed data in real time. When the data quality is lower than the set threshold, an alarm is automatically triggered and a detailed quality problem report is provided, such as which features are abnormal, the specific manifestations of insufficient data integrity, etc. Based on the feedback information, the data processing strategy is adjusted in a timely manner to further optimize the data processing process to ensure that effective enterprise data that meets the enterprise data characteristics and business needs is finally obtained.

[0077] In summary, according to the above embodiments provided by the present invention, by acquiring enterprise characteristic data and establishing a data quality monitoring model, data standardization management can be achieved, data quality and standardization can be improved, and the data source can be ensured to be reliable and the format is unified, thus laying a solid foundation for subsequent analysis.

[0078] Furthermore, by collecting multi-source enterprise data, data resource integration can be achieved, and internal business data, production data and external data of the enterprise can be fully aggregated to enrich the data dimensions and provide a comprehensive perspective for enterprise operations.

[0079] Furthermore, through data processing and quality monitoring, accurate data optimization can be achieved, outliers can be removed, missing values ​​can be filled, and a data quality assessment model can be established to ensure data accuracy and completeness, improve data quality control, and thus enhance data availability.

[0080] The above is a detailed description of an embodiment of the present invention, but the content is only a preferred embodiment of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent coverage of the present invention.

Claims

1. A full-cycle data management method for digital transformation of small and medium-sized enterprises, characterized in that: The following steps are involved: Acquire enterprise characteristic data and establish a data quality monitoring model; the data quality monitoring model includes a configuration function to provide configuration of monitoring standards through an interface, including selecting a data source, database, table or field, and configuring a custom monitoring standard; Collecting enterprise-related data related to the enterprise's business goals and digital transformation, including building a specific association map between business goals and required data based on the enterprise's specific business indicators and sub-goals, and setting the data type, data source, and time node and frequency of data collection through the specific association map; wherein the enterprise-related data includes internal business data, production data, and external data of the enterprise; the data source of the external data of the enterprise includes but is not limited to one or more of the following: data released by industry associations, reports from market research institutions, government public data, social media platform data, and partner shared data; The data quality monitoring model is used to check the data quality of the enterprise-related data to obtain valid data, including embedding a data quality verification link in real time during the data collection process and conducting data quality assessments regularly.

2. The full-cycle data management method according to claim 1, characterized in that: The establishing of the data quality monitoring model comprises: Establish an enterprise data governance framework and organizational structure based on the enterprise characteristics of the digital transformation, including defining the goals, scope, roles and responsibilities of data governance; Using machine learning algorithms, analyzing the enterprise's existing data standards and business needs to obtain new data standards that meet the enterprise's specific needs; and matching appropriate data standards for data fields when creating tables in the database, and matching appropriate data standards for existing data based on the enterprise's new data standards; the new data standards include field standards, code value standards, and dictionary management standards; Based on the new data standards, a multi-dimensional data quality inspection model is established, and the multi-dimensional data quality inspection model constitutes a data quality monitoring model; Among them, the data quality inspection models of multiple dimensions include: accuracy inspection model, compliance inspection model, completeness inspection model, timeliness inspection model, consistency inspection model, and availability inspection model.

3. The full-cycle data management method according to claim 2, characterized in that: The multi-dimensional data quality inspection model also includes a data security model and a data governance effect verification and evaluation model, wherein: The data security model includes data desensitization, data security classification and data security monitoring to ensure that data is not accessed or leaked without authorization; The data governance effectiveness verification and evaluation model includes the maximization of data assets, the integrity and traceability of data lineage; wherein the data lineage refers to the association relationship formed between data throughout the entire life cycle of the data.

4. The full-cycle data management method according to claim 2, characterized in that: The use of machine learning algorithms to analyze the company's existing data standards and business needs includes: Obtain historical data of the enterprise; Polynomial terms and interaction terms were introduced to construct regularized logistic regression models; Using the enterprise-related data, training the regularized logistic regression model to predict the category label to which the data feature of the historical data belongs; Using kernel techniques and introducing soft margins and regularization parameters, a soft margin support vector machine is constructed. According to the nonlinear data of the enterprise-related data, the soft margin support vector machine is trained to find the best separation hyperplane between different categories to determine the category labels of the data features of the historical data.

5. The full-cycle data management method according to claim 4, characterized in that: The step of training the regularized logistic regression model by using the enterprise-related data includes: To prevent overfitting, a regularization term is added to the loss function to construct a regularized logistic regression model, which is expressed as a log-likelihood function with the formula: ； in, , represents the probability of the model predicting the positive class; is the label of the i-th feature sample, which is the information used for supervision during model training; are input features, is the weight parameter, where is the intercept term, is the feature corresponding to The coefficient of is a regularization parameter to control the strength of the regularization.

6. The full-cycle data management method according to claim 4, characterized in that: The step of training the soft margin support vector machine according to the nonlinear data of the enterprise-related data comprises: The kernel technique maps data into a high-dimensional space to transform nonlinear problems into linear problems; A soft margin and a regularization parameter C are introduced into the soft margin support vector machine, wherein the regularization parameter C is used to control the strength of regularization.

7. The full-cycle data management method according to claim 1, characterized in that: The enterprise-related data collected in relation to the enterprise’s business objectives and digital transformation includes: Determine collection targets based on the company’s customer needs, market trends, and product feedback; According to the collection target, data is collected through one or more of the following methods: directly obtaining data from the data source; collecting enterprise on-site environmental data through various enterprise sensor devices; using distributed crawler systems to crawl and collect external network data.

8. The full-cycle data management method according to claim 1, characterized in that: The methods for collecting the internal business data of the enterprise include: Database extraction: extracting business-related data information from the database of the enterprise's internal related systems; Interface collection: by calling the API interfaces of various business systems within the enterprise, data is obtained and processed and analyzed; Document collection: Collect various documents within the enterprise, analyze and process them, and extract useful information from them; Tracking technology: Add tracking codes to the company's applications to record user behaviors and operations in the applications.

9. The full-cycle data management method according to claim 1, characterized in that: The using of the data quality monitoring model to check the data quality of the enterprise-related data to obtain valid data includes: Clean and process the data, including: outlier detection and processing, data standardization and normalization, and data missing value processing; Use artificial intelligence algorithms to extract and select features from text data, image data, and structured data; A data quality assessment model is established, and the data quality monitoring model is used to perform data monitoring and feedback.

10. The full-cycle data management method according to claim 9, characterized in that: The use of artificial intelligence algorithms to extract and select features from text data, image data, and structured data includes: For text data, the word vector model in natural language processing technology is used to convert the text into a numerical vector representation; For image data, convolutional neural networks are used to extract features and convert the original image data into representative feature vectors; For structured data, principal component analysis dimensionality reduction technology is used to extract the main features and project high-dimensional data into low-dimensional space.

Images