Data cleaning method and system based on neural network model

Abstract

The invention provides a data cleaning method and system based on a neural network model, and is applied to the technical field of data processing. The method comprises the following steps: processing data feature description information to generate a data cleaning influence factor; processing the original data set to generate preprocessed data sequence information; processing the preprocessed data sequence information on the basis of data use demand information of a user to generate target data parameter information; processing related information of a data source to generate a data prediction cleaning feature vector; processing a preset data cleaning neural network model based on the training set to generate a target data cleaning neural network model; and processing the data cleaning influence factor, the target data parameter information and the data prediction cleaning feature vector based on the target data cleaning neural network model to generate cleaned data.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data cleaning method and system based on a neural network model. Background Art

[0002] With the explosive growth of data and the increasing complexity of data application scenarios, high-quality data has become a key prerequisite for ensuring the accuracy and effectiveness of data analysis. However, the current data quality issues are prominent, bringing many challenges to data processing and application, which are specifically reflected in the following aspects: In many practical application scenarios, such as sales data on e-commerce platforms, there are often problems such as negative sales quantity, abnormal sales price, and confusing order time format; in the field of medical data, patient information may be missing or incorrectly recorded; financial data may also face data inconsistency and duplication. These problems seriously interfere with the accuracy of data analysis results, reduce data availability, and hinder the effective mining of data value. Traditional data cleaning methods are mostly based on simple rules or statistical methods, which are difficult to deal with complex and changeable data problems. Faced with large-scale, high-dimensional and complex data, traditional methods cannot accurately identify noise and errors in data, and are difficult to adapt to different application scenarios and user needs. For example, in social media data processing, due to the diverse data formats and complex semantics, traditional cleaning methods cannot effectively remove invalid information, resulting in poor data cleaning effects and unable to meet users' needs for in-depth data analysis. Different users have huge differences in their data usage requirements, and data cleaning needs to fully consider these requirements, while combining the characteristics and source information of the data itself. However, existing data cleaning solutions often cannot comprehensively and comprehensively deal with these factors, resulting in the difficulty of cleaned data to accurately meet user-specific needs.

[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0004] The purpose of this application is to provide a data cleaning method and system based on a neural network model, which at least to a certain extent overcomes the problems existing in the prior art by processing each type of information separately. Generate data cleaning influencing factors from data feature description information; process, convert and sample missing values ​​and outliers on the original data set to obtain pre-processed data sequence information; generate target data parameter information according to user needs; generate data prediction cleaning feature vectors from data source information; use the training set to train the preset model to obtain a target data cleaning neural network model. In the data cleaning stage, forward propagation calculations, feature fusion and decision processing are performed on relevant information based on the target model, and the target data parameter information is processed to obtain preliminary cleaning results, and then the cleaned data is generated through format conversion. Integrate multi-source information, use the learning ability of neural networks to effectively clean complex data, improve data quality, and lay a solid data foundation for subsequent analysis applications.

[0005] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present invention.

[0006] According to one aspect of the present application, a data cleaning method based on a neural network model is provided, comprising: obtaining an original data set, data feature description information, user usage demand information for data, relevant information information of the data source, a preset data cleaning neural network model and a training set, wherein the original data set is used to characterize a data set containing various noises and errors; processing the data feature description information to generate a data cleaning influencing factor; processing the original data set to generate preprocessed data sequence information; processing the preprocessed data sequence information based on the user usage demand information for data to generate target data parameter information; processing the relevant information information of the data source to generate a data prediction cleaning feature vector; processing the preset data cleaning neural network model based on the training set to generate a target data cleaning neural network model; processing the data cleaning influencing factor, the target data parameter information and the data prediction cleaning feature vector based on the target data cleaning neural network model to generate cleaned data.

[0007] Another aspect of the present application is a data cleaning device based on a neural network model, characterized in that it includes: an acquisition module, used to acquire an original data set, data feature description information, user usage demand information for data, relevant information information of the data source, a preset data cleaning neural network model and a training set, wherein the original data set is used to characterize a data set containing various noises and errors; a processing module, used to process the data feature description information to generate a data cleaning influencing factor; process the original data set to generate preprocessed data sequence information; process the preprocessed data sequence information based on the user usage demand information for data to generate target data parameter information; process the relevant information information of the data source to generate a data prediction cleaning feature vector; process the preset data cleaning neural network model based on the training set to generate a target data cleaning neural network model; process the data cleaning influencing factor, the target data parameter information and the data prediction cleaning feature vector based on the target data cleaning neural network model to generate cleaned data.

[0008] According to another aspect of the present application, an electronic device is characterized in that it includes: a first processor; and a memory for storing executable instructions of the first processor; wherein the first processor is configured to execute the above-mentioned data cleaning method based on the neural network model by executing the executable instructions.

[0009] According to another aspect of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a second processor, the above-mentioned data cleaning method based on the neural network model is implemented.

[0010] The present application provides a data cleaning method and system based on a neural network model. In the data acquisition stage, the original data set, data feature description information, user data usage demand information, relevant information about the data source, a preset data cleaning neural network model and a training set are collected. The original data set contains noise and errors, and other information supplements the description of the data from different dimensions. In the multi-information processing stage, each type of information is processed separately. Data cleaning influencing factors are generated from data feature description information; missing values ​​and outliers are processed, converted and sampled on the original data set to obtain pre-processed data sequence information; target data parameter information is generated according to user needs; data prediction cleaning feature vectors are generated from data source information; and the preset model is trained with the training set to obtain the target data cleaning neural network model. In the data cleaning stage, forward propagation calculation, feature fusion and decision processing are performed on the relevant information based on the target model, and the target data parameter information is processed to obtain preliminary cleaning results, and then the cleaned data is generated through format conversion. This method integrates multi-source information, uses the learning ability of neural networks to effectively clean complex data, improve data quality, and lay a solid data foundation for subsequent analysis applications.

[0011] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 A flowchart of a data cleaning method based on a neural network model provided by an embodiment of the present application is shown; Figure 2 A schematic structural diagram of a data cleaning device based on a neural network model provided in one embodiment of the present application is shown. DETAILED DESCRIPTION

[0013] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0014] Combine the following Figure 1 To describe the data cleaning method based on the neural network model according to the exemplary embodiment of the present application. It should be noted that the following application scenarios are only shown to facilitate understanding of the spirit and principles of the present application, and the embodiments of the present application are not limited in this regard. On the contrary, the embodiments of the present application are applied to any applicable scenario.

[0015] In one implementation, the present application also proposes a data cleaning method and system based on a neural network model. Figure 1 The following schematically shows a flow chart of a data cleaning method based on a neural network model according to an embodiment of the present application. Figure 1As shown, the method is applied to a server, comprising: S101, obtaining the original data set, data feature description information, user demand information for data use, relevant information about the data source, preset data cleaning neural network model and training set.

[0016] In one implementation, it is assumed that the sales data of an e-commerce platform is being processed. The sales records of the e-commerce platform in the past year cover data such as the sales quantity, sales price, order time, and customer information of the goods. However, there are many problems in the data. Some sales quantities have negative values, which may be data entry errors; the sales prices of some goods are significantly lower than the cost price, which are abnormal data; and some order time formats are chaotic, such as "2023 / 01-01", which does not conform to the conventional date format. These are noise and erroneous data. Regarding the characteristic description document of sales data, the meaning of each field is recorded, for example, "sales quantity" represents the actual number of goods sold, and "sales price" is the transaction price of the goods; in terms of data type, "sales quantity" and "sales price" are numerical data, "order time" is date data, and "customer information" contains text data (such as customer name and address); it also describes the association between data, such as one order corresponds to one customer, and the order time is associated with sales data, which can be used to analyze sales trends in different time periods.

[0017] The market analysis team hopes to understand the sales of various commodities in different seasons through these sales data in order to formulate targeted marketing strategies. They require the data to be summarized and counted on a quarterly basis, focusing on the two key indicators of sales volume (sales quantity × sales price) and sales volume, and have high requirements for data accuracy, giving priority to data cleaning related to key indicators. The sales data comes from the transaction system of the e-commerce platform, which collects and updates data on a regular basis every day. During the data collection process, some data is automatically entered when the user places an order, and some is supplemented by manual review. It is understood that the transaction system has undergone a version upgrade recently, which may have affected the accuracy of some data. At the same time, the data is stored in a relational database. The database architecture information shows that the value range of the "sales quantity" and "sales price" fields should theoretically be positive numbers. This information is of great reference value for evaluating data reliability and subsequent cleaning. The preset data cleaning neural network model uses a multilayer perceptron (MLP). The multilayer perceptron is a feedforward artificial neural network model consisting of an input layer, multiple hidden layers, and an output layer. It can handle complex nonlinear relationships and is suitable for identifying and correcting various noise and error patterns in data cleaning tasks.

[0018] The number of neurons in the input layer depends on the number of features of the input data. In the e-commerce sales data cleaning scenario, the input data includes data cleaning influencing factors, target data parameter information, and data prediction cleaning feature vectors. Assuming that the data cleaning influencing factors contain 5 features (such as the degree of data volatility, the proportion of missing values, etc.), the target data parameter information contains 3 features (such as indicators such as sales volume and sales volume that users are concerned about), and the data prediction cleaning feature vector contains 4 features (such as the reliability score of the data source, the stability of the data acquisition equipment, etc.), then the number of neurons in the input layer is 5+3+4=12. Each neuron receives a feature value as input. The number of hidden layers is set to 2. Multiple hidden layers allow the model to learn more complex feature representations and data patterns. The first hidden layer can extract some basic feature combinations from the input data, and the second hidden layer further abstracts and integrates features based on the first hidden layer to capture deeper data relationships.

[0019] Regarding the number of neurons, the first hidden layer is set to 20 neurons and the second hidden layer is set to 15 neurons. The choice of the number of neurons is an empirical value and needs to be adjusted according to the performance of the model in actual training. More neurons can increase the expressive power of the model, but may also lead to overfitting; fewer neurons may prevent the model from learning sufficiently complex patterns. The neurons in the hidden layer use the ReLU (RectifiedLinearUnit) activation function. The mathematical expression of ReLU is ,It has the advantages of simple calculation and fast convergence speed, which can effectively alleviate the gradient vanishing problem and make the model converge to the optimal solution faster.

[0020] The number of neurons in the output layer depends on the specific cleaning task. If the data is cleaned, such as correcting the sales quantity and sales price, the output layer can be set with 2 neurons, corresponding to the sales quantity and sales price after cleaning. If the data is classified and cleaned (such as marking normal data, abnormal data, and missing data), assuming it is divided into 3 categories, then the output layer is set with 3 neurons. If it is a numerical data cleaning, the output layer can directly output the predicted value without using an activation function; if it is a classification cleaning task, the output layer uses the Softmax activation function to convert the output value into a probability distribution, which is convenient for determining the category to which the data belongs.

[0021] In addition, the learning rate (LearningRate) is set to 0.001. The learning rate controls the step size of the model's parameter update in each iteration. A smaller learning rate can make the model converge to the optimal solution more stably, but the convergence speed is slower; a larger learning rate may cause the model to oscillate near the optimal solution or even fail to converge. 0.001 is a commonly used initial learning rate, which can be adjusted according to the convergence of the model during training. The number of iterations (Epochs) is initially set to 100. The number of iterations refers to the number of rounds that the model trains the entire training set. As the number of iterations increases, the performance of the model usually gradually improves, but when it reaches a certain number of times, overfitting may occur. Therefore, it is necessary to monitor the performance of the model through the validation set during training and select an appropriate number of iterations. The batch size (BatchSize) is set to 32. The batch size refers to the number of samples used to update the model parameters in each iteration. A larger batch size can take advantage of parallel computing and speed up training, but it may cause the model to fall into a local optimal solution; a smaller batch size can increase the randomness of the model and help it escape the local optimal solution, but the training speed is slower. 32 is a commonly used batch size that can achieve a good balance between speed and performance.

[0022] For numerical data cleaning, the mean square error (MSE) loss function is used. The calculation formula of MSE is ,in is the true value, is the model prediction value, and n is the number of samples. MSE can measure the average error between the model prediction value and the true value. By minimizing MSE, the model prediction value can be closer to the true value. For classification cleaning tasks, the cross-entropy loss function (Cross-EntropyLoss) is used, which can measure the difference between the probability distribution of the model output and the probability distribution of the true label.

[0023] A representative part of the sales data from the past of the e-commerce platform has been selected. These data have been manually labeled and cleaned, and the noise and error data and the correct processing methods have been identified. For example, it is marked which data with negative sales volume is wrong and how to correct it; which abnormal price data needs to be adjusted or deleted, etc. The training set contains sales data of different time periods and different types of goods, which is used to train the preset data cleaning neural network model so that it can learn the patterns and rules of data cleaning.

[0024] S102: Process the data feature description information to generate a data cleaning impact factor.

[0025] In one implementation, feature extraction and classification processing are performed on the data feature description information to generate attribute features, semantic features, and statistical features. Assume that the feature description information of the sales data of the e-commerce platform we obtain contains various descriptions of the data, such as "sales data covers the sales of different categories of goods in various regions and time periods, including sales quantity, sales price, order time, customer region and other information". The basic attributes of the data are extracted from the above description, such as the dimensions of the data (involving multiple dimensions such as product categories, regions, and time), data types (sales quantity and sales price are numerical, order time is date type, and customer region is text type), data storage format (assuming it is stored in a two-dimensional table in a relational database), etc. These attribute features are used to describe the basic characteristics and storage methods of the data.

[0026] Mining the semantic information of data, for example, the sales quantity represents the actual number of goods sold, and the sales price represents the transaction price of the goods. Through this information, we can understand the business meaning represented by the data. We can also further extract the semantic relationship between the product category and the sales quantity and price. For example, the prices of some high-end categories of goods are generally high, and the sales quantity is relatively low; the sales quantity of seasonal goods will increase significantly in a specific time period, etc. These semantic features help us understand the business logic behind the data. Perform statistical analysis on sales data to obtain statistical features. Calculate statistics such as the mean, median, and standard deviation of sales quantity and sales price to describe the central tendency and dispersion of data. Count the total sales in different regions, the number of orders in different time periods, etc. These statistical features can reflect the overall situation of data in different dimensions and help discover potential patterns and anomalies in the data.

[0027] Quantitatively encode and fuse the attribute features, semantic features, and statistical features to generate numerical vector information. Process the extracted attribute features, semantic features, and statistical features so that they can be effectively processed by machine learning algorithms. For attribute features, quantify and encode the data type, such as encoding the numeric type as 1, the date type as 2, and the text type as 3. For data dimension information, it can be encoded according to the number of dimensions. If three dimensions are involved, it is encoded as 3. For semantic features, natural language processing technology is used to convert textual semantic information into numerical vectors. For example, semantic words such as product categories, sales quantities, and prices are converted into vector representations of fixed dimensions using word vector models (such as Word2Vec). For statistical features, their numerical form remains unchanged, but they are standardized to convert statistical quantities such as mean, median, and standard deviation into a unified scale. For example, the Z-score standardization method is used to convert the data into a standard normal distribution with a mean of 0 and a standard deviation of 1. Fusion is performed on the quantified encoded attribute features, semantic features, and statistical features. Different types of feature vectors can be connected in sequence by splicing to form a comprehensive numerical vector information. For example, the attribute feature vector, semantic feature vector and statistical feature vector are first arranged in sequence, and finally a numerical vector containing all feature information is obtained. This vector integrates the various characteristics of the data and provides a comprehensive data basis for the subsequent generation of data cleaning influencing factors.

[0028] Based on the machine learning algorithm, the numerical vector information is processed to generate the data cleaning impact factor. The fused numerical vector information is processed using the random forest algorithm. Random forest is an ensemble learning algorithm composed of multiple decision trees. The numerical vector information is used as the input of the random forest algorithm, and the relationship between different features and data cleaning is learned through the training model. During the training process, the random forest algorithm automatically selects the features that have a greater impact on data cleaning and calculates the importance of each feature. For example, after training, it is found that the standard deviation of the sales price and the semantic relationship between the commodity category and the sales quantity are of high importance in judging whether the data is abnormal (such as abnormal price fluctuations, sales quantity and category inconsistency, etc.). Based on the calculation results of the model, a comprehensive numerical value is generated as the data cleaning impact factor. This impact factor can reflect the complexity of the data, the potential degree of abnormality, etc. The higher the impact factor, the more problems there may be in the data, and more attention and in-depth processing are needed during the data cleaning process; the lower the impact factor, the data is relatively "clean" and the cleaning difficulty is less.

[0029] S103, processing the original data set to generate pre-processed data sequence information.

[0030] In one implementation, the original data set is processed to generate data missing information and data anomaly information, wherein the original data set includes numerical data and non-numerical data. The original e-commerce platform sales data covers information such as the sales quantity, sales price, order date, customer ID, and product category of goods, wherein the sales quantity and sales price are numerical data, and the order date, customer ID, and product category are non-numerical data. By checking the original data set, it is found that the sales price information is missing in some orders, for example, the sales price field of 50 orders is empty; some orders are missing customer IDs, involving a total of 30 orders. These missing information are recorded to form data missing information. In numerical data, the sales quantity appears negative, such as the sales quantity of a certain product is -3, which is obviously not in line with the actual situation; the sales price is also abnormal, such as the sales price of a high-end product is only 1 yuan, which is far lower than its normal selling price. In terms of non-numerical data, the order date has an unreasonable format, such as "2023 / 15 / 32", and the customer ID has repeated values. The location and abnormality of these abnormal data are recorded to form data anomaly information.

[0031] Process missing data information and generate missing value filling information. For the case of missing sales prices, since sales prices are related to factors such as product category and sales time, the average sales price of similar products in similar time periods can be used to fill in missing values. For example, the product with a missing sales price is "smart watch". By querying the sales prices of smart watches with similar brands and functions in the same month, the average price is calculated to be 1,500 yuan, and this value is used as the filling information for missing values. For orders with missing customer IDs, if other information in the order (such as delivery address, purchase product combination, etc.) can be associated with the customer ID of other orders, the association is supplemented; if the association cannot be made, a temporary ID is generated according to the business rules, such as "temp_001" or "temp_002" as filling information.

[0032] Process the data anomaly information and generate anomaly processing information. For anomaly values ​​with negative sales quantity, if it is caused by returns after verification, adjust the sales quantity to a positive number and indicate in the remarks that it is a return order; if the cause cannot be determined and it is obviously an entry error, estimate a reasonable value based on the past sales data of the product and the sales of similar products for correction, such as correcting -3 to 3. For abnormally low sales prices, if it is confirmed that it is caused by promotional activities, supplement relevant information about the promotional activities; if it is an entry error, refer to the cost price of the product, the normal market price, and the fluctuation range of past sales prices, and correct it to a reasonable price, such as correcting 1 yuan to 1,000 yuan. For data with abnormal order date format, adjust it according to the correct date format specification, such as correcting "2023 / 15 / 32" to "2023 / 05 / 31" (assuming it is caused by an entry error). For duplicate customer IDs, further verify the order information, merge duplicate orders or correct one of the IDs to the correct unique identifier. Record the processing methods of these anomalies to form anomaly processing information.

[0033] The original data set is processed based on the missing value filling information and the outlier processing information to generate the target data set. The original data set is processed based on the missing value filling information and the outlier processing information. The filled sales price and customer ID information are updated to the corresponding order records, and the outlier data is corrected according to the outlier processing information. After these operations, a relatively complete and accurate data set is obtained, namely the target data set. In this target data set, the completeness and accuracy of the sales data are improved, and the interference caused by missing and anomalies is reduced.

[0034] Perform data conversion and data sampling on the target data set to generate preprocessed data sequence information. Standardize the numerical data in the target data set, such as normalizing the sales quantity and sales price, and mapping them to the [0,1] interval for subsequent analysis and model processing. For non-numerical data, use the unique hot encoding method for conversion. For example, if there are three categories of goods, "electronic products", "clothing" and "food", "electronic products" is encoded as [1,0,0], "clothing" is encoded as [0,1,0], and "food" is encoded as [0,0,1]. The order date extracts features such as year, month, and week, and encodes them separately. Since the amount of sales data on the e-commerce platform is large, data sampling can be performed to improve processing efficiency. Assuming that the original data set has 1 million order records, a random sampling method is used to extract 100,000 order records as samples at a ratio of 10%. During the sampling process, ensure that the data of various types of goods, different time periods, and various types of customers have a certain proportion of coverage to ensure the representativeness of the sample. After data conversion and data sampling, the data are arranged in a certain order (such as in chronological order of order date) to generate preprocessed data sequence information, providing a high-quality data foundation for the subsequent generation of target data parameter information based on user needs.

[0035] S104: Process the preprocessed data sequence information based on the user's data usage requirement information to generate target data parameter information.

[0036] In one implementation, the user's demand information for data usage is processed to generate data application scenario information, target data granularity information, target data indicator information, and data processing priority for target users. Assume that the market analysis team of an e-commerce platform has proposed data usage requirements to the data processing department, hoping to gain an in-depth understanding of the sales of various types of goods in different seasons in order to formulate accurate marketing strategies. According to the team's needs, the application scenario of the data is clearly defined as formulating seasonal marketing strategies. This means that data processing should focus on how to provide strong support for marketing strategies, such as analyzing changes in consumer purchasing preferences in different seasons and sales trends of different goods in each season, so as to determine the key promotional products and promotional activities for each season.

[0037] Considering the need to analyze the sales of various commodities in different seasons, the target data granularity is determined to be statistically analyzed by season and commodity category. That is, it is necessary to count the sales data of various commodities (such as clothing, electronic products, food, etc.) in each season (spring, summer, autumn, winter). This granularity can meet the needs of the market analysis team for analyzing the sales trends of commodities in different seasons and help discover seasonal consumption patterns. The core indicators that the market analysis team focuses on are sales, sales volume, and gross profit margin. Sales can directly reflect the income brought by commodity sales, sales volume reflects the market demand for commodities, and gross profit margin reflects the profit margin of commodities. By analyzing these indicators, the team can judge the sales benefits of different commodities in each season and provide a quantitative basis for the formulation of marketing strategies. Since the work of the market analysis team is crucial to the business development of the e-commerce platform, and they need to obtain analysis results in a short time to formulate the marketing strategy for the next quarter, the data processing priority of the target user (market analysis team) is set to high. This means that in the process of data processing, the needs of this team should be met first to ensure the timeliness and accuracy of data processing.

[0038] Based on the data application scenario information, target data granularity information, target data indicator information and data processing priority of the target user, the preprocessed data sequence information is matched and filtered to generate the target data sequence information. The preprocessed data contains a wealth of sales records, such as order date, product name, sales quantity, sales price, etc. According to the target data granularity requirements, the order date is matched with the season. For example, orders from March to May are marked as spring data, and orders from June to August are marked as summer data. At the same time, the product name is matched with the product category, such as "T-shirt" is matched to the clothing category, and "mobile phone" is matched to the electronic product category. According to the target data indicator information and data application scenario, the data that meets the requirements is filtered out. Only data fields related to sales, sales volume and gross profit margin calculation are retained, such as sales quantity, sales price, product cost (used to calculate gross profit margin), etc. At the same time, according to the data application scenario, some special orders are excluded, such as test orders, gift orders, etc., because these orders do not reflect the actual sales situation and will interfere with the market analysis results. After matching and screening, we get the sales data subsets classified by season and product category, that is, the target data sequence information. For example, we get the sales record subsets of spring clothing products and summer electronic products products.

[0039] Process the target data sequence information to generate target data parameter information. For each target data sequence (such as a subset of sales records for spring clothing products), calculate sales, sales volume, and gross profit margin. Sales are obtained by multiplying the sales quantity by the sales price; sales volume directly counts the total number of quantities in the sales records; and gross profit margin is calculated according to the formula (sales - total cost of goods) ÷ sales × 100%. The calculated sales, sales volume, and gross profit margin of each type of goods in each season are summarized and sorted to form structured data. For example, it is presented in a table form, with rows representing different categories of goods, columns representing different seasons, and the data in the table representing the corresponding sales, sales volume, and gross profit margin. These calculated and sorted data are the target data parameter information. The market analysis team can directly use these parameters to conduct in-depth market analysis, such as drawing sales trend charts for each season, comparing the gross profit margin differences of different product categories, and so on, and then formulate more targeted seasonal marketing strategies.

[0040] S105, processing the relevant information of the data source to generate a data prediction cleaning feature vector.

[0041] In one implementation, data extraction and processing are performed on relevant information of the data source to generate data structure information, wherein the data structure information includes field meaning, data type, and association information between data. Assume that the sales data of the e-commerce platform comes from multiple channels, including online store transaction systems and offline store cashier systems, and these data are finally aggregated into the data warehouse. We extract and process the relevant information of the data source. Obtain information from the data dictionary or relevant technical documents to clarify the meaning of each field. For example, "order number" is used to uniquely identify each order; "product ID" corresponds to the specific product sold; "sales quantity" indicates the number of products sold in the order; "sales amount" is the total sales of the order; "transaction time" records the time when the order occurs; "store ID" is used to distinguish different online stores or offline stores.

[0042] Determine the data type of each field. "Order number" and "Product ID" are usually string types; "Sales quantity" and "Sales amount" are numeric types (such as integer or floating point); "Transaction time" is date and time type; "Store ID" can be string type or integer type, depending on the encoding method of the system. Analyze the internal connection between data. An "Order number" corresponds to a unique order, which contains one or more "Product IDs", that is, the order and the product are in a one-to-many relationship; each order is associated with a specific store through the "Store ID"; "Sales quantity" and "Sales amount" are closely related to the "Product ID" and are used to record the sales of the product in the order; "Transaction time" is associated with orders and stores, and can be used to analyze the sales trends of different stores at different times. This information constitutes data structure information.

[0043] The data structure information is processed to generate data source information and data reliability information. After sorting out the source of data generation, it is found that the online store transaction system automatically generates sales data through users placing orders on the e-commerce platform, while the offline store records sales data through the cash register system and then uploads it to the data warehouse regularly. Data from different sources differ in data format, collection method and update frequency. Online data has strong real-time performance and can basically reflect the order status in real time; offline data may have a certain lag due to upload delays. Evaluate the reliability of data. The data collection of the online store transaction system depends on a stable network environment and a mature transaction platform. The data accuracy is high, but it may be affected by factors such as network failures and system vulnerabilities; the offline store cash register system may cause data deviations due to manual operation errors (such as scanning errors and amount entry errors). Through the analysis of historical data, it is found that the error rate of online data is about 1%, and the error rate of offline data is about 3%. At the same time, the integrity of the data is checked and it is found that some fields of offline data are occasionally missing, such as incomplete transaction time records. This information constitutes data reliability information.

[0044] Based on the data source information and data reliability information, the relevant information of the data source is processed to generate an initial feature vector, where the initial feature vector is used to characterize the key attributes of the data source, the reliability of the data, and their correlation characteristics with the data prediction cleaning needs. Consider key attributes such as the type of data source (online or offline), the collection method (automatic collection or manual entry), and the update frequency (real-time, scheduled or irregular). Encode the online data source as 1 and the offline as 2; automatic collection as 1 and manual entry as 2; real-time update as 3, scheduled update as 2, and irregular update as 1. Take data error rate and integrity into consideration. Low error rate (such as 1% error rate of online data) is coded as 3, medium error rate (such as 3% error rate of offline data) is coded as 2, and high error rate is coded as 1; high data integrity (no missing fields) is coded as 3, occasional small number of missing fields are coded as 2, and frequent large number of missing fields are coded as 1. Analyze the impact of data source and reliability on data prediction cleaning. For example, although the error rate of online data is low, abnormal data may appear due to system upgrades or network fluctuations, so it is necessary to focus on data consistency; offline data is more concerned about data accuracy and completeness due to manual operation factors. According to these correlation characteristics, different influencing factors are assigned corresponding weights. These encoding and weight information are combined into a vector, such as [1,2,3,2,2,0.5,0.3,0.2]. This vector is the initial feature vector, which comprehensively represents the key attributes of the data source, the reliability of the data, and their correlation characteristics with the data prediction and cleaning needs.

[0045] The initial feature vector is subjected to correlation analysis and dimensionality reduction processing to generate a data prediction cleaning feature vector. Statistical methods (such as Pearson correlation coefficient) are used to calculate the correlation between the dimensions in the initial feature vector. It is found that there is a high correlation between the data source type and the collection method, because most online data is collected automatically, and offline data is mostly manually entered; there is also a certain correlation between the data error rate and integrity, and data with a high error rate often has poor integrity. Based on the results of the correlation analysis, dimensionality reduction techniques such as principal component analysis (PCA) are used to reduce the dimension of the feature vector while retaining the original information as much as possible. After PCA processing, the initial 8-dimensional feature vector is reduced to 5 dimensions to obtain the data prediction cleaning feature vector. This vector not only retains information that is important for data prediction cleaning, but also reduces the complexity of data processing, and improves the efficiency and accuracy of subsequent data cleaning work. For example, the vector after dimensionality reduction is [0.8, 1.5, 2.2, 1.0, 1.8], which can be used as the input feature of the data cleaning model in the future to help the model better predict and process noise and errors in the data.

[0046] S106, processing the preset data cleaning neural network model based on the training set to generate a target data cleaning neural network model.

[0047] In one implementation, any number of data features in a training set are obtained. The training set is a representative data set selected from the past sales data of the e-commerce platform, including multiple data features such as product name, sales quantity, sales price, order time, customer ID, etc. Suppose we randomly selected the three data features of "sales quantity", "sales price" and "order time". These three features reflect the sales quantity, revenue and time dimension information respectively, which are of great significance for the training data cleaning neural network model to identify anomalies and errors in sales data.

[0048] Based on the number of each data feature in the training set, a sampling ratio is generated, and the training set is processed based on the sampling ratio to generate a preset number of sampling features. In the training set, there are 1000 data points for the "sales quantity" feature, 1000 data points for the "sales price" feature, and 1000 data points for the "order time" feature. Suppose we want to generate 200 sampling features. Then, for the "sales quantity" feature, the sampling ratio is 200÷1000=0.2; the sampling ratio of the "sales price" feature is also 0.2; the sampling ratio of the "order time" feature is also 0.2. This sampling ratio will be used to extract data from the original training set to generate sampling features. According to the calculated sampling ratio, each selected data feature in the training set is sampled. For the "sales quantity" feature, 200 data points are randomly selected from 1000 data points at a sampling ratio of 0.2 to obtain a set of sampling features about the sales quantity. Similarly, the same operation is performed on the "sales price" and "order time" features to obtain 200 sampling features for each. These sampled features constitute a new feature set for subsequent processing, which are representative data subsets selected from the original training set.

[0049] Based on any data feature and each sampling feature, multiple data groups are generated, wherein each data group contains a target number of data samples, and at least one data sample includes identification information. Taking the data feature of "sales quantity" as an example, it is combined with the sampling features of "sales price" and "order time". It is assumed that each data group contains 50 data samples, and at least one data sample in each data group contains identification information (such as order number, which is used to uniquely identify this sales order). Data is selected from the 200 sampling data points of "sales quantity" in sequence and combined with the corresponding sampling data points of "sales price" and "order time". For example, the first data group contains the 1st to 50th sampling data of "sales quantity", the 1st to 50th sampling data of "sales price" and the 1st to 50th sampling data of "order time", wherein the order number of the 10th data sample is used as identification information. In this way, multiple data groups are generated, which contain sales data samples with different data feature combinations for training data cleaning neural network models.

[0050] The preset data cleaning neural network model is trained based on the data samples in the multiple data sets to generate a trained data cleaning neural network model. The generated multiple data sets are used as input to train the preset data cleaning neural network model. During the training process, the model will learn the relationship between different data features and the pattern of normal sales data. For example, the model will learn that within a certain period of time, the sales quantity and sales price of a certain type of goods are usually within a certain range. If it exceeds this range, it may be abnormal data. As the training progresses, the model continuously adjusts its own parameters (such as weights and biases in the neural network) to improve the ability to identify anomalies and errors in sales data. After multiple rounds of training, the trained data cleaning neural network model is obtained.

[0051] The trained data cleaning neural network model is processed based on the test sample set to generate test results. If the data sample containing identification information in the test result is a risk factor that represents the risk factor affecting speech recognition classification, the trained data cleaning neural network model is used as the target data cleaning neural network model. Prepare a test sample set, which also comes from the sales data of the e-commerce platform, but is different from the data in the training set. Input the test sample set into the trained data cleaning neural network model for processing. The model analyzes the sales data in the test sample set, determines whether each data sample has an anomaly or error, and outputs the corresponding results. For example, the model may mark some data samples with abnormally high sales volume or abnormally low sales price, as well as data samples whose order time does not conform to the normal sales law. In the test results, check the data samples containing identification information (order number). Assume that we pre-set that if the sales volume of a data sample exceeds 5 times the average historical sales volume of similar products, or the sales price is lower than 50% of the cost price, the data sample is considered to be a risk factor affecting the accuracy of sales data (analogous to the risk factors affecting speech recognition classification in the original text). If the data sample containing identification information in the test result meets the definition of these risk factors, it means that the model can effectively identify abnormal data. When the model can accurately identify the risk factor data samples in the test sample set, the trained data cleaning neural network model is used as the target data cleaning neural network model to clean the actual e-commerce platform sales data. If the model cannot accurately identify the risk factor data samples, it is necessary to adjust the model parameters, increase the training data or improve the model structure, and then retrain and test until the model meets the requirements.

[0052] S107: Process the data cleaning influencing factors, the target data parameter information and the data prediction cleaning feature vector based on the target data cleaning neural network model to generate cleaned data.

[0053] In one embodiment, the data cleaning influence factor and the data prediction cleaning feature vector are forward propagated and processed based on the target data cleaning neural network model to generate patterns and regularity information related to data cleaning. It is assumed that a trained target data cleaning neural network model, as well as the corresponding data cleaning influence factor and data prediction cleaning feature vector have been obtained. After the previous calculation, the data cleaning influence factor reflects the complexity of the data itself, the potential abnormality, and other information. For example, if the data cleaning influence factor obtained by the random forest algorithm has a high value, it means that there may be more abnormal or complex situations in the data. The data prediction cleaning feature vector integrates the key attributes of the data source, the reliability, and other information, such as [0.8, 1.5, 2.2, 1.0, 1.8]. These data are input into the target data cleaning neural network model for forward propagation calculation. The input layer of the model receives the data cleaning influence factor and the data prediction cleaning feature vector, and then passes them to the hidden layer. The neurons in the hidden layer perform weighted summation on the input data and perform nonlinear transformation through an activation function (such as a ReLU function). In this process, the model gradually extracts features from the data. For example, it may discover the fluctuation pattern of the sales volume of certain commodities in the sales data within a specific time period, as well as the potential connection between the reliability of the data source and data anomalies. After processing through multiple hidden layers, the model finally outputs information about patterns and regularities related to data cleaning. For example, the model finds that whenever the reliability score of a data source is low, the probability of abnormal fluctuations in the sales price in the source data is higher; it also finds that the sales volume of certain commodities will show a periodic change pattern in a specific season. If data that does not conform to this pattern appears, it may be abnormal data.

[0054] Based on the target data cleaning neural network model, the data cleaning influencing factors, target data parameter information and data prediction cleaning feature vector are subjected to feature fusion and decision processing to generate data cleaning decisions. After obtaining the pattern and regularity information related to data cleaning, the model will perform feature fusion on the data cleaning influencing factors, target data parameter information and data prediction cleaning feature vector. The target data parameter information contains key indicator data generated according to user needs, such as sales, sales volume, gross profit margin, etc. The model will comprehensively consider the information from these different sources to determine whether the data needs to be cleaned and how to clean it. For example, when the data cleaning influencing factors show a high degree of data anomaly and the data prediction cleaning feature vector indicates that the reliability of the data source is low, the model will focus on the sales and sales volume data in the target data parameter information. If it is found that the sales of a certain type of goods have dropped significantly in a certain quarter, and the sales volume does not conform to the normal seasonal fluctuation pattern, combined with the previously obtained pattern and regularity information, the model may make a data cleaning decision: mark these data as suspicious data and further check the accuracy of the data. The decision may include operations such as revising the sales quantity and sales price, or further verifying the source of the data.

[0055] Based on the data cleaning related patterns and regular information and data cleaning decisions, the target data parameter information is processed to generate preliminary data cleaning results. Assume that the data cleaning decision is to check and correct the sales data of a certain type of goods in a certain quarter. By referring to the data cleaning related patterns and regular information, it is found that the sales price of a certain brand of clothing in the spring sales data is generally low, and the sales volume fluctuates abnormally. Combined with the market situation and the brand's past sales data, it is judged that these data may be affected by data entry errors or abnormal promotional activities. In response to this situation, the model will adjust the target data parameter information based on the previously learned patterns and regularities, as well as the decision results. For example, the sales price is corrected to a reasonable range, and the sales volume and sales amount are recalculated to obtain more accurate data. After these processes, the preliminary data cleaning results are generated. For example, the original sales of a certain brand of clothing in the spring were 100,000 yuan. After correcting the sales price and sales volume, the preliminary data cleaning results show that the sales should be 150,000 yuan, and the sales volume has also been adjusted accordingly. At the same time, the relevant data are marked to indicate that the data has been cleaned.

[0056] Perform format conversion on the preliminary data cleaning results to generate cleaned data. The preliminary data cleaning results may need to be format converted to meet the user's usage needs or the requirements of subsequent data analysis tools. Assume that the market analysis team of the e-commerce platform requires a specific table format, including fields such as product category, season, sales, sales volume, gross profit margin, and the data type and format are clearly specified. For the preliminary data cleaning results, first adjust the data type as required, such as converting the data type of sales and sales volume to the specified numeric type to ensure that the data accuracy meets the requirements. Then organize the data into the specified table format, and accurately fill in the corresponding fields with information such as product category and season. After format conversion, the cleaned data is obtained. For example, the previously corrected spring sales data of a certain brand of clothing is organized into the following table format: In this way, the cleaned data can be directly provided to the market analysis team for in-depth market analysis and formulation of marketing strategies, completing the entire data cleaning process.

[0057] In this application, the server obtains multiple types of data, models, and training sets, including original data sets, data feature description information, etc. The original data set contains noise and errors, and other information describes data characteristics, user needs, source conditions, etc. from different dimensions. Then, each type of information is processed separately. The data feature description information is processed through feature extraction, quantization coding, and machine learning algorithms to generate data cleaning influencing factors; the original data set is processed through generating missing and abnormal information, processing missing values ​​and abnormal values, data conversion, and sampling to obtain pre-processed data sequence information; based on user demand information, relevant scenarios, granularity, indicator information, and processing priorities are generated to obtain target data parameter information; structural information is extracted from data source information, the source and reliability are evaluated, and data prediction cleaning feature vectors are generated through correlation analysis and dimensionality reduction; the preset model is processed using the training set, and the target data cleaning neural network model is determined based on the test results.

[0058] Finally, based on the target model, forward propagation calculation, feature fusion and decision processing are performed on the factors affecting data cleaning. The target data parameter information is processed according to the processing results to obtain preliminary cleaning results, and then the format is converted to generate cleaned data. Integrate multi-source data information, use the powerful learning ability of the neural network model, comprehensively analyze data characteristics, user needs and source conditions, realize effective cleaning of complex data, improve data quality, and provide a reliable data foundation for subsequent data analysis and application.

[0059] In one embodiment, if Figure 2 As shown, the present application also provides a data cleaning device based on a neural network model, comprising: The acquisition module 201 is used to acquire the original data set, data feature description information, user demand information for data use, relevant information about the data source, a preset data cleaning neural network model and a training set, wherein the original data set is used to represent a data set containing various noises and errors; The processing module 202 is used to process the data feature description information to generate a data cleaning influencing factor; process the original data set to generate pre-processed data sequence information; process the pre-processed data sequence information based on the user's data usage demand information to generate target data parameter information; process the relevant information of the data source to generate a data prediction cleaning feature vector; process the preset data cleaning neural network model based on the training set to generate a target data cleaning neural network model; process the data cleaning influencing factor, the target data parameter information and the data prediction cleaning feature vector based on the target data cleaning neural network model to generate cleaned data.

[0060] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the data cleaning method based on the neural network model provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0061] Each embodiment in the present application is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the evaluation of the data cleaning method based on the neural network model, the electronic device, the electronic device, and the readable storage medium embodiment, since they are basically similar to the above-mentioned data cleaning method based on the neural network model, the description is relatively simple, and the relevant parts can be referred to the partial description of the embodiment of the data cleaning method based on the neural network model.

Claims

1. A data cleaning method based on a neural network model, characterized in that: include: Obtain the original data set, data feature description information, user demand information for data use, relevant information about the data source, preset data cleaning neural network model and training set, where the original data set is used to characterize the data set containing various noises and errors; Processing the data feature description information to generate a data cleaning impact factor; Processing the original data set to generate preprocessed data sequence information; Processing the preprocessed data sequence information based on the user's use requirement information for the data to generate target data parameter information; Processing relevant information of the data source to generate a data prediction cleaning feature vector; Processing the preset data cleaning neural network model based on the training set to generate a target data cleaning neural network model; The data cleaning influencing factors, the target data parameter information and the data prediction cleaning feature vector are processed based on the target data cleaning neural network model to generate cleaned data.

2. The method according to claim 1, characterized in that The data feature description information is processed to generate data cleaning impact factors, including: Extracting and classifying the data feature description information to generate attribute features, semantic features and statistical features; Performing quantization coding and feature fusion processing on the attribute features, the semantic features and the statistical features to generate numerical vector information; The numerical vector information is processed based on a machine learning algorithm to generate a data cleaning impact factor.

3. The method according to claim 1, characterized in that Processing the original data set to generate preprocessed data sequence information includes: Processing the original data set to generate data missing information and data abnormality information, wherein the original data set includes numerical data and non-numerical data; Processing the data missing information to generate missing value filling information; Processing the data abnormality information to generate abnormal value processing information; Processing the original data set based on the missing value filling information and the outlier processing information to generate a target data set; Perform data conversion and data sampling processing on the target data set to generate preprocessed data sequence information.

4. The method according to claim 3, characterized in that The pre-processed data sequence information is processed based on the user's use requirement information for the data to generate target data parameter information, including: Processing the user's data usage demand information to generate data application scenario information, target data granularity information, target data indicator information and data processing priority of the target user; Performing data matching and screening processing on the pre-processed data sequence information based on the data application scenario information, the target data granularity information, the target data indicator information and the data processing priority of the target user to generate target data sequence information; The target data sequence information is processed to generate target data parameter information.

5. The method according to claim 1, characterized in that Process the relevant information of the data source to generate a data prediction cleaning feature vector, including: Performing data extraction and processing on the relevant information of the data source to generate data structure information, wherein the data structure information includes field meaning, data type, and association information between data; Processing the data structure information to generate data source information and data reliability information; Based on the data source information and the data reliability information, relevant information of the data source is processed to generate an initial feature vector, wherein the initial feature vector is used to characterize key attributes of the data source, the reliability of the data, and their correlation characteristics with data prediction and cleaning requirements; The initial feature vector is subjected to correlation analysis and dimensionality reduction processing to generate a data prediction cleaning feature vector.

6. The method according to claim 1, characterized in that The preset data cleaning neural network model is processed based on the training set to generate a target data cleaning neural network model, including: Get any number of data features in the training set; Generate a sampling ratio based on the number of each data feature in the training set; Processing the training set based on the sampling ratio to generate a preset number of sampling features; Processing based on any data feature and each sampling feature to generate multiple data groups, wherein each data group includes a target number of data samples and at least one data sample includes identification information; Training the preset data cleaning neural network model based on data samples in the multiple data groups to generate a trained data cleaning neural network model; Processing the trained data cleaning neural network model based on the test sample set to generate a test result; If the data sample containing identification information in the test result represents a risk factor affecting speech recognition classification, the trained data cleaning neural network model is used as the target data cleaning neural network model.

7. The method according to claim 1, characterized in that Processing the data cleaning influencing factor, the target data parameter information and the data prediction cleaning feature vector based on the target data cleaning neural network model to generate cleaned data includes: Based on the target data cleaning neural network model, forward propagation calculation processing is performed on the data cleaning influencing factor and the data prediction cleaning feature vector to generate pattern and regularity information related to data cleaning; Based on the target data cleaning neural network model, feature fusion and decision processing are performed on the data cleaning influencing factor, the target data parameter information and the data prediction cleaning feature vector to generate a data cleaning decision; Processing the target data parameter information based on the data cleaning related pattern and regularity information and the data cleaning decision to generate a preliminary data cleaning result; The preliminary data cleaning results are formatted and processed to generate cleaned data.

8. A data cleaning device based on a neural network model, characterized in that: The device comprises: The acquisition module is used to obtain the original data set, data feature description information, user demand information for data use, relevant information about the data source, the preset data cleaning neural network model and the training set, wherein the original data set is used to characterize the data set containing various noises and errors; A processing module is used to process the data feature description information to generate a data cleaning influencing factor; process the original data set to generate preprocessed data sequence information; process the preprocessed data sequence information based on the user's data usage demand information to generate target data parameter information; process the relevant information of the data source to generate a data prediction cleaning feature vector; process the preset data cleaning neural network model based on the training set to generate a target data cleaning neural network model; process the data cleaning influencing factor, the target data parameter information and the data prediction cleaning feature vector based on the target data cleaning neural network model to generate cleaned data.

9. An electronic device, characterized in that: include: a first processor; and a memory for storing executable instructions of the first processor; Wherein, the first processor is configured to execute the data cleaning method based on the neural network model described in any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by the second processor, the data cleaning method based on the neural network model described in any one of claims 1 to 7 is implemented.

Images