Method and device for improving integrity of data assets, terminal and medium

Abstract

The invention discloses a method and device for improving the integrity of data assets, a terminal and a medium, and relates to the technical field of data processing.The method comprises the steps that to-be-perfected data assets are obtained, and the to-be-perfected data assets comprise first data; obtaining second data according to the first data through a missing value marking model; obtaining marked data based on the first data and the second data; and through a missing value prediction model, according to the marked data, obtaining data assets after missing value prediction. The objective of the invention is to accurately predict missing data in current data assets of an enterprise through the pre-trained neural network model, and help to fill missing values caused by various reasons in a data asset collection process, thereby improving the integrity of the data assets of the enterprise. And influence on subsequent enterprise analysis and enterprise decision due to partial loss of the data assets is avoided.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a method, device, terminal and medium for improving the integrity of data assets. Background Art

[0002] Data assets refer to all kinds of data owned by an enterprise, including structured data (such as table data in a database), unstructured data (such as text, images, and audio), and semi-structured data (such as JSON and XML data). Data assets have become crucial in modern enterprises because they are the basis for decision support, business operations, market analysis, and innovation. If the integrity of an enterprise's data assets is insufficient, it will lead to a series of problems such as wrong decisions, unsmooth business processes, reduced customer trust, compliance risks, and waste of resources.

[0003] At present, the method to improve the completeness of data assets is to find and supplement the missing parts in the data assets through manual query and collection through the collaboration of personnel from the company's statistics department or other departments. Since modern enterprises usually use multiple data sources, this makes data integration and standardization more complicated. Existing methods are often insufficient when processing these multi-source data and difficult to scale. In addition, traditional data filling methods (such as mean filling, front and back value filling, etc.) may lead to information loss or deviation, and cannot truly reflect the data distribution. The supplementary data obtained is not accurate enough, and thus may not be able to improve the completeness of data assets. Summary of the invention

[0004] The main purpose of the present invention is to provide a method, device, terminal and medium for improving the integrity of data assets, aiming to accurately predict the missing data in the current data assets of an enterprise through a pre-trained neural network model, and help fill in the missing values ​​caused by various reasons in the process of data asset collection, thereby improving the integrity of the enterprise's data assets and avoiding the impact of partial missing data assets on subsequent enterprise analysis and enterprise decision-making.

[0005] To achieve the above object, the present invention provides a method for improving the integrity of data assets, the method comprising:

[0006] Acquire data assets to be improved, wherein the data assets to be improved include first data, and the first data is used to represent data generated by production and business activities of the enterprise;

[0007] Obtaining second data according to the first data through a missing value marking model, wherein the second data is used to characterize the data missing situation of the first data;

[0008] Obtaining labeled data based on the first data and the second data;

[0009] The target data is obtained according to the labeled data through the missing value prediction model, wherein the target data is used to characterize the data assets after the missing values ​​are predicted.

[0010] Specifically, the missing value marking model includes a first input layer, a hidden layer and a first output layer, and the second data is a marking matrix;

[0011] The step of obtaining the second data according to the first data by using the missing value marking model includes:

[0012] Obtaining, through the first input layer, a first matrix corresponding to the first data according to the first data;

[0013] Obtaining a second matrix according to the first matrix through the hidden layer;

[0014] The labeling matrix is ​​obtained according to the second matrix through the first output layer, wherein the labeling matrix has the same shape as the first matrix.

[0015] Specifically, the hidden layer includes a first fully connected layer and a second fully connected layer, and the number of neurons in the first fully connected layer is twice the number of neurons in the second fully connected layer;

[0016] The step of obtaining a second matrix through the hidden layer according to the first matrix includes:

[0017] Inputting the first matrix into the first fully connected layer to obtain an intermediate matrix;

[0018] The intermediate matrix is ​​input into the second fully connected layer to obtain the second matrix.

[0019] Specifically, the first output layer is a third fully connected layer, and the number of neurons in the third fully connected layer is the same as the number of elements in the first matrix;

[0020] The step of obtaining the label matrix according to the second matrix through the first output layer includes:

[0021] Input the second matrix to the first output layer, and generate the missing probability corresponding to the element according to the elements in the second matrix through the preset activation function of the first output layer, wherein the missing probability is used to represent the probability that the element is regarded as a missing value;

[0022] The marking matrix is ​​generated based on a comparison result between the missing probability and a preset missing probability threshold.

[0023] Specifically, the labeling matrix includes a first element and a second element, the first element is used to represent a missing value, and the second element is used to represent a non-missing value;

[0024] The generating the label matrix based on the comparison result between the missing probability and a preset missing probability threshold comprises:

[0025] Traversing the elements, if the missing probability corresponding to the element is greater than or equal to the preset missing probability threshold, determining the element as the first element; if the missing probability corresponding to the element is less than the preset missing probability threshold, determining the element as the second element;

[0026] The labeling matrix is ​​generated based on the first element and the second element.

[0027] Specifically, the missing value prediction model includes an encoder and a decoder, and the target data is the reconstructed data corresponding to the marked data;

[0028] The method of obtaining target data according to the labeled data through the missing value prediction model includes:

[0029] Inputting the labeled data into the encoder to obtain a feature vector corresponding to the labeled data;

[0030] The feature vector is input into the decoding layer to obtain the reconstructed data.

[0031] Specifically, the missing value prediction model includes a second input layer, a recurrent neural network layer, an attention layer, and a second output layer, wherein the second output layer is a third fully connected layer;

[0032] The method of obtaining target data according to the labeled data through the missing value prediction model includes:

[0033] Inputting the labeled data into the second input layer to obtain an input vector corresponding to the labeled data;

[0034] Inputting the input vector into the recurrent neural network layer to obtain a hidden state vector;

[0035] Inputting the hidden state vector into the attention layer to obtain a context vector;

[0036] Inputting the context vector into the third fully connected layer to obtain a prediction vector, wherein the dimension value of the prediction vector is the same as the number of missing values ​​in the first data, and the elements of the prediction vector are used to represent the target prediction values ​​corresponding to the missing values;

[0037] The target data is obtained based on the labeled data and the prediction vector.

[0038] To achieve the above object, the present invention further provides a device for improving the integrity of data assets, the device comprising:

[0039] A first unit is used to obtain data assets to be improved, wherein the data assets to be improved include first data, and the first data is used to represent data generated by production and business activities of the enterprise;

[0040] A second unit is used to obtain second data according to the first data by using a missing value marking model, wherein the second data is used to characterize the data missing situation of the first data;

[0041] A third unit, configured to obtain labeled data based on the first data and the second data;

[0042] The fourth unit is used to obtain target data according to the labeled data through a missing value prediction model, wherein the target data is used to characterize the data assets after the missing values ​​are predicted.

[0043] To achieve the above objective, the present invention further provides a medium, wherein the medium stores a plurality of instructions, wherein the instructions are suitable for loading by a processor to execute the steps in any one of the methods provided by the present invention.

[0044] The present invention provides a method, device, terminal and medium for improving the integrity of data assets. First, the data assets to be improved can be obtained, wherein the data assets to be improved include first data; then, second data is obtained according to the first data through a missing value marking model; then, marked data is obtained based on the first data and the second data; and finally, the data assets after the missing values ​​are predicted are obtained according to the marked data through a missing value prediction model.

[0045] Compared with the existing methods of improving the completeness of data assets, the present invention can accurately find the missing parts of data assets through the missing value marking model, and accurately supplement the missing parts of data assets through the missing value prediction model, thereby accurately improving the completeness of data assets and avoiding the impact of partial missing data assets on subsequent enterprise analysis and enterprise decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A schematic diagram of a method flow chart provided by an embodiment of the present invention;

[0047] Figure 2 A schematic diagram of the structure of a device for improving data asset integrity provided by an embodiment of the present invention;

[0048] Figure 3 A schematic diagram of the structure of a terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0050] At present, the method to improve the integrity of data assets is to find and supplement the missing parts in data assets through the cooperation of personnel from the statistics department or other departments of the enterprise, and adopt the form of manual query and collection. Since modern enterprises usually use multiple data sources, data integration and standardization are more complicated. Existing methods are often insufficient in processing these multi-source data and difficult to process on a large scale. In addition, traditional data filling methods (such as mean filling, front and back value filling, etc.) may cause information loss or deviation, and cannot truly reflect the data distribution. The obtained supplementary data is not accurate enough, and thus may not be able to improve the integrity of data assets. Therefore, the embodiments of the present invention provide a method, device, terminal and medium for improving the integrity of data assets to solve practical technical problems.

[0051] In some embodiments, the device may be specifically integrated into an electronic device, which may be a terminal, a server or other device.

[0052] In some embodiments, the server may also be implemented in the form of a terminal.

[0053] Among them, the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0054] The terminal may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, and the present invention does not limit this.

[0055] It should be noted that the serial numbers of the following embodiments are not intended to limit the preferred order of the embodiments.

[0056] An embodiment of the present invention provides a method for improving the integrity of data assets. The method can accurately predict the missing data in the current data assets of an enterprise through a pre-trained neural network model, and help fill in the missing values ​​caused by various reasons during the data asset collection process, thereby improving the integrity of the enterprise's data assets.

[0057] A method to improve the integrity of data assets, such as Figure 1 As shown, the specific process of the method can be as follows:

[0058] S110. Acquire data assets to be improved, wherein the data assets to be improved include first data, and the first data is used to represent data generated by production and business activities of the enterprise.

[0059] In some embodiments, the first data may include structured data (such as tabular data in a database), unstructured data (such as text, images, and audio), semi-structured data (such as JSON and XML data), and the like.

[0060] In some embodiments, AIGC (AI-Generated Content) big model technology can be used to improve the integrity of different types of data.

[0061] In some embodiments, for structured data, such as table data in a database, the following steps may be taken to improve data integrity:

[0062] Data analysis: Analyze the statistical characteristics of the data set, such as mean, standard deviation, etc., to understand the data distribution;

[0063] Missing value filling: Use statistical methods (such as mean, median filling) or machine learning models (such as KNN, decision tree) to predict and fill missing values;

[0064] Anomaly detection: Identify and handle outliers using clustering or outlier detection algorithms;

[0065] Feature Engineering: Creating new features or transforming existing features to enhance a dataset.

[0066] In some embodiments, for unstructured data, such as text, images, and audio, the following steps can be taken to improve data integrity:

[0067] Text data: Use natural language processing (NLP) techniques, such as word embedding and sequence-to-sequence models, to perform text completion, sentiment analysis, topic modeling, etc.

[0068] Image data: Use computer vision techniques such as image recognition, object detection, image segmentation, etc. to identify the content in the image and possibly add metadata or tags;

[0069] Audio data: Use speech recognition technology to transcribe the audio into text, and then apply text processing techniques to further enrich the data.

[0070] In some embodiments, for semi-structured data, such as JSON and XML, the following steps can be taken to improve data integrity:

[0071] Pattern recognition: automatically identify data patterns and extract key fields;

[0072] Data mapping: Mapping the extracted data into a suitable structured format;

[0073] Content expansion: Use natural language processing or other related technologies to enrich node content.

[0074] In some embodiments, different types of existing data may be combined, such as linking text data with corresponding image or audio data to form a richer data set, and third-party data sources may be introduced to supplement existing data sets.

[0075] In some embodiments, the processed data can also be used to train the AIGC model to generate more complete and rich new data. Finally, the model is verified to ensure that the quality of the generated data meets expectations.

[0076] S120. Obtain second data based on the first data through a missing value marking model, wherein the second data is used to characterize the data missing situation of the first data.

[0077] In some embodiments, the missing value labeling model includes a first input layer, a hidden layer, and a first output layer, and the second data is a labeling matrix.

[0078] At present, the method to improve the integrity of data assets is through the collaboration of personnel from the company's statistics department or other departments. Manual query and collection are needed to find the missing parts in the data assets. Considering that the data assets of modern enterprises often include massive amounts of data, and the missing values ​​in the data assets may also be an erroneous value far from the normal value range, it requires staff with corresponding skills and experience to spend a lot of time to search and confirm, and then mark the missing parts.

[0079] In some embodiments, missing values ​​are marked in large-scale data assets through a trained neural network model, which has the following advantages over manual marking:

[0080] A trained neural network model can process large amounts of data in a short period of time. For very large data sets, manually checking and labeling each piece of data is impractical, time-consuming, and error-prone. Once the neural network model is trained, it will label the data in a consistent manner, avoiding the consistency issues caused by subjectivity or fatigue that may exist in manual labeling.

[0081] Trained neural network models are able to recognize complex data patterns, including high-dimensional associations that are difficult for humans to detect. This makes them more accurate in predicting which data points may be missing values; neural network models can automatically learn useful features from raw data without the need for manual feature engineering. This is especially important for detecting missing values, because effective features can reveal potential structures in the data;

[0082] Neural network models are easy to deploy in distributed computing environments, making it easy to handle growing amounts of data;

[0083] At the same time, with the addition of more data and continuous training of the model, the neural network model can continuously improve its performance and continuously improve the labeling accuracy.

[0084] In some embodiments, the first input layer is the first layer of the missing value labeling model, and its main function is to receive external input data and pass it to subsequent layers in the model.

[0085] In some embodiments, the hidden layers can automatically learn useful feature representations from the original input data.

[0086] In some embodiments, the first output layer is the last layer of the missing value labeling model, and outputs a labeling matrix that identifies the missing parts.

[0087] In some embodiments, obtaining the second data according to the first data by using the missing value labeling model includes the following steps S121 to S123:

[0088] S121. Obtain a first matrix corresponding to the first data through the first input layer according to the first data.

[0089] In some embodiments, the first matrix may include multiple features, each feature may include missing values, and the missing status of the first matrix is ​​determined by the status of the current data assets.

[0090] S122. Obtain a second matrix through the hidden layer according to the first matrix.

[0091] In some embodiments, the hidden layer includes a first fully connected layer and a second fully connected layer, and the number of neurons in the first fully connected layer is twice the number of neurons in the second fully connected layer.

[0092] In some embodiments, the first fully connected layer and the second fully connected layer may be fully connected layers (FC layers for short). The fully connected layer is one of the most basic and widely used layers in a neural network. The fully connected layer can perform weighted summation and nonlinear transformation on features from the previous layer, thereby learning more complex feature representations.

[0093] Neuron is the basic unit of artificial neural network. In artificial neural network, neuron is a mathematical model that simulates the behavior of biological neurons.

[0094] Neurons receive input signals from other neurons or the external environment. Each input signal is associated with a weight, which determines the intensity of the input signal's impact on the neuron. Neurons multiply all input signals with their corresponding weights and add the results. The result of the weighted summation is nonlinearly transformed through an activation function to produce the neuron's output.

[0095] In some embodiments, obtaining the second matrix according to the first matrix through the hidden layer includes the following steps S1221 to S1222:

[0096] S1221. Input the first matrix into the first fully connected layer to obtain an intermediate matrix.

[0097] S1222. Input the intermediate matrix into the second fully connected layer to obtain the second matrix.

[0098] In some embodiments, the number of neurons in the first fully connected layer may be 2n, the activation function used by the first fully connected layer is the ReLU function, the number of neurons in the second fully connected layer is n, and the activation function used by the second fully connected layer is the ReLU function.

[0099] The ReLU (Rectified Linear Unit) function is a commonly used activation function, which is widely used in deep learning and neural networks.

[0100] S123. Obtain the labeling matrix according to the second matrix through the first output layer, wherein the labeling matrix has the same shape as the first matrix.

[0101] In some embodiments, the first output layer is a third fully connected layer, and the number of neurons in the third fully connected layer is the same as the number of elements in the first matrix.

[0102] In some embodiments, obtaining the labeling matrix according to the second matrix through the first output layer includes the following steps S1231 to S1232:

[0103] S1231. Input the second matrix into the first output layer, and generate the missing probability corresponding to the element according to the elements in the second matrix through the preset activation function of the first output layer, wherein the missing probability is used to characterize the probability that the element is regarded as a missing value.

[0104] In some embodiments, the first output layer uses a Sigmoid function to limit the output of each neuron to between [0, 1] to represent the probability of missing values.

[0105] The Sigmoid function is a commonly used activation function, which is used to introduce nonlinear characteristics in neural networks, enabling neural networks to learn and approximate complex function mappings and map real numbers to the (0,1) interval.

[0106] S1232: Generate the marking matrix based on a comparison result between the missing probability and a preset missing probability threshold.

[0107] In some embodiments, the labeling matrix includes a first element and a second element, wherein the first element is used to represent a missing value, and the second element is used to represent a non-missing value.

[0108] In some embodiments, the generating the marking matrix based on the comparison result between the missing probability and a preset missing probability threshold comprises the following steps S12321 to S12322:

[0109] S12321. Traverse the elements, and if the missing probability corresponding to the element is greater than or equal to the preset missing probability threshold, determine the element as the first element; if the missing probability corresponding to the element is less than the preset missing probability threshold, determine the element as the second element.

[0110] In some embodiments, the preset missing probability threshold can be 0.5. If the missing probability corresponding to the element is greater than or equal to 0.5, the element is determined to be the first element, and the value of the first element is determined to be 1; if the missing probability corresponding to the element is less than 0.5, the element is determined to be the second element, and the value of the second element is determined to be 0.

[0111] S12322. Generate the marking matrix based on the first element and the second element.

[0112] Continuing with the above embodiment, a binary marking matrix is ​​generated based on the first element with a value of 1 and the second element with a value of 0.

[0113] S130: Obtain labeled data based on the first data and the second data.

[0114] Continuing with the above embodiment, the first data may be data in a table form or a key-pair value form, and the second data may be a binary marking matrix, in which missing values ​​may be represented by 1, 0 or null.

[0115] Specifically, the 1 corresponding to the missing value in the marking matrix is ​​transferred to the position where the missing value in the first data is located according to the corresponding position relationship to obtain the marked data, that is, the marked part of the marked data is the part where 1, 0 or null is located.

[0116] S140. Obtain target data based on the labeled data through a missing value prediction model, wherein the target data is used to characterize data assets after predicting missing values.

[0117] At present, the method to improve the completeness of data assets is through the collaboration of personnel from the company's statistics department or other departments. It is necessary to supplement the missing parts in the data assets through re-collection and manual direct filling. Considering that the data assets of modern enterprises often include massive data, and re-collecting data requires additional time costs, labor costs and even equipment loss costs; at the same time, the re-collected data or the data directly supplemented by humans based on technical experience may not be able to accurately restore the missing values ​​due to changes in current technical conditions and environment.

[0118] In some embodiments, the trained neural network prediction model can process large-scale data sets in a short period of time without human intervention. This is especially important for large-scale data sets containing tens of thousands of records; the neural network prediction model can capture complex patterns in the data and use these patterns to predict missing values. This pattern recognition ability is usually beyond the scope of human cognition, especially when it comes to multi-dimensional and nonlinear relationships; it can automatically extract key features that help prediction without manual feature engineering. This means that even if there are complex interactions in the data, the model can effectively use this information. Artificially filling in data may introduce bias or errors, especially when faced with large amounts of data, while the neural network prediction model reduces this subjectivity and provides more objective and consistent results; as more data is added and the model is continuously trained, the neural network prediction model can continuously improve its performance and improve prediction accuracy.

[0119] In some embodiments, the missing value prediction model may include an encoder and a decoder, and the target data is reconstructed data corresponding to the labeled data.

[0120] The encoder is responsible for converting the input data into a lower dimensional representation, the so-called encoding or hidden state.

[0121] The decoder reconstructs the low-dimensional representation (also called encoding or hidden state) produced by the encoder back into the shape and form of the original data space.

[0122] In some embodiments, obtaining target data according to the labeled data by using a missing value prediction model includes steps A1 to A2 as shown below:

[0123] A1. Input the labeled data into the encoder to obtain a feature vector corresponding to the labeled data.

[0124] In some embodiments, missing values ​​in the labeled data may be marked as specific symbols, such as -999 or NaN, etc., and the feature vector represents a compressed state of the labeled data.

[0125] A2. Input the feature vector into the decoding layer to obtain the reconstructed data.

[0126] In some embodiments, the feature vector is reconstructed back to the spatial shape and form of the labeled data to obtain the reconstructed data with the same spatial shape as the labeled data.

[0127] In some embodiments, the missing value prediction model may include a second input layer, a recurrent neural network layer, an attention layer, and a second output layer, wherein the second output layer is a third fully connected layer.

[0128] The recurrent neural network layer can be a network layer based on a recurrent neural network (RNN), which can capture the time dependency in the data and is very suitable for processing sequence data such as text, speech, and time series. When processing sequence data, the input at each time point is called a "time step". RNNs receive an input at each time step and produce an output. The key feature of RNNs is that their hidden layers are connected to themselves, forming a recurrent connection. This means that at each time step, RNNs not only consider the input of the current time step, but also the hidden state of the previous time step.

[0129] The attention layer is a specific component that implements the attention mechanism in a neural network. It can have different forms. The most common one is the query-key-value based attention mechanism. When processing sequence data, the attention layer assigns different weights to different inputs, and can focus on some of the more important parts.

[0130] The method of obtaining target data according to the labeled data by using the missing value prediction model includes steps B1 to B5 as shown below:

[0131] B1. Input the labeled data into the second input layer to obtain an input vector corresponding to the labeled data.

[0132] B2. Input the input vector into the recurrent neural network layer to obtain a hidden state vector.

[0133] In some embodiments, the input of the recurrent neural network layer may be an input vector X with a time step of t. t , the output can be the hidden vector h t .

[0134] B3. Input the hidden state vector into the attention layer to obtain a context vector.

[0135] Continuing with the above embodiment, the input of the attention layer can be the hidden states h1, h2, ..., h T , the context vector may be a weighted hidden state c.

[0136] B4. Input the context vector into the third fully connected layer to obtain a prediction vector, wherein the dimension value of the prediction vector is the same as the number of missing values ​​in the first data, and the elements of the prediction vector are used to represent the target prediction values ​​corresponding to the missing values.

[0137] Continuing with the above embodiment, the input of the third fully connected layer may be the weighted hidden state c, and the output of the third fully connected layer may be a prediction vector represented by X^miss.

[0138] B5. Obtain the target data based on the labeled data and the prediction vector.

[0139] Continuing with the above embodiment, the elements of the prediction vector are added to the corresponding positions in the marked data to obtain the target data.

[0140] From the above, it can be seen that the embodiment of the present invention can first obtain the data assets to be improved, wherein the data assets to be improved include first data; then, through the missing value marking model, the second data is obtained according to the first data; thereafter, based on the first data and the second data, the marked data is obtained; through the missing value prediction model, according to the marked data, finally the data assets after the missing values ​​are predicted are obtained.

[0141] Compared with the existing methods of improving the completeness of an enterprise's data assets, the present invention can accurately find out the missing parts of data assets through a missing value marking model, and accurately supplement the missing parts of data assets through a missing value prediction model, helping to fill in the missing values ​​caused by various reasons in the process of data asset collection, thereby improving the completeness of the enterprise's data assets and avoiding the impact of partial missing data assets on subsequent enterprise analysis and enterprise decision-making.

[0142] In order to better implement the above method, the embodiment of the present invention also provides a device for improving the integrity of data assets, which can be integrated in an electronic device, and the electronic device can be a terminal, a server, etc. Among them, the terminal can be a mobile phone, a tablet computer, a smart Bluetooth device, a laptop, a personal computer, etc.; the server can be a single server or a server cluster composed of multiple servers.

[0143] For example, in this embodiment, the method of the embodiment of the present invention is described in detail by taking the device for improving the integrity of data assets as an example in which the device is specifically integrated in a terminal.

[0144] For example, Figure 2 As shown, the method and device for improving data asset integrity may include a first unit 201, a second unit 202, a third unit 203, and a fourth unit 204, and the device includes:

[0145] The first unit 201 is used to obtain data assets to be improved, wherein the data assets to be improved include first data, and the first data is used to represent data generated by production and business activities of the enterprise;

[0146] The second unit 202 is used to obtain second data according to the first data through a missing value marking model, wherein the second data is used to characterize the data missing situation of the first data;

[0147] The third unit 203 is used to obtain marked data based on the first data and the second data;

[0148] The fourth unit 204 is used to obtain target data according to the labeled data through a missing value prediction model, wherein the target data is used to characterize the data assets after the missing values ​​are predicted.

[0149] In specific implementation, the above units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units can refer to the previous method embodiments, which will not be repeated here.

[0150] From the above, it can be seen that the embodiments of the present invention can accurately predict the missing data in the current data assets of the enterprise through a pre-trained neural network model, help fill in the missing values ​​caused by various reasons in the data asset collection process, and thus improve the integrity of the enterprise's data assets.

[0151] The embodiment of the present invention further provides an electronic device, which may be a terminal, a server, or the like. The terminal may be a mobile phone, a tablet computer, a smart Bluetooth device, a notebook computer, a personal computer, or the like; the server may be a single server or a server cluster composed of multiple servers, or the like.

[0152] In some embodiments, the device may also be integrated into multiple electronic devices. For example, the device may be integrated into multiple servers, and the method for improving the integrity of data assets of the present invention may be implemented by multiple servers.

[0153] In this embodiment, the electronic device of this embodiment is a terminal as an example for detailed description, for example, Figure 3 As shown, it shows a schematic diagram of the structure of the terminal involved in the embodiment of the present invention, specifically:

[0154] The terminal may include one or more processors 301 of processing cores, one or more storage media 302, a power supply 303, an input module 304, and a communication module 305. Those skilled in the art will appreciate that Figure 3 The terminal structure shown in the figure does not constitute a limitation on the terminal, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0155] in:

[0156] The processor 301 is the control center of the terminal. It uses various interfaces and lines to connect various parts of the entire terminal. It executes various functions of the terminal and processes data by running or executing software programs and / or modules stored in the memory 302, and calling data stored in the memory 302, so as to monitor the terminal as a whole. In some embodiments, the processor 301 may include one or more processing cores; in some embodiments, the processor 301 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 301.

[0157] The memory 302 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 302. The memory 302 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 302 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 302 may also include a memory controller to provide the processor 301 with access to the memory 302.

[0158] The terminal also includes a power supply 303 for supplying power to various components. In some embodiments, the power supply 303 can be logically connected to the processor 301 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 303 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.

[0159] The terminal may further include an input module 304, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls.

[0160] The terminal may also include a communication module 305. In some embodiments, the communication module 305 may include a wireless module. The terminal may perform short-range wireless transmission through the wireless module of the communication module 305, thereby providing the user with wireless broadband Internet access. For example, the communication module 305 may be used to help the user send and receive emails, browse web pages, and access streaming media.

[0161] Although not shown, the terminal may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 301 in the terminal will load the executable files corresponding to the processes of one or more applications into the memory 302 according to the following instructions, and the processor 301 will run the applications stored in the memory 302, thereby realizing various functions, as follows:

[0162] Acquire data assets to be improved, wherein the data assets to be improved include first data, and the first data is used to represent data generated by production and business activities of the enterprise;

[0163] Obtaining second data according to the first data through a missing value marking model, wherein the second data is used to characterize the data missing situation of the first data;

[0164] Obtaining labeled data based on the first data and the second data;

[0165] The target data is obtained according to the labeled data through the missing value prediction model, wherein the target data is used to characterize the data assets after the missing values ​​are predicted.

[0166] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0167] From the above, it can be seen that the embodiments of the present invention can accurately predict the missing data in the current data assets of the enterprise through a pre-trained neural network model, help fill in the missing values ​​caused by various reasons in the data asset collection process, and thus improve the integrity of the enterprise's data assets.

[0168] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a medium and loaded and executed by a processor.

[0169] To this end, an embodiment of the present invention provides a medium in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps in any one of the methods for improving the integrity of data assets provided in the embodiment of the present invention. For example, the instructions can execute the following steps:

[0170] Acquire data assets to be improved, wherein the data assets to be improved include first data, and the first data is used to represent data generated by production and business activities of the enterprise;

[0171] Obtaining second data according to the first data through a missing value marking model, wherein the second data is used to characterize the data missing situation of the first data;

[0172] Obtaining labeled data based on the first data and the second data;

[0173] The target data is obtained according to the labeled data through the missing value prediction model, wherein the target data is used to characterize the data assets after the missing values ​​are predicted.

[0174] The medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0175] According to one aspect of the present invention, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a medium. A processor of a computer device reads the computer instructions from the medium, and the processor executes the computer instructions, so that the computer device executes the method provided in the various optional implementations provided in the above embodiments.

[0176] Since the instructions stored in the medium can execute the steps of any method for improving data asset integrity provided in the embodiments of the present invention, the beneficial effects that can be achieved by any method for improving data asset integrity provided in the embodiments of the present invention can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0177] The above is a detailed introduction to a method, device, terminal and medium for improving the integrity of data assets provided by an embodiment of the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for improving the integrity of data assets, characterized in that: The method comprises: Acquire data assets to be improved, wherein the data assets to be improved include first data, and the first data is used to represent data generated by production and business activities of the enterprise; Obtaining second data according to the first data through a missing value marking model, wherein the second data is used to characterize the data missing situation of the first data; Obtaining labeled data based on the first data and the second data; The target data is obtained according to the labeled data through the missing value prediction model, wherein the target data is used to characterize the data assets after the missing values ​​are predicted.

2. The method according to claim 1, characterized in that The missing value marking model includes a first input layer, a hidden layer and a first output layer, and the second data is a marking matrix; The step of obtaining the second data according to the first data by using the missing value marking model includes: Obtaining, through the first input layer, a first matrix corresponding to the first data according to the first data; Obtaining a second matrix according to the first matrix through the hidden layer; The labeling matrix is ​​obtained according to the second matrix through the first output layer, wherein the labeling matrix has the same shape as the first matrix.

3. The method according to claim 2, characterized in that The hidden layer includes a first fully connected layer and a second fully connected layer, and the number of neurons in the first fully connected layer is twice the number of neurons in the second fully connected layer; The step of obtaining a second matrix through the hidden layer according to the first matrix includes: Inputting the first matrix into the first fully connected layer to obtain an intermediate matrix; The intermediate matrix is ​​input into the second fully connected layer to obtain the second matrix.

4. The method according to claim 2, characterized in that The first output layer is a third fully connected layer, and the number of neurons in the third fully connected layer is the same as the number of elements in the first matrix; The step of obtaining the label matrix according to the second matrix through the first output layer includes: Input the second matrix to the first output layer, and generate the missing probability corresponding to the element according to the elements in the second matrix through the preset activation function of the first output layer, wherein the missing probability is used to represent the probability that the element is regarded as a missing value; The marking matrix is ​​generated based on a comparison result between the missing probability and a preset missing probability threshold.

5. The method according to claim 4, characterized in that The labeling matrix includes a first element and a second element, the first element is used to represent a missing value, and the second element is used to represent a non-missing value; The generating the label matrix based on the comparison result between the missing probability and a preset missing probability threshold comprises: Traversing the elements, if the missing probability corresponding to the element is greater than or equal to the preset missing probability threshold, determining the element as the first element; if the missing probability corresponding to the element is less than the preset missing probability threshold, determining the element as the second element; The labeling matrix is ​​generated based on the first element and the second element.

6. The method according to claim 1, characterized in that The missing value prediction model includes an encoder and a decoder, and the target data is the reconstructed data corresponding to the marked data; The method of obtaining target data according to the labeled data through the missing value prediction model includes: Inputting the labeled data into the encoder to obtain a feature vector corresponding to the labeled data; The feature vector is input into the decoding layer to obtain the reconstructed data.

7. The method according to claim 1, characterized in that The missing value prediction model includes a second input layer, a recurrent neural network layer, an attention layer and a second output layer, wherein the second output layer is a third fully connected layer; The method of obtaining target data according to the labeled data through the missing value prediction model includes: Inputting the labeled data into the second input layer to obtain an input vector corresponding to the labeled data; Inputting the input vector into the recurrent neural network layer to obtain a hidden state vector; Inputting the hidden state vector into the attention layer to obtain a context vector; Inputting the context vector into the third fully connected layer to obtain a prediction vector, wherein the dimension value of the prediction vector is the same as the number of missing values ​​in the first data, and the elements of the prediction vector are used to represent the target prediction values ​​corresponding to the missing values; The target data is obtained based on the labeled data and the prediction vector.

8. A device for improving the integrity of data assets, characterized in that: The device comprises: A first unit is used to obtain data assets to be improved, wherein the data assets to be improved include first data, and the first data is used to represent data generated by production and business activities of the enterprise; A second unit is used to obtain second data according to the first data by using a missing value marking model, wherein the second data is used to characterize the data missing situation of the first data; A third unit, configured to obtain labeled data based on the first data and the second data; The fourth unit is used to obtain target data according to the labeled data through a missing value prediction model, wherein the target data is used to characterize the data assets after the missing values ​​are predicted.

9. A terminal, characterized in that: The method comprises a processor and a memory, wherein the memory stores a plurality of instructions; the processor loads instructions from the memory to execute the steps in the method according to any one of claims 1 to 7.

10. A medium, characterized in that The medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the method according to any one of claims 1 to 7.

Images