Data evaluation method and device, terminal and storage medium
By acquiring data configuration information and executing evaluation procedures while the network is offline, the inefficiency and credibility of data quality assessment in the prior art in the complex network environment is solved, and accurate and efficient data quality assessment is achieved.
Patent Information
- Application Number
- CN202411338797.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-25
- Publication Date
- 2025-01-07
AI Technical Summary
The existing networked data quality inspection and evaluation are difficult to perform in network environments with security risks or in situations where the network is disconnected. It is inefficient, costly, and the evaluation results are not accurate and credible enough.
A data evaluation method and device are provided, which can generate an evaluation program based on the data evaluation criteria by obtaining configuration information of the data to be evaluated in the network offline state, and execute the evaluation program through the data consistency verification model to obtain data quality detection results.
Accurate evaluation of data quality in an inability to connect to the Internet or in complex network environments improves the evaluation efficiency and credibility of results, avoiding security risks and high costs.
Smart Images
Figure CN119271654A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of data processing technology, and in particular to a data evaluation method, device, terminal and storage medium. Background Art
[0002] Data quality assessment is the process of ensuring that data is accurate, complete, consistent, and usable. Good data quality is critical to making reliable decisions, especially in areas such as data analytics, machine learning, and business intelligence.
[0003] However, existing networked data quality detection and assessment cannot cope with network environments with security risks or network disconnections. Data quality can only be assessed by independently deploying applications, which is inefficient and costly. The data quality assessment process is difficult to monitor, and the assessment results are separated from the monitoring environment. Therefore, the assessment results are not accurate enough and have low credibility. Summary of the invention
[0004] The main purpose of the present invention is to provide a data evaluation method, device, terminal and storage medium, aiming to accurately implement data quality evaluation when the network is not connected or in a relatively complex network environment.
[0005] To achieve the above object, the present invention provides a data evaluation method, which is applied to a data evaluation system, wherein the data evaluation system is in a network offline state, and the data evaluation system includes a database, and the database is used for data storage in the network offline state;
[0006] The method comprises:
[0007] Obtain configuration information corresponding to the data to be evaluated;
[0008] Based on the data evaluation criteria and the data to be evaluated, generating an evaluation procedure;
[0009] Based on the configuration information and the data to be evaluated, the evaluation program is executed to obtain a target evaluation result, wherein the target evaluation result is used to characterize a data quality detection result of the data to be evaluated.
[0010] Specifically, the obtaining of configuration information corresponding to the data to be evaluated includes:
[0011] Screening out the data to be evaluated from the initial data;
[0012] Based on the data to be evaluated, the data evaluation system is configured to generate the configuration information, wherein the configuration information includes an evaluation object corresponding to the data to be evaluated, evaluation parameters corresponding to the data to be evaluated, and a data extraction method corresponding to the data to be evaluated.
[0013] Specifically, before generating an evaluation program based on the data evaluation standard and the data to be evaluated, the method further includes:
[0014] Establish initial data evaluation criteria;
[0015] Based on the initial data evaluation standard and the initial data, the initial data evaluation standard is updated to obtain the data evaluation standard.
[0016] Specifically, before executing the evaluation program based on the configuration information and the data to be evaluated to obtain a target evaluation result, the method further includes:
[0017] Filtering sample data from the data to be evaluated;
[0018] Through the data consistency verification model, a data consistency verification result is generated according to the configuration information and the sample data. If the data consistency verification result is that the data to be evaluated meets the preset data consistency standard, the data to be evaluated and the configuration information are stored in the database; if the data consistency verification result is that the data to be evaluated does not meet the preset data consistency standard, the data to be evaluated is re-screened from the initial data until the data consistency verification result is that the data to be evaluated meets the preset data consistency standard.
[0019] Specifically, the data consistency verification model includes an input layer, an embedding layer, an encoding layer, a fully connected layer, and an output layer;
[0020] The step of generating a data consistency verification result according to the configuration information and the sample data through the data consistency verification model includes:
[0021] Generate a first feature vector through the input layer according to the configuration information;
[0022] Generate a numerical vector corresponding to the sample data according to the sample data through the input layer;
[0023] Generate a second feature vector according to the first feature vector through the embedding layer;
[0024] Generate a serialized feature vector according to the numerical vector and the second feature vector through the encoding layer;
[0025] Generate an output vector according to the serialized feature vector through the fully connected layer;
[0026] The data consistency verification result is obtained through the output layer according to the output vector.
[0027] Specifically, the encoding layer includes a long short-term memory network layer;
[0028] The step of generating a serialized feature vector according to the numerical vector and the second feature vector through the encoding layer includes:
[0029] Input the numerical vector and the second vector into the long short-term memory network layer, and output a hidden state vector and a unit state vector;
[0030] The hidden state vector and the unit state vector are integrated to obtain the serialized feature vector.
[0031] Specifically, obtaining the data consistency verification result according to the output vector through the output layer includes:
[0032] Inputting the output vector into the output layer to obtain an output probability value, wherein the output probability value is used to characterize the possibility that the data to be evaluated meets the preset data consistency standard;
[0033] If the output probability value is greater than or equal to the preset probability threshold, it is determined that the data to be evaluated meets the preset data consistency standard; if the output probability value is less than the preset probability threshold, it is determined that the data to be evaluated does not meet the preset data consistency standard.
[0034] To achieve the above object, the present invention further provides a data evaluation device, which is applied to a data evaluation system, wherein the data evaluation system is in a network offline state, and the data evaluation system includes a database, and the database is used for data storage in the network offline state;
[0035] The device comprises:
[0036] The first unit is used to obtain configuration information corresponding to the data to be evaluated;
[0037] A second unit is used to generate an evaluation program based on the data evaluation standard and the data to be evaluated;
[0038] The third unit is used to execute the evaluation program based on the configuration information and the data to be evaluated to obtain a target evaluation result, wherein the target evaluation result is used to characterize a data quality detection result of the data to be evaluated.
[0039] To achieve the above objective, the present invention further provides a storage medium, wherein the storage medium stores a plurality of instructions, wherein the instructions are suitable for loading by a processor to execute the steps in any one of the methods provided by the present invention.
[0040] The present invention provides a data evaluation method, device, terminal and storage medium, which can first obtain configuration information corresponding to the data to be evaluated; then, based on the data evaluation standard and the data to be evaluated, generate an evaluation program; based on the configuration information and the data to be evaluated, execute the evaluation program, and finally obtain the data quality detection result of the data to be evaluated.
[0041] Compared with the existing data quality assessment methods, the present invention can accurately obtain the configuration data of the data to be evaluated when the network cannot be connected or in a more complex network environment, and execute the assessment program based on the data assessment standard in an offline state to accurately evaluate the data quality of the data to be evaluated. BRIEF DESCRIPTION OF THE DRAWINGS
[0042] Figure 1 A schematic diagram of a method flow chart provided by an embodiment of the present invention;
[0043] Figure 2 A schematic diagram of the structure of a data evaluation device provided by an embodiment of the present invention;
[0044] Figure 3 A schematic diagram of the structure of a terminal provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.
[0046] Since there are still technicians on site in the workshop, they evaluate the production equipment and internal environment of the workshop based on their work experience and knowledge reserves. Due to the different experiences and levels of different technicians, the evaluation results of the workshop are biased or cannot truly reflect the current situation of the workshop; the data generated by the production equipment during operation is difficult to collect, and the cost of manual data collection is too high; massive data requires efficient processing algorithms and powerful computing power for analysis and processing, and technicians cannot quickly analyze the data collected on site to complete the workshop evaluation; at the same time, in complex network conditions, such as workshop production environment, external network interference and intrusion that affect network security, the workshop and the Internet cannot be interconnected, and it is difficult to remotely install and deploy data quality evaluation programs to complete data quality evaluation. Therefore, the embodiments of the present invention provide a data evaluation method, device, terminal and storage medium to solve practical technical problems.
[0047] In some embodiments, the device may be specifically integrated into an electronic device, which may be a terminal, a server or other device.
[0048] In some embodiments, the server may also be implemented in the form of a terminal.
[0049] Among them, the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.
[0050] The terminal may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, and the present invention does not limit this.
[0051] It should be noted that the serial numbers of the following embodiments are not intended to limit the preferred order of the embodiments.
[0052] An embodiment of the present invention provides a data evaluation method, which can accurately implement data quality evaluation when the network is unavailable or in a relatively complex network environment.
[0053] A data evaluation method is applied to a data evaluation system, wherein the data evaluation system is in a network offline state and comprises a database, wherein the database is used for data storage in the network offline state.
[0054] like Figure 1 As shown, the specific process of the method can be as follows:
[0055] S110: Obtain configuration information corresponding to the data to be evaluated.
[0056] The data to be evaluated can be data sets that need to be evaluated for data quality. These data sets may come from various channels, such as databases, files, API interfaces, or other data sources. Usually, before conducting a data quality assessment, it is necessary to clarify the source, format, structure, and relevant business background of the data to be evaluated.
[0057] The configuration information of the data to be evaluated can be metadata that describes the structure, content, and attributes of the data set. This configuration information is crucial for understanding and evaluating data quality. The configuration information can include the structure of the data set, field definitions, data types, data ranges, and other metadata related to data quality assessment.
[0058] In some embodiments, the step of obtaining configuration information corresponding to the data to be evaluated includes the following steps S111 to S112:
[0059] S111, filtering out the data to be evaluated from the initial data.
[0060] In some embodiments, SQL queries, Python scripts, or other tools may be used to apply screening conditions to obtain the data to be evaluated.
[0061] S112. Based on the data to be evaluated, configure the data evaluation system to generate the configuration information, wherein the configuration information includes an evaluation object corresponding to the data to be evaluated, evaluation parameters corresponding to the data to be evaluated, and a data extraction method corresponding to the data to be evaluated.
[0062] In some embodiments, the process of generating configuration information by configuring the data evaluation system generally includes defining metadata such as the structure of the data set, field definition, data type, data range, etc. The configuration information will be used for subsequent data quality evaluation.
[0063] In some embodiments, before generating an evaluation program based on the data evaluation criteria and the data to be evaluated, the method further includes steps A1 to A2 as shown below:
[0064] A1. Establish initial data evaluation criteria.
[0065] In some embodiments, an initial data assessment standard is established based on data quality assessment requirements. For example, the data quality assessment method for data assets is performed with reference to the national standard "GB / T 36344-2018 Information Technology Data Quality Evaluation Indicators" to establish the initial data assessment standard.
[0066] A2. Based on the initial data evaluation standard and the initial data, update the initial data evaluation standard to obtain the data evaluation standard.
[0067] In some embodiments, the initial data evaluation standard may also be updated according to information such as the source of the initial data and the application scenario to which it belongs, to obtain the data evaluation standard.
[0068] In some embodiments, before executing the evaluation program based on the configuration information and the data to be evaluated to obtain the target evaluation result, the method further includes steps B1 to B2 as shown below:
[0069] B1. Filtering sample data from the data to be evaluated.
[0070] In some embodiments, the purpose of screening sample data from the data to be evaluated is to better understand the characteristics and structure of the data, which is very important for subsequent data quality evaluation. Screening sample data can help understand the distribution, format and potential problems of the data to be evaluated.
[0071] Specifically, the screening can be performed by determining a standard for screening the sample data, such as selecting the latest batch of data, randomly selecting a portion of the data, and the like.
[0072] B2. Generate a data consistency verification result according to the configuration information and the sample data through a data consistency verification model; if the data consistency verification result is that the data to be evaluated meets the preset data consistency standard, store the data to be evaluated and the configuration information in the database; if the data consistency verification result is that the data to be evaluated does not meet the preset data consistency standard, re-screen the data to be evaluated from the initial data until the data consistency verification result is that the data to be evaluated meets the preset data consistency standard.
[0073] In some embodiments, if the data consistency verification result is that the data to be evaluated meets the preset data consistency standard, the data to be evaluated and the configuration information are determined to be unchanged, and can be transferred to the database inside the data evaluation system for storage and backup to avoid the occurrence of security risks caused by networking.
[0074] In some embodiments, the data consistency verification model includes an input layer, an embedding layer, an encoding layer, a fully connected layer, and an output layer.
[0075] The input layer is usually the first layer in a neural network. Its main function is to receive raw data and pass it to subsequent layers in the network for processing.
[0076] Embedding layer, which is a commonly used layer in deep learning, is mainly used to convert discrete symbols or categorical variables into continuous numerical vectors. This conversion helps capture the relationship between categorical variables and represents categorical variables in a high-dimensional space, so that similar categories are close to each other in the vector space.
[0077] The encoding layer is one of the key components in a neural network for processing sequence data. It is mainly used to capture the temporal dependency and contextual information in the input data. When processing sequence data such as text, speech, time series, etc., the encoding layer can effectively learn the patterns and relationships in the sequence.
[0078] Fully connected layer, the fully connected layer (Fully Connected Layer) is a basic layer in a neural network, which is very common when processing various types of input data. Each neuron in the fully connected layer is connected to all neurons in the previous layer, and is usually used to integrate the information output by the previous layer and convert it into the desired output form.
[0079] The output layer is usually the last layer in the neural network, and its main function is to generate the final prediction result.
[0080] Specifically, the data consistency verification model generates a data consistency verification result according to the configuration information and the sample data, including steps B21 to B26 as shown below:
[0081] B21. Generate a first feature vector through the input layer according to the configuration information.
[0082] In some embodiments, the first feature vector is used to represent the configuration information, and the first feature vector may include the name, type, allowed value range, etc. of the field.
[0083] B22. Generate a numerical vector corresponding to the sample data through the input layer according to the sample data.
[0084] In some embodiments, the example data may be vectorized through an input layer to obtain the numerical vector, and the numerical vector is used to characterize data features of the example data.
[0085] B23, generating a second feature vector according to the first feature vector through the embedding layer;
[0086] In some embodiments, for the text features of the configuration information, each word can be mapped to the second feature vector through the embedding layer using word embedding; for the category features of the configuration information, each category can be mapped to the second feature vector through the embedding layer using one-hot encoding or category embedding.
[0087] B24. Generate a serialized feature vector according to the numerical vector and the second feature vector through the encoding layer.
[0088] In some embodiments, the encoding layer comprises a long short-term memory network layer.
[0089] The Long Short-Term Memory (LSTM) network layer is a special type of network layer based on the recurrent neural network (RNN). It is specially designed to process and predict long-range dependencies in sequence data. It can effectively remember long-term historical information while avoiding the gradient vanishing problem in traditional RNNs.
[0090] The step of generating a serialized feature vector according to the numerical vector and the second feature vector through the encoding layer includes the following steps B241 to B242:
[0091] B241. Input the numerical vector and the second vector into the long short-term memory network layer, and output a hidden state vector and a unit state vector.
[0092] In some embodiments, the numerical vector and the second vector can be regarded as an input with a shape of (batch_size, sequence_length, input_dim), where batch_size represents the number of batch processed data, sequence_length represents the length of the sequence, and input_dim represents the dimension of the numerical vector plus the dimension of the second vector. Thus, the output hidden state vector (Hidden State Vector) is a vector output at the end of each time step, and its shape is (batch_size, hidden_units); the output cell state vector (Cell State Vector) is a vector updated at the end of each time step, and its shape is (batch_size, hidden_units), where hidden_units represents the number of hidden units of the long short-term memory network layer.
[0093] B242. Integrate the hidden state vector and the unit state vector to obtain the serialized feature vector.
[0094] Continuing with the above embodiment, the integration process can be performed in the following manner:
[0095] Pooling operation: For example, using average pooling, the shape of the serialized feature vector is (batch_size, hidden_units);
[0096] Attention mechanism: The obtained serialized feature vector is also (batch_size, hidden_units);
[0097] The last step: only use the hidden state vector of the last time step of the sequence, that is, the serialized feature vector is still (batch_size, hidden_units).
[0098] B25. Generate an output vector according to the serialized feature vector through the fully connected layer;
[0099] Continuing with the above embodiment, the serialized feature vector is input into the fully connected layer to perform key feature extraction processing to obtain an output vector of a fixed length with a shape of (batch_size, output_dim), where output_dim is the output dimension of the fully connected layer.
[0100] B26. Obtain the data consistency verification result through the output layer according to the output vector.
[0101] Continuing with the above-mentioned embodiment, through the output layer, a binary classification prediction process can be performed on the output vector, and the output result can also be compared with a preset threshold to obtain a comparison result, and then the data consistency verification result is determined based on the comparison result.
[0102] In some embodiments, obtaining the data consistency verification result according to the output vector through the output layer includes the following steps B261 to B262:
[0103] B261. Input the output vector to the output layer to obtain an output probability value, wherein the output probability value is used to characterize the possibility that the data to be evaluated meets the preset data consistency standard.
[0104] Continuing with the above embodiment, the output layer may use a Sigmoid function as an activation function to generate the output probability value representing the probability distribution.
[0105] The Sigmoid function is particularly suitable for binary classification problems and can convert the output into a probability value between 0 and 1, that is, the output probability value.
[0106] B262. If the output probability value is greater than or equal to the preset probability threshold, it is determined that the data to be evaluated meets the preset data consistency standard; if the output probability value is less than the preset probability threshold, it is determined that the data to be evaluated does not meet the preset data consistency standard.
[0107] S120, generating an evaluation program based on the data evaluation standard and the data to be evaluated;
[0108] In some embodiments, appropriate tools and technical means can be selected based on the data evaluation criteria and the data to be evaluated, such as programming languages such as Python and R, or data analysis and visualization tools such as Pandas, NumPy, Excel, etc., to write program code; the evaluation program is generated through the program code, and the evaluation program may include modules such as reading data to be evaluated, evaluation calculation, and outputting evaluation results.
[0109] S130. Based on the configuration information and the data to be evaluated, execute the evaluation program to obtain a target evaluation result, wherein the target evaluation result is used to characterize a data quality detection result of the data to be evaluated.
[0110] In some embodiments, the configuration information and the data to be evaluated may be retrieved from the database, and the data to be evaluated may be loaded in the data evaluation system based on the configuration information.
[0111] The evaluation program will read the data file corresponding to the data to be evaluated to execute each evaluation link in the data evaluation standard, and finally obtain the target evaluation result.
[0112] In some embodiments, a corresponding assessment report may be generated based on the target assessment results.
[0113] For example, the data integrity analysis section of the assessment report may include the following:
[0114] Data asset name: Equipment maintenance record table, with a total of 10,000,000 data items, "Equipment Name" data record integrity of 96%, "Maintenance Time" data record integrity of 100%, "Maintenance Content" data record integrity of 80%, and "Fault Condition" data record integrity of 72%.
[0115] In some embodiments, in order to perform encryption processing to prevent tampering, the unique identification code of the evaluated unit performing the data evaluation, the name of the evaluated unit, the list of evaluated data resources, and other information can be added to the evaluation report, and a hash code can be generated for encryption.
[0116] At the same time, the encrypted assessment report can also be transmitted to a controlled environment using a CD or other means to read the assessment results. Specifically, the remote executor uses a CD to burn the assessment report to the Internet, and transmits the results to the assessment agency's result assessor in a safe and controllable manner. After the result assessor determines that the assessment report is credible and valid, he or she can issue a stamped version of the assessment report.
[0117] As can be seen from the above, the embodiment of the present invention can first obtain the configuration information corresponding to the data to be evaluated; then, based on the data evaluation standard and the data to be evaluated, generate an evaluation program; based on the configuration information and the data to be evaluated, execute the evaluation program, and finally obtain the data quality detection result of the data to be evaluated.
[0118] Compared with the existing data quality assessment methods, the present invention can accurately obtain the configuration data of the data to be evaluated when the network cannot be connected or is in a more complex network environment, and execute the assessment program based on the data assessment standard in an offline state, accurately evaluate the data quality of the data to be evaluated, and encrypt the assessment results, etc., to ensure the security and reliability of the remote execution of data assessment results.
[0119] In order to better implement the above method, an embodiment of the present invention further provides a data evaluation device, which can be integrated in an electronic device, and the electronic device can be a terminal, a server, etc. Among them, the terminal can be a mobile phone, a tablet computer, a smart Bluetooth device, a laptop, a personal computer, etc.; the server can be a single server or a server cluster composed of multiple servers.
[0120] For example, in this embodiment, the method of the embodiment of the present invention is described in detail by taking the data evaluation device being specifically integrated in the terminal as an example.
[0121] For example, Figure 2 As shown, the data evaluation method device may include a first unit 201, a second unit 202, and a third unit 203, which are applied to a data evaluation system, wherein the data evaluation system is in a network offline state, and the data evaluation system includes a database, and the database is used for data storage in the network offline state;
[0122] The device comprises:
[0123] The first unit 201 is used to obtain configuration information corresponding to the data to be evaluated;
[0124] The second unit 202 is used to generate an evaluation program based on the data evaluation standard and the data to be evaluated;
[0125] The third unit 203 is used to execute the evaluation program based on the configuration information and the data to be evaluated to obtain a target evaluation result, wherein the target evaluation result is used to characterize a data quality detection result of the data to be evaluated.
[0126] In specific implementation, the above units can be implemented as independent entities, or can be arbitrarily combined to be implemented as the same or several entities. The specific implementation of the above units can refer to the previous method embodiments, which will not be repeated here.
[0127] It can be seen from the above that the embodiments of the present invention can accurately implement data quality assessment when the network is unavailable or in a relatively complex network environment.
[0128] The embodiment of the present invention further provides an electronic device, which may be a terminal, a server, or the like. The terminal may be a mobile phone, a tablet computer, a smart Bluetooth device, a notebook computer, a personal computer, or the like; the server may be a single server or a server cluster composed of multiple servers, or the like.
[0129] In some embodiments, the device may also be integrated into multiple electronic devices. For example, the device may be integrated into multiple servers, and the data evaluation method of the present invention may be implemented by multiple servers.
[0130] In this embodiment, the electronic device of this embodiment is a terminal as an example for detailed description, for example, Figure 3 As shown, it shows a schematic diagram of the structure of the terminal involved in the embodiment of the present invention, specifically:
[0131] The terminal may include one or more processors 301 of processing cores, one or more storage media 302, a power supply 303, an input module 304, and a communication module 305. Those skilled in the art will appreciate that Figure 3 The terminal structure shown in the figure does not constitute a limitation on the terminal, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.
[0132] The processor 301 is the control center of the terminal. It uses various interfaces and lines to connect various parts of the entire terminal. It executes various functions of the terminal and processes data by running or executing software programs and / or modules stored in the memory 302, and calling data stored in the memory 302, so as to monitor the terminal as a whole. In some embodiments, the processor 301 may include one or more processing cores; in some embodiments, the processor 301 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 301.
[0133] The memory 302 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 302. The memory 302 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 302 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 302 may also include a memory controller to provide the processor 301 with access to the memory 302.
[0134] The terminal also includes a power supply 303 for supplying power to various components. In some embodiments, the power supply 303 can be logically connected to the processor 301 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 303 can also include any components such as one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, and power status indicators.
[0135] The terminal may further include an input module 304, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls.
[0136] The terminal may also include a communication module 305. In some embodiments, the communication module 305 may include a wireless module. The terminal may perform short-range wireless transmission through the wireless module of the communication module 305, thereby providing the user with wireless broadband Internet access. For example, the communication module 305 may be used to help the user send and receive emails, browse web pages, and access streaming media.
[0137] Although not shown, the terminal may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 301 in the terminal will load the executable files corresponding to the processes of one or more applications into the memory 302 according to the following instructions, and the processor 301 will run the applications stored in the memory 302, thereby realizing various functions, as follows:
[0138] Obtain configuration information corresponding to the data to be evaluated;
[0139] Based on the data evaluation criteria and the data to be evaluated, generating an evaluation procedure;
[0140] Based on the configuration information and the data to be evaluated, the evaluation program is executed to obtain a target evaluation result, wherein the target evaluation result is used to characterize a data quality detection result of the data to be evaluated.
[0141] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.
[0142] It can be seen from the above that the embodiments of the present invention can accurately implement data quality assessment when the network is unavailable or in a relatively complex network environment.
[0143] A person skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a storage medium and loaded and executed by a processor.
[0144] To this end, an embodiment of the present invention provides a storage medium in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps in any data evaluation method provided in the embodiment of the present invention. For example, the instructions can execute the following steps:
[0145] Obtain configuration information corresponding to the data to be evaluated;
[0146] Based on the data evaluation criteria and the data to be evaluated, generating an evaluation procedure;
[0147] Based on the configuration information and the data to be evaluated, the evaluation program is executed to obtain a target evaluation result, wherein the target evaluation result is used to characterize a data quality detection result of the data to be evaluated.
[0148] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0149] According to one aspect of the present invention, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a storage medium. A processor of a computer device reads the computer instructions from the storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in the various optional implementations provided in the above embodiments.
[0150] Since the instructions stored in the storage medium can execute the steps in any data evaluation method provided in the embodiments of the present invention, the beneficial effects that can be achieved by any data evaluation method provided in the embodiments of the present invention can be achieved. Please refer to the previous embodiments for details and will not be repeated here.
[0151] The above is a detailed introduction to a data evaluation method, device, terminal and storage medium provided in an embodiment of the present invention. Specific examples are used herein to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.
Claims
1. A data evaluation method, characterized in that: Applied to a data evaluation system, the data evaluation system is in a network offline state, the data evaluation system comprises a database, and the database is used for data storage in the network offline state; The method comprises: Obtain configuration information corresponding to the data to be evaluated; Based on the data evaluation criteria and the data to be evaluated, generating an evaluation procedure; Based on the configuration information and the data to be evaluated, the evaluation program is executed to obtain a target evaluation result, wherein the target evaluation result is used to characterize a data quality detection result of the data to be evaluated.
2. The method according to claim 1, characterized in that The obtaining of configuration information corresponding to the data to be evaluated includes: Screening out the data to be evaluated from the initial data; Based on the data to be evaluated, the data evaluation system is configured to generate the configuration information, wherein the configuration information includes an evaluation object corresponding to the data to be evaluated, evaluation parameters corresponding to the data to be evaluated, and a data extraction method corresponding to the data to be evaluated.
3. The method according to claim 2, characterized in that Before generating an evaluation program based on the data evaluation criteria and the data to be evaluated, the method further includes: Establish initial data evaluation criteria; Based on the initial data evaluation standard and the initial data, the initial data evaluation standard is updated to obtain the data evaluation standard.
4. The method according to claim 2, characterized in that Before executing the evaluation program based on the configuration information and the data to be evaluated to obtain a target evaluation result, the method further includes: Filtering sample data from the data to be evaluated; Through the data consistency verification model, a data consistency verification result is generated according to the configuration information and the sample data. If the data consistency verification result is that the data to be evaluated meets the preset data consistency standard, the data to be evaluated and the configuration information are stored in the database; if the data consistency verification result is that the data to be evaluated does not meet the preset data consistency standard, the data to be evaluated is re-screened from the initial data until the data consistency verification result is that the data to be evaluated meets the preset data consistency standard.
5. The method according to claim 4, characterized in that The data consistency verification model includes an input layer, an embedding layer, an encoding layer, a fully connected layer and an output layer; The step of generating a data consistency verification result according to the configuration information and the sample data through the data consistency verification model includes: Generate a first feature vector through the input layer according to the configuration information; Generate a numerical vector corresponding to the sample data according to the sample data through the input layer; Generate a second feature vector according to the first feature vector through the embedding layer; Generate a serialized feature vector according to the numerical vector and the second feature vector through the encoding layer; Generate an output vector according to the serialized feature vector through the fully connected layer; The data consistency verification result is obtained through the output layer according to the output vector.
6. The method according to claim 5, characterized in that The encoding layer includes a long short-term memory network layer; The step of generating a serialized feature vector according to the numerical vector and the second feature vector through the encoding layer includes: Input the numerical vector and the second vector into the long short-term memory network layer, and output a hidden state vector and a unit state vector; The hidden state vector and the unit state vector are integrated to obtain the serialized feature vector.
7. The method according to claim 5, characterized in that The step of obtaining the data consistency verification result according to the output vector through the output layer includes: Inputting the output vector into the output layer to obtain an output probability value, wherein the output probability value is used to characterize the possibility that the data to be evaluated meets the preset data consistency standard; If the output probability value is greater than or equal to the preset probability threshold, it is determined that the data to be evaluated meets the preset data consistency standard; if the output probability value is less than the preset probability threshold, it is determined that the data to be evaluated does not meet the preset data consistency standard.
8. A data evaluation device, characterized in that: Applied to a data evaluation system, the data evaluation system is in a network offline state, the data evaluation system comprises a database, and the database is used for data storage in the network offline state; The device comprises: The first unit is used to obtain configuration information corresponding to the data to be evaluated; A second unit is used to generate an evaluation program based on the data evaluation standard and the data to be evaluated; The third unit is used to execute the evaluation program based on the configuration information and the data to be evaluated to obtain a target evaluation result, wherein the target evaluation result is used to characterize a data quality detection result of the data to be evaluated.
9. A terminal, characterized in that: The method comprises a processor and a memory, wherein the memory stores a plurality of instructions; the processor loads instructions from the memory to execute the steps in the method according to any one of claims 1 to 7.
10. A storage medium, characterized in that: The storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the method according to any one of claims 1 to 7.