Data quality improvement method, related apparatus and medium

Abstract

Embodiments of the present disclosure provide a data quality improvement method, related apparatus and medium. The method comprises: inputting to-be-improved data into a target neural network model to obtain an output distance; generating intermediate data according to the output distance, the to-be-improved data, first gradient information and a pre-set gradient descent rate hyperparameter; updating the to-be-improved data according to the intermediate data, and returning the step of inputting the to-be-improved data into the target neural network model to obtain the output distance, so as to iteratively update the to-be-improved data until a pre-set iteration stop condition is met, and taking the to-be-improved data meeting the pre-set iteration stop condition as target data. Embodiments of the present disclosure can improve the reliability of data quality improvement processing and improve the final data quality. Embodiments of the present disclosure can be applied to image processing, digital human rendering, industrial anomaly detection and the like.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of data processing, and particularly to a data quality improvement method, related apparatus and medium. BACKGROUND

[0002] In the fields of image recognition, 3D portrait technology, industrial error detection, etc., a deep learning model is usually used for data processing to optimize images or portraits or to improve the accuracy of industrial error detection. In this process, the deep learning model learns from a large amount of input data to optimize the model output. However, real data is often affected by various problems such as noise, missing values, etc., resulting in low data quality.

[0003] In related technologies, pre-set rules, statistical methods, etc. are usually used to improve data quality, but these methods do not work well for problems such as noise and missing values in the data, which limits the performance of the model. How to improve the reliability of data quality improvement processing is a problem that needs to be discussed. SUMMARY

[0004] Embodiments of the present disclosure provide a data quality improvement method, related apparatus and medium, which aims to improve the reliability of data quality improvement processing and improve the final data quality.

[0005] In a first aspect, embodiments of the present disclosure provide a data quality improvement method, which comprises:

[0006] inputting the data to be improved into a target neural network model to obtain an output distance, wherein the output distance represents the minimum distance between the data to be improved and an implicit manifold, and the implicit manifold is obtained based on a high-quality data set;

[0007] generating intermediate data according to the output distance, the data to be improved, first gradient information and a pre-set gradient descent rate hyperparameter, wherein the first gradient information is used to represent the direction of the data to be improved to the nearest data point on the implicit manifold;

[0008] updating the data to be improved according to the intermediate data, returning to the step of inputting the data to be improved into the target neural network model to obtain the output distance, to iteratively update the data to be improved until a pre-set iteration stopping condition is met, and taking the data to be improved that meets the pre-set iteration stopping condition as target data.

[0009] In a second aspect, embodiments of the present disclosure provide a data quality improvement apparatus, which comprises:

[0010] an output distance calculation unit, configured to input the to-be-promoted data into a target neural network model to obtain an output distance, wherein the output distance represents a minimum distance between the to-be-promoted data and an implicit manifold, and the implicit manifold is obtained based on a high-quality data set;

[0011] an intermediate data calculation unit, configured to generate intermediate data according to the output distance, the to-be-promoted data, first gradient information, and a preset gradient descent rate hyperparameter, wherein the first gradient information is used to represent a direction of the to-be-promoted data to a nearest data point on the implicit manifold;

[0012] a target data determination unit, configured to update the to-be-promoted data according to the intermediate data, and send the to-be-promoted data to the output distance calculation unit to iteratively update the to-be-promoted data until a preset iteration stopping condition is met, and take the to-be-promoted data that meets the preset iteration stopping condition as target data.

[0013] Optionally, the data quality promotion apparatus further includes:

[0014] a training data construction unit, configured to obtain an original data set, and construct a training data set according to the original data set, wherein the training data set includes a plurality of high-quality data and a plurality of low-quality data;

[0015] a manifold construction unit, configured to construct the implicit manifold according to the plurality of high-quality data, wherein a first distance between each high-quality data and the implicit manifold is 0;

[0016] a first calculation unit, configured to calculate a second distance between each low-quality data and the implicit manifold, wherein the second distance is greater than 0;

[0017] a neural network training unit, configured to input the training data set, the first distance, and the second distance into a preset neural network model, train the preset neural network model, and obtain the target neural network model.

[0018] Optionally, the training data construction unit is specifically configured to:

[0019] obtain the original data set, wherein the original data set is a high-quality data set corresponding to the to-be-promoted data;

[0020] perform noise adding processing on the high-quality data set to obtain a low-quality data set;

[0021] merge the high-quality data set and the low-quality data set to obtain the training data set.

[0022] Optionally, the first calculation unit is specifically configured to:

[0023] For each low-quality data, a plurality of nearest neighbor data closest to the low-quality data is determined from the set of high-quality data by a K-nearest neighbor algorithm;

[0024] A distance value between the low-quality data and each of the nearest neighbor data is determined.

[0025] The second distance of the low-quality data is obtained according to an average of the plurality of distance values.

[0026] Optionally, the neural network training unit is specifically configured to:

[0027] A first loss function of the pre-set neural network model is constructed according to the output distance corresponding to each of the high-quality data and the first distance corresponding to each of the high-quality data, and the output distance corresponding to each of the low-quality data and the second distance corresponding to each of the low-quality data.

[0028] Optionally, the neural network training unit is specifically configured to:

[0029] Second gradient information of each of the low-quality data is determined according to the derivative of each of the low-quality data and the output distance corresponding to the low-quality data.

[0030] A second loss function of the pre-set neural network model is constructed according to the second gradient information of each of the low-quality data.

[0031] Optionally, the intermediate data calculation unit is specifically configured to:

[0032] The first gradient information is determined according to the derivative of the to-be-promoted data and the output distance.

[0033] A gradient descent value is determined according to a product of the first gradient information, the gradient descent rate hyperparameter and the output distance.

[0034] The intermediate data is determined according to a difference between the to-be-promoted data and the gradient descent value.

[0035] In a third aspect, an electronic device is provided, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the data quality improvement method in the first aspect when executing the computer program.

[0036] In a fourth aspect, a computer readable storage medium is provided, which stores computer executable instructions, and the computer executable instructions implement the data quality improvement method in the first aspect when executed by a computer.

[0037] In a fifth aspect, the embodiments of the present disclosure provide a computer program product, including a computer program or computer instructions, the computer program or the computer instructions are stored in a computer readable storage medium, a processor of a computer device reads the computer program or the computer instructions from the computer readable storage medium, and the processor executes the computer program or the computer instructions, so that the computer device executes the data quality improvement method in the first aspect.

[0038] The implicit manifold in the embodiments of the present disclosure is constructed by high-quality data corresponding to target data. By the distance between the data to be improved and the implicit manifold, the data that needs to be improved in quality can be accurately obtained (the output distance is greater than 0), and the distance to be improved can be approximated to the implicit manifold based on the distance, so as to obtain the target data corresponding to the data to be improved, thereby realizing data quality improvement. Since the implicit manifold is usually irregular, the distance relationship between the data points and the implicit manifold is relatively complex, and the calculation amount of directly calculating the distance relationship between the data points and the implicit manifold is very large. The present disclosure uses the target neural network model that fits the implicit manifold to output the output distance of the data to be improved, and the output distance is the distance relationship between the data to be improved and the implicit manifold. By fitting the implicit manifold through the neural network model, each data to be improved can be input into the target neural network model to output the distance relationship between the data to be improved and the implicit manifold more accurately. Then, the data to be improved is updated to approximate the implicit manifold based on the output distance output by the model in the gradient descent iteration mode, so as to obtain the corresponding target data. Since the output distance used in each iteration is obtained through the target neural network model, the calculation amount in the iteration approximation process is reduced, and the distance relationship between the data to be improved and the implicit manifold in each iteration update can be accurately obtained through the target neural network model, so that the target data obtained through iteration is more accurate. Finally, the reliability of the data quality improvement process is improved, and the final data quality is improved.

[0039] Other features and advantages of the present disclosure will be set forth in the description that follows, and in part will become apparent to those skilled in the art upon examination of the description, or be learned by practice of the present disclosure. The objects and other advantages of the present disclosure can be realized and attained by the structure particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 System architecture diagram applied to the data quality improvement method according to the embodiments of the present disclosure;

[0041] Figure 2 Schematic diagram applied in the digital human rendering scene according to the embodiments of the present disclosure;

[0042] Figure 3A flowchart of a data quality improvement method according to one embodiment of the present disclosure;

[0043] Figure 4 A flowchart of a data quality improvement method according to one embodiment of the present disclosure; Figure 3 A flowchart of a training target neural network model before step 310 in the method of the present disclosure;

[0044] Figure 5 A schematic diagram of an unsigned distance field according to one embodiment of the present disclosure;

[0045] Figure 6 A flowchart of a training target neural network model according to one embodiment of the present disclosure; Figure 4 A flowchart of a training target neural network model according to one embodiment of the present disclosure;

[0046] Figure 7 A flowchart of a training target neural network model according to one embodiment of the present disclosure; Figure 4 A flowchart of a training target neural network model according to one embodiment of the present disclosure;

[0047] Figure 8 A flowchart of a training target neural network model according to one embodiment of the present disclosure; Figure 7 A flowchart of a training target neural network model according to one embodiment of the present disclosure;

[0048] Figure 9 A flowchart of a training target neural network model according to one embodiment of the present disclosure; Figure 4 A flowchart of a training target neural network model according to one embodiment of the present disclosure;

[0049] Figure 10 A flowchart of a training target neural network model according to one embodiment of the present disclosure;

[0050] Figure 11 A flowchart of a training target neural network model according to one embodiment of the present disclosure; Figure 3 A flowchart of a training target neural network model according to one embodiment of the present disclosure;

[0051] Figure 12 A flowchart of a training target neural network model according to one embodiment of the present disclosure;

[0052] Figure 13 A flowchart of a training target neural network model according to one embodiment of the present disclosure;

[0053] Figure 14 A flowchart of a training target neural network model according to one embodiment of the present disclosure;

[0054] Figure 15 A flowchart of a training target neural network model according to one embodiment of the present disclosure; Figure 4 A flowchart of a training target neural network model according to one embodiment of the present disclosure;

[0055] Figure 16 A flowchart of a training target neural network model according to one embodiment of the present disclosure; Figure 4 A flowchart of a training target neural network model according to one embodiment of the present disclosure; DETAILED DESCRIPTION

[0056] In order to make the purposes, technical methods and advantages of the present disclosure clearer, the present disclosure will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present disclosure and should not be used to limit the present disclosure.

[0057] It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described can be performed in an order different from that in the flowchart. The terms "first", "second", and the like in the specification and claims and the above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0058] In the description of the embodiments of the present disclosure, unless otherwise explicitly limited, the words such as arrangement, installation, connection and the like should be broadly understood, and those skilled in the art can reasonably determine the specific meanings of the above words in the embodiments of the present disclosure in combination with the specific content of the technical solutions. In the embodiments of the present disclosure, the words "further", "exemplarily" or "optionally" are used to represent as an example, illustration or description, and should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. The use of the words "further", "exemplarily" or "optionally" is intended to present the relevant concept in a specific way.

[0059] Before the embodiments of the present disclosure are further described, the terms and terms involved in the embodiments of the present disclosure are explained, and the terms and terms involved in the embodiments of the present disclosure are applicable to the following explanations:

[0060] Artificial intelligence (AI): is a new technical science of studying, developing theories, methods, technologies and application systems for simulating, extending and expanding human intelligence; artificial intelligence is a branch of computer science, and artificial intelligence attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. The research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. Artificial intelligence can simulate the information process of human consciousness and thinking. Artificial intelligence is also the theory, method, technology and application system of using digital computers or digital computer-controlled machines to simulate, extend and expand human intelligence, to perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0061] Deep Learning (DL): is a new research direction in the field of Machine Learning (ML), which is introduced into machine learning to make it closer to the original goal - Artificial Intelligence (AI). Deep learning is to learn the internal rules and representation levels of sample data, and the information obtained in the learning process is very helpful for the interpretation of data such as text, images and sound. Its ultimate goal is to enable machines to have analysis and learning ability like people, and to recognize text, images and sound data.

[0062] Artificial Neural Network (ANN): is an abstraction of human brain neuron network from the perspective of information processing, which establishes a certain simple model and forms different networks according to different connection modes. It is also called neural network or neural network. Neural network is an operation model composed of a large number of nodes (or neurons) connected with each other. Each node represents a specific output function, called activation function. The connection between each two nodes represents a weight value for the signal passing through the connection, called weight, which is equivalent to the memory of artificial neural network. The output of the network depends on the connection mode of the network, the weight value and the excitation function. The network itself is usually an approximation of some algorithm or function in nature, or an expression of a certain logic strategy.

[0063] Neural Unsigned Distance Fields (NUDF): is a neural network model used to represent three-dimensional shapes. It is an unsigned distance field, which means it can represent the distance from the surface of an object to any point, not just the distance inside or outside the object. The input of NUDF is a point in three-dimensional space, and the output is the distance from these points to the nearest surface. This representation method can be used to generate fine three-dimensional shapes, such as three-dimensional modeling in computer graphics or object surface reconstruction in virtual reality.

[0064] Processing unit: processing unit is a unit for processing data, which includes Central Processing Unit (CPU), Graphics Processing Unit (GPU), Tensor Processing Unit (TPU) and Neural network Processing Unit (NPU).

[0065] In the fields of image recognition, 3D portrait technology, and industrial error detection, deep learning models are usually used for data processing to optimize images or portraits or to improve the accuracy of industrial error detection. In this process, the deep learning model learns from a large amount of input data to optimize the model output. However, real data is often affected by various problems such as noise, missing values, etc., resulting in low data quality. In related technologies, pre-set rules, statistical methods, etc. are usually used to improve data quality, but these methods are not effective in dealing with noise, missing values, etc. in the data, limiting the performance of the model.

[0066] Based on this, the embodiments of the present disclosure provide a data quality improvement method, related devices and media. The data quality improvement method provided by the embodiments of the present disclosure obtains the distance relationship between the data to be improved and the implicit manifold by using the target neural network model fitted with the implicit manifold corresponding to the target data, and iteratively updates the data to be improved by gradient descent to approximate the implicit manifold, finally obtaining the target data, which can improve the reliability of data quality improvement processing and improve the final data quality.

[0067] System architecture and scenario description applied by the embodiments of the present disclosure

[0068] Figure 1 It is a system architecture diagram applied by the data quality improvement method according to the embodiments of the present disclosure. It includes object terminal 140, Internet 130, gateway 120, server 110, etc.

[0069] The server 110 refers to a computer system that can provide data quality improvement services and target neural network model training services to the object terminal 140. Compared with the object terminal 140, the server 110 requires higher stability, security, performance, etc. The server 110 can also be a database that can provide data acquisition services to the object terminal 140. The server 110 can be a high-performance computer in a network platform, a cluster of multiple high-performance computers, a portion of a high-performance computer (such as a virtual machine), a combination of a portion of multiple high-performance computers (such as a virtual machine), etc. The server 110 can also communicate with the Internet 130 in a wired or wireless manner to exchange data.

[0070] The gateway 120 is also called an internetworking connector or a protocol converter. The gateway implements network interconnection at the transport layer and is a computer system or device that acts as a conversion device. In two systems that use different communication protocols, data formats or languages, or even have completely different architectures, the gateway is a translator. At the same time, the gateway can also provide filtering and security functions. The message sent by the object terminal 140 to the server 110 is sent to the corresponding server 110 through the gateway 120. The message sent by the server 110 to the object terminal 140 is also sent to the corresponding object terminal 140 through the gateway 120.

[0071] The object terminal 140 is a device for displaying the results of data quality improvement and / or target neural network model training for the object to view. The object can input the data set that needs to be improved in data quality and / or the neural network model that is expected to be trained in the object terminal 140. The object terminal 140 includes desktop computers, laptop computers, PDAs (personal digital assistants), mobile phones, car terminals, home theater terminals, dedicated terminals, digitizing tablets, and various forms. In addition, it can be a single device or a collection of multiple devices. For example, multiple devices are connected through a local area network, share a display device for collaborative work, and together constitute a terminal. The terminal can also communicate with the Internet 130 in a wired or wireless manner to exchange data.

[0072] In the process of model training and data quality improvement, it can be performed by the server 110 alone, and the object terminal 140 obtains and displays the results of the execution of the server 110; it can also be performed by the server 110 and the object terminal 140 together; or it can be performed by the server 110 providing the required data to the object terminal 140, and then the object terminal 140 performs model training and data quality improvement.

[0073] The embodiments of the present disclosure can be applied in various scenarios, for example Figure 2 The application in the scenario of digital human rendering is shown in the schematic diagram.

[0074] As shown in Figure 2 In the scenario of digital human rendering, the data used for rendering is often affected by various problems, such as noise, missing values, etc., resulting in low data quality, so data quality improvement is needed for the data used for rendering. In the embodiments of the present disclosure, first, a high-quality digital human limb action data set without noise, missing values, etc. is selected from a digital human limb action database. The high-quality digital human limb action data set is input into a training data construction module.

[0075] The training data construction module obtains a low-quality digital human limb action data set by adding noise to a high-quality digital human limb action data set. The training data construction module constructs an implicit manifold based on the high-quality digital human limb action data set, that is, the high-quality digital human limb action data is located on the implicit manifold, and the distance between the high-quality digital human limb action data and the implicit manifold is 0. The training data construction module calculates the distance between each low-quality digital human limb action data and the implicit manifold, and the distance between the low-quality digital human limb action data and the implicit manifold is greater than 0. The training data construction module inputs the high-quality digital human limb action data set, the low-quality digital human limb action data set, and the distance between each data and the implicit manifold as training data samples into the neural network model.

[0076] The neural network model is trained based on the high-quality digital human limb action data set, the low-quality digital human limb action data set, and the distance between each data and the implicit manifold, and fits an unsigned distance field corresponding to the implicit manifold. After training, the neural network model can output the distance between the input data and the implicit manifold. The specific neural network model can be selected according to the data to be processed. In the embodiments of the present disclosure, a Multilayer Perceptron (MLP) model is selected as an example.

[0077] The digital human limb action data currently required for digital human rendering is input into the trained neural network model. The trained neural network model outputs the distance corresponding to each data, wherein the digital human limb action data with a distance of 0 belongs to high-quality data and does not need to be improved in data quality, and the digital human limb action data with a distance greater than 0 has problems and needs to be improved in data quality.

[0078] For the digital human limb action data that needs to be improved in data quality, the data quality improvement module updates the digital human limb action data to approximate the implicit manifold based on the distance of the digital human limb action data output by the trained neural network model, the digital human limb action data, the first gradient information, and the gradient descent rate hyperparameter of the preset value. The data quality improvement module inputs the updated digital human limb action data into the trained neural network model for training to obtain the distance of the updated digital human limb action data and performs updating and approximation again. Until the number of iterations is satisfied, or the distance output by the model is less than the preset value, it is indicated that the digital human limb action data has been updated to high-quality data (the distance between the digital human limb action data and the implicit manifold approximates to 0).

[0079] The data quality improvement module sends the digital human limb action data improved in data quality to the digital human rendering module for rendering and display.

[0080] In summary, the embodiment of the present disclosure constructs a training sample by using high-quality digital human limb motion data, trains a neural network model, and makes the neural network model fit the unsigned distance field corresponding to the high-quality digital human limb motion data. Thus, the trained neural network model can be used to output the distance between the current digital human limb motion data and the implicit manifold corresponding to the high-quality data. The distance relationship between the current digital human limb motion data and the implicit manifold can more accurately represent the relationship between the current digital human limb motion data and the high-quality data. Based on the gradient descent method, the digital human limb motion data is iteratively updated to approach the implicit manifold (high-quality data), and finally the digital human limb motion data is updated to the corresponding high-quality digital human limb motion data, so as to improve the data quality. Since the trained neural network model can accurately obtain the distance relationship between the digital human limb motion data updated at each iteration and the implicit manifold, the digital human limb motion data obtained through iteration is more accurate. Finally, the reliability of the data quality improvement process is improved, and the final data quality is improved.

[0081] It should be understood that the above only illustrates the description of part of the application scenarios of the present disclosure. The business scenarios to which the present disclosure can be applied can include but are not limited to the specific embodiments described above.

[0082] Overall description of the embodiment of the present disclosure

[0083] It should be emphasized that the embodiment of the present disclosure can be applied to various application scenarios, such as graphics processing, image recognition, digital human rendering, and factory anomaly detection. In related technologies, deep learning models are usually used in the above scenarios for related data processing. The deep learning model can use a large amount of input data for self-learning and optimize the model output. However, real data is often affected by various problems, such as noise and missing values, resulting in low data quality. Some embodiments of the present disclosure provide a data quality improvement method, related device and medium, which aims to improve the reliability of the data quality improvement process and improve the final data quality.

[0084] The data quality improvement method is a method for improving or repairing the quality problems of data before using the data in a scenario that requires a large amount of data for related processing or analysis.

[0085] The data quality improvement method of the embodiment of the present disclosure can be executed on a server, or on a terminal, or partially on a server and partially on a terminal.

[0086] As shown in Figure 3 According to one embodiment of the present disclosure, the data quality improvement method comprises:

[0087] In step 310, the to-be-promoted data is input into the target neural network model to obtain an output distance, wherein the output distance represents the minimum distance between the to-be-promoted data and an implicit manifold, and the implicit manifold is obtained based on a high-quality data set.

[0088] In step 320, intermediate data is generated according to the output distance, the to-be-promoted data, first gradient information, and a pre-set gradient descent rate hyperparameter, wherein the first gradient information is used to represent the direction of the to-be-promoted data to the nearest data point on the implicit manifold.

[0089] In step 330, the to-be-promoted data is updated according to the intermediate data, and the step of inputting the to-be-promoted data into the target neural network model to obtain the output distance is returned, so as to iteratively update the to-be-promoted data until a pre-set iteration stopping condition is met, and the to-be-promoted data meeting the pre-set iteration stopping condition is taken as the target data.

[0090] The steps 310 to 330 are briefly described below.

[0091] In step 310, the to-be-promoted data refers to data that may have data quality problems and needs to be promoted in data quality. The data quality problems include missing values, abnormal values, inconsistency, etc.

[0092] The target neural network model is a neural network model obtained by training a pre-set neural network using training samples constructed from existing data in the corresponding field, so as to fit the implicit manifold. The neural network model can be selected according to the data in the corresponding field, for example, a Multilayer Perceptron (MLP) model.

[0093] The output distance refers to the distance output by the target neural network model when the to-be-promoted data is input into the target neural network model. The distance is the distance between the input to-be-promoted function and the implicit manifold fitted in the training process of the target neural network model.

[0094] The implicit manifold refers to an implicit surface, which is represented by a distance function of a data point in space to the surface. In the present disclosure, the implicit manifold is obtained based on a high-quality data set, that is, the high-quality data in the data in the corresponding field are all on the implicit manifold, and the distance between the high-quality data points and the corresponding implicit manifold should be 0. The high-quality data set refers to a set of data without data quality problems such as missing values, abnormal values, and inconsistency.

[0095] The data in the corresponding field refers to the data in the current application scenario. For example, in the digital human rendering scenario, the data in the corresponding field is the digital human limb motion data; in the image processing scenario, the data in the corresponding field is the image feature data; and in the industrial anomaly detection scenario, the data in the corresponding field is the abnormal feature data.

[0096] In step 320, the first gradient information represents the direction of the data to be promoted to the nearest data point on the implicit manifold, and the direction of the data to be promoted when performing gradient descent iteration can be controlled through the first gradient information, so that the intermediate data can be closer to the implicit manifold. The gradient descent rate hyperparameter refers to a parameter for controlling the change rate of the data to be promoted when performing gradient descent iteration. The intermediate data refers to data that is closer to the implicit manifold based on the data to be promoted, that is, the data to be promoted after one iteration update.

[0097] In step 330, the target data refers to data with improved data quality obtained through data update iteration of the data to be promoted. The target data is more accurate, complete, consistent and reliable than the original data to be promoted, and subsequent processing using the target data can improve the reliability and effectiveness of data analysis and decision-making. For example, in the digital human rendering scene, before rendering, the data to be used for rendering is obtained through steps 310 to 330 to obtain the corresponding target data, and the target data is used for rendering, so that the digital human limb movement presented can be more smooth and natural.

[0098] The iteration stop condition refers to a condition that can be used as a judgment basis for whether the data to be promoted after iteration update meets the expectation and the data quality has been improved.

[0099] In an example, the iteration stop condition is that the output distance is less than a preset distance threshold. That is, the data to be promoted obtained after the current update is input into the target neural network model, and if the obtained output distance is less than the preset distance threshold, it means that the data to be promoted is already on the implicit manifold or is close enough to the implicit manifold, and it can be considered that the high-quality data point corresponding to the data to be promoted has been found on the implicit manifold, and the data to be promoted is the target data required. If the obtained output distance is greater than the preset distance threshold, the gradient descent update calculation is continued for the data to be promoted to obtain a new data to be promoted that is closer to the implicit manifold.

[0100] In another example, the iteration stop condition is that the number of updates for the data to be promoted is greater than a preset frequency threshold. That is, the number of times of gradient descent iteration update calculation (step 320) for the data to be promoted has reached the target number, and it can be determined that the data to be promoted obtained after the last update calculation is already on the implicit manifold or is close enough to the implicit manifold, and it can be considered that the high-quality data point corresponding to the data to be promoted has been found on the implicit manifold, and the data to be promoted is the target data required.

[0101] The embodiments of steps 310 to 330 can accurately obtain data (output distance greater than 0) that needs to be improved in quality by the distance between the data to be improved and the implicit manifold, and can approximate the distance to be improved to the implicit manifold based on the distance to obtain the target data corresponding to the data to be improved, thereby realizing data quality improvement. Since the implicit manifold is usually irregular, the distance relationship between the data points and the implicit manifold is relatively complex, and the calculation amount of directly calculating the distance relationship between the data points and the implicit manifold is very large. The output distance of the target neural network model fitted with the implicit manifold is used to output the output distance of the data to be improved, that is, the distance relationship between the data to be improved and the implicit manifold. By fitting the implicit manifold with the neural network model, the distance relationship between the data to be improved and the implicit manifold can be more accurately output by inputting each data to be improved into the target neural network model. Then, the output distance output by the model is used to update the data to be improved to approximate the implicit manifold by using the gradient descent iteration method, to obtain the corresponding target data. Since the output distance used in each iteration is obtained by the target neural network model, the calculation amount in the iteration approximation process is reduced, and the distance relationship between the data to be improved and the implicit manifold in each iteration update can be accurately obtained by the target neural network model, so that the target data obtained by iteration is more accurate. Finally, the reliability of the data quality improvement process is improved, and the final data quality is improved.

[0102] The above is a general description of steps 310 to 330, and the specific implementation process of steps 310 and 320 is described in detail below.

[0103] In step 310, the data to be improved is input into the target neural network model to obtain the output distance, wherein the output distance represents the minimum distance between the data to be improved and the implicit manifold, and the implicit manifold is obtained based on the high-quality data set.

[0104] In an embodiment, referring to Figure 4 Before step 310, the data quality improvement method further includes:

[0105] Step 410, obtaining an original data set, and constructing a training data set according to the original data set, wherein the training data set includes a plurality of high-quality data and a plurality of low-quality data;

[0106] Step 420, constructing an implicit manifold according to the plurality of high-quality data, wherein the first distance between each high-quality data and the implicit manifold is 0;

[0107] Step 430, calculating the second distance between each low-quality data and the implicit manifold, wherein the second distance is greater than 0;

[0108] Step 440, input the training data set, the first distance and the second distance into the pre-set neural network model, train the pre-set neural network model, and obtain a target neural network model.

[0109] In step 410, the original data set refers to a set of data selected from a database in a field to which the application scenario belongs for neural network model training. The training data set refers to a set of data obtained based on the original data set, which can be directly obtained from the original data set or obtained after processing the original data set. The high-quality data refers to data without data quality problems such as missing values, outliers, and inconsistencies. Correspondingly, the low-quality data refers to data with at least one data quality problem such as missing values, outliers, and inconsistencies.

[0110] In an example, when selecting data in the original data set, a plurality of high-quality data and a plurality of low-quality data in the database are selected respectively to obtain an original data set containing high-quality and low-quality data. The original data set is directly used as the training data set or is processed to obtain the training data set, so that the training data set can include high-quality data and low-quality data.

[0111] In another example, when selecting data in the original data set, all high-quality data in the database is selected. Corresponding low-quality data is obtained by processing the high-quality data. The high-quality data of the original data set and the processed low-quality data are integrated to construct the training data set, so that the training data set can include high-quality data and low-quality data.

[0112] In step 420, the specific concept of the implicit manifold is as follows:

[0113] A signed distance field (SDF) is often used to construct an implicit surface, that is, an implicit manifold. The representation of the implicit surface is defined as a distance function, that is, a distance function of a point in space to the surface. For example, the distance function of a sphere can be represented as f(x) = |x-c|-r, where x is any point in space, c is the center of the sphere, and r is the radius of the sphere. According to the function relationship, when the point is inside the sphere, the distance f(x) < 0; when the point is on the surface of the sphere, the distance f(x) = 0; and when the point is outside the sphere, the distance f(x) > 0. Any point in space has a corresponding distance to the surface. This distance function allows the definition method of the implicit surface with negative values to be called a signed distance field.

[0114] In the embodiments of the present disclosure, however, an unsigned distance field (UDF) is adopted. The UDF ignores the directionality of the implicit surface, and when defining a surface, its distance function f(x) is the absolute value of the distance from a point in space to the surface. As shown in Figure 5 , when the point is on the represented surface in space, its distance function f(x) = 0, and when the point is not on the surface, f(x) > 0 (for example, f(x) = 0.8 and f(x) = 1.0 in the figure).

[0115] As shown in Figure 5 , in the unsigned distance field, the value of f(x) is the shortest distance from the selected data point to the implicit manifold, that is, the length of the perpendicular line from the data point to the implicit manifold. The gradient direction of f(x) is the direction in which the function value changes most rapidly, so the gradient direction is the normal vector direction of the implicit manifold, which can be represented as:

[0116]

[0117] where represents the gradient vector of f(x).

[0118] "Constructing an implicit manifold according to multiple high-quality data" means that each high-quality data is arranged on the implicit manifold, that is, the distance between each high-quality data and the implicit manifold is 0. The implicit manifold is represented by the first distance (f(x) = 0) between each high-quality data and the implicit manifold.

[0119] In step 430, the second distance refers to the minimum distance between the low-quality data point and the implicit manifold. Since the unsigned distance field is adopted in the embodiments of the present disclosure, the second distance is greater than 0.

[0120] In step 440, each high-quality data of the training data set and its respective corresponding first distance, each low-quality data of the training data set and its respective corresponding second distance, together constitute an unsigned distance field. These data and corresponding distances are input into the pre-set neural network model for training, so that the neural network model learns the distance function of the data to the implicit manifold, to realize the fitting of the neural network model and the unsigned distance field of the tone, and finally obtain the target neural network model.

[0121] Referring to Figure 6 , the process of training the target neural network model is explained and described in combination with Figure 6 . In order to train the neural network model, each data point in the training data set P is needed, including the set M established using the high-quality data and the set N constructed using the low-quality data outside the implicit manifold. Each data point x e P used for training has a corresponding distance D e R *For each high-quality data x e M, its first distance D = 0 because it is located on the defined likelihood data manifold; and for each low-quality data x e N, its second distance D > 0. In summary, the training data for training the neural network model is a set P = {(x i ,D i )} 1≤i≤n , where n is the total number of data in the training data set. The set P is input into the pre-set neural network model for deep learning. Finally, the target neural network model is obtained. The data that needs to be improved in data quality is input into the target neural network model, and the distance between the data and the implicit manifold learned by the target neural network model can be obtained.

[0122] Since the implicit manifold is usually irregular, the distance relationship between the data points and the implicit manifold is relatively complex, and the calculation amount of directly calculating the distance relationship between the data points and the implicit manifold is very large, and the accuracy of the result cannot be guaranteed. The steps 410 to 440 described above train the neural network model by constructing a training sample pair containing an unsigned distance field. The neural network model is fitted to the implicit manifold, and each data to be improved is input into the target neural network model to output a more accurate distance relationship between the data to be improved and the implicit manifold.

[0123] In step 410, an original data set is obtained, and a training data set is constructed according to the original data set, wherein the training data set includes a plurality of high-quality data and a plurality of low-quality data.

[0124] In an embodiment, for some scenarios, according to the requirements of the scene, it is possible to perform data processing on the data in the original data set before constructing the training data set. For example, in an image processing scenario, if the data in the obtained original data set is original image data, it is necessary to first perform feature extraction on the original image data to obtain image feature data. The training data set is constructed based on the data set composed of the image feature data.

[0125] In an embodiment, with reference to Figure 7 , step 410 includes:

[0126] Step 710, obtaining an original data set, wherein the original data set is a high-quality data set corresponding to the data to be improved;

[0127] Step 720, performing noise adding processing on the high-quality data set to obtain a low-quality data set;

[0128] Step 730, merging the high-quality data set and the low-quality data set to obtain a training data set.

[0129] In step 720, the noise adding process refers to adding noise to the data in the high-quality data set, so that the high-quality data generates data quality problems and is converted into corresponding low-quality data.

[0130] Referring to Figure 8 , in combination Figure 8 , the process of constructing the training data set is explained and described. The training data construction mainly constructs the neural network training data set, including the collection of high-quality data, and the generation of low-quality data by adding noise, and the flow is as shown in Figure 8 The high-quality data set of the target field is obtained from the network, for example, applied in the digital human rendering scene, and the target field is digital human, and the obtained data is digital human limb action data. The high-quality data set is screened from the network large database, and there is no noise, missing value and other quality problems, and it is comprehensive and can represent the distribution characteristics of the data in this field. Here, the high-quality data is labeled as: x e M, where M represents the set of all high-quality data;

[0131] Gaussian noise ε is added to each high-quality data, and the mean of the noise is 0, and the variance σ depends on the specific data distribution. Here, the low-quality data is labeled as: y e N, where N represents the set of all low-quality data. Exemplarily, by adding multiple Gaussian noises with different variances, low-quality data with a wider distribution is obtained, that is:

[0132]

[0133] Integrate the high-quality data and the low-quality data to obtain the final training data set, P = M U N.

[0134] If the high-quality data and the low-quality data are directly selected from the database, the relative relationship between the high-quality data and the low-quality data may not be obvious, and the difficulty of data selection is increased. The above steps 710 to 730 only select high-quality data from the database, and obtain the quality data by adding noise to the high-quality data. The relative relationship between the high-quality data and the low-quality data obtained in this way is more clear, which is more conducive to subsequent training of the neural network model.

[0135] In step 430, the second distance of each low-quality data to the implicit manifold is calculated, wherein the second distance is greater than 0.

[0136] In an embodiment, referring to Figure 9 , step 430 includes:

[0137] Step 910, for each low-quality data, a plurality of nearest neighbor data closest to the low-quality data is determined from the high-quality data set by a K-nearest neighbor algorithm.

[0138] Step 920, determine the distance value between the low-quality data and each neighbor data;

[0139] Step 930, obtain the second distance of the low-quality data according to the average value of the plurality of distance values.

[0140] In step 910, the K-Nearest Neighbor (KNN) algorithm refers to an instance-based learning algorithm for classification and regression problems. The algorithm principle is that if the majority of the k most similar (i.e. the nearest neighbor in the feature space) samples in the feature space belong to a certain class, then the sample also belongs to this class. Therefore, by K-Nearest Neighbor algorithm, a plurality of high-quality data closest to the low-quality data can be obtained. The neighbor data is the high-quality data in the high-quality data set closest to the low-quality data.

[0141] In step 920, the specific calculation formula of the distance value between the low-quality data and the neighbor data is as follows:

[0142] d j =||y-x j ||2

[0143] Where d j is the distance value between the low-quality data and the jth neighbor data (the nearest high-quality data), y is the low-quality data, and x j is the jth neighbor data.

[0144] In step 930, the calculation formula of the second distance is as follows:

[0145]

[0146] Where D is the second distance, and k is the total number of neighbor data. By calculating the average value of the distance value between the low-quality data and each neighbor data, the average value is taken as the approximate value of the minimum distance between the low-quality data and the implicit manifold.

[0147] Referring to Figure 10 , the training data set is data-labeled by the data labeling module. The data of the training data set is input into the data labeling module in sequence, and it is judged whether the input data is high-quality data. If it is high-quality data, it is determined that the distance value D1 of the first distance corresponding to the high-quality data is 0. If it is low-quality data, the distance value D2 of the second distance of the low-quality data is calculated by the above steps 910 to 930. After each data of the training data set obtains its own distance value, the data labeling is completed, and the training data set P={(x i ,D i )} 1≤i≤n is obtained.

[0148] The steps 910 to 930 determine a plurality of high-quality data closest to the low-quality data by using the K-Nearest Neighbor algorithm, and use the average of the plurality of distance values as the second distance. In this way, the second distance is closer to the true minimum distance, and data errors can be reduced.

[0149] In step 440, the training data set, the first distance, and the second distance are input into the pre-set neural network model, the pre-set neural network model is trained, and a target neural network model is obtained.

[0150] In an embodiment, during the training of the pre-set neural network model, the first loss function is constructed by the following method: according to the output distance corresponding to each high-quality data and the corresponding first distance, and the output distance corresponding to each low-quality data and the corresponding second distance, the first loss function of the pre-set neural network model is constructed. The first loss function is a standard distance loss function.

[0151] Specifically, the first loss function is obtained by the following method: the data pair (x, D) of the constructed training data set P is obtained according to the method provided in the above embodiment. The data pair is input into the pre-set neural network model, and the output f udf (x) of the neural network model can be obtained. udf (x) represents the minimum distance between the input data and the implicit manifold. The absolute difference between f udf (x) and the distance value D obtained in the corresponding data labeling stage is calculated, and this difference value is minimized. The specific formula is as follows:

[0152]

[0153] L UDF , i.e., the first loss function, L UDF , is used to constrain the output of the neural network model to a given implicit manifold distance. The given implicit manifold distance is the distance value corresponding to the data in the training data set determined in the training data construction stage.

[0154] In an embodiment, during the training of the pre-set neural network model, the second loss function is also constructed by the following method: according to the derivative of each low-quality data and the output distance corresponding to the low-quality data, the second gradient information of each low-quality data is determined; according to the second gradient information of each low-quality data, the second loss function of the pre-set neural network model is constructed. The second loss function is a regularization loss function. The second gradient information represents the direction of the closest data point on the implicit manifold to the low-quality data. For each low-quality data, the second gradient information is obtained according to the product of the derivative of the low-quality data and the output distance output by the neural network model to which the low-quality data is input.

[0155] Specifically, all low-quality data (D>0) is used to minimize a regular loss L eik to encourage the implicit manifold corresponding to the signed distance field to have a unit norm gradient. The specific formula of the second loss function is as follows:

[0156]

[0157] L eik i.e. the second loss function, (x, D) e P, D≠0 represents that the second loss function value only considers low-quality data (D≠0). L eik The role of is to constrain the gradient of the trained neural network model, so that the process of improving the data quality by using the trained target neural network model is more stable.

[0158] In some embodiments, the first loss function and the second loss function provided in the above embodiments are used simultaneously in the process of training the pre-set neural network model. Training the neural network model in combination with the first loss function and the second loss function enables the target neural network model obtained by training to be able to fit the unsigned distance field corresponding to the training data set. The distance field preserves the relative distance between data points and allows traversal along the negative gradient of the distance function, which points to the direction in which the distance from the data point to the implicit manifold decreases most quickly. Thus, the target data is obtained by updating the data to be improved based on the target neural network model using an iterative gradient descent method to approximate the implicit manifold, so that the distance relationship between the data to be improved and the implicit manifold can be accurately obtained by the target neural network model, making the target data obtained by iteration more accurate. Finally, the reliability of the data quality improvement process is improved, and the final data quality is improved.

[0159] In step 320, intermediate data is generated according to the output distance, the data to be improved, the first gradient information, and a pre-set gradient descent rate hyperparameter, wherein the first gradient information is used to represent the direction of the data to be improved to the nearest data point on the implicit manifold.

[0160] In an embodiment, with reference to Figure 11 , step 320 includes:

[0161] Step 1110, determining first gradient information according to the derivative of the data to be improved and the output distance;

[0162] Step 1120, determining a gradient descent value according to the product of the first gradient information, the gradient descent rate hyperparameter, and the output distance;

[0163] Step 1130, determining intermediate data according to the difference between the data to be improved and the gradient descent value.

[0164] Specifically, based on the trained target neural network model f udf , the target neural network model can output the minimum distance f udf (x) between any given data to be promoted x and the implicit manifold. By the product of the derivative of the data to be promoted x and the minimum distance f udf (x), the first gradient information of the data to be promoted x can be determined, which represents the direction from the nearest high-quality data x' on the implicit manifold to the data to be promoted x. Therefore, by applying the gradient descent method, the data to be promoted x can be moved (updated) towards the implicit manifold, and finally the target data with improved data quality is obtained. wherein S represents the implicit manifold, and d(x, x') represents the distance from the data point x to be promoted to the high-quality data point x' in the implicit manifold S.

[0165] The formula of the intermediate data obtained by each gradient descent is as follows:

[0166]

[0167] wherein a is a pre-set gradient descent rate hyperparameter, x i represents the intermediate data obtained by the i-th step iteration, x i-1 represents the data to be promoted obtained by the x i-1 -th step iteration.

[0168] The implementation details of the data promotion method of the embodiments of the present disclosure

[0169] The implementation details of the data promotion method of the embodiments of the present disclosure will be described in detail below with reference to Figure 12 .

[0170] In step 1210, the training data is constructed: the system obtains high-quality data, and constructs a batch of low-quality data by adding noise and the like, and these data collectively construct the training set. Specifically, an original data set is obtained, wherein the original data set is a high-quality data set corresponding to the data to be promoted; noise adding processing is performed on the high-quality data set to obtain a low-quality data set; and the high-quality data set and the low-quality data set are merged to obtain the training data set.

[0171] In step 1220, an unsigned distance field is constructed: an implicit unsigned distance field is constructed to calculate the distance of each data to the implicit manifold, and the unsigned distance field reflects the relative position relationship of the data in the feature space. Specifically, according to a plurality of high-quality data, an implicit manifold is constructed, wherein each high-quality data has a first distance to the implicit manifold of 0; a second distance of each low-quality data to the implicit manifold is calculated, wherein the second distance is greater than 0. Each high-quality data of the training data set and its respective first distance, each low-quality data of the training data set and its respective second distance collectively constitute an unsigned distance field. It should be noted that in the embodiments of the present disclosure, the construction of the unsigned distance field is embodied by labeling the corresponding distance value of each data in the training data set.

[0172] In step 1230, a neural network model is trained: using the constructed training data, a pre-set neural network model is trained to fit the unsigned distance field and learn the distance relationship between the data. Specifically, the training data set and the corresponding data values of each data are input into the pre-set neural network model for training, so that the neural network model fits the unsigned distance field and obtains a target neural network model. In the training process, the pre-set neural network model is trained in combination with a first loss function and a second loss function; the first loss function is L UDF (x,D)∈P |f udf (x)-D|, and the second loss function is

[0173]

[0174] In step 1240, the quality of the data is improved: the target neural network model trained is used to improve the quality of the input low-quality data, thereby improving the quality and reliability of the data. Specifically, the data to be improved is input into the target neural network model obtained by training to obtain an output distance, wherein the output distance represents the minimum distance between the data to be improved and the implicit manifold, and the implicit manifold is obtained based on the high-quality data set; according to the output distance, the data to be improved, the first gradient information and the pre-set gradient descent rate hyperparameter, intermediate data is generated, wherein the first gradient information is used to represent the direction of the nearest data point on the implicit manifold to the data to be improved; the intermediate data is used to update the data to be improved, and the step of inputting the data to be improved into the target neural network model to obtain the output distance is returned to iteratively update the data to be improved until a pre-set iteration stopping condition is met, and the data to be improved that meets the pre-set iteration stopping condition is taken as the target data.

[0175] The benefits of steps 1210 to 1240 include but are not limited to effectively processing the quality problems in the data, improving the quality and reliability of the data, and being suitable for various data quality processing scenarios. ​

[0176] As shown in Figure 13 , the upper diagram in FIG. 1 is the data to be improved, and the scattered points around the annulus (implicit manifold) represent noise data. Figure 13 The lower diagram in FIG. 1 is the data after the data quality is improved. As can be seen, the original noise data has obtained the corresponding target data through the data quality improvement method of the embodiment of the present disclosure, and together with the data without data quality problems, forms a complete annulus (implicit manifold). Figure 13 Figure 13 The device and equipment of the embodiment of the present disclosure are described

[0177] It can be understood that although each step in each of the above flowcharts is displayed in sequence according to the arrow representation, these steps are not necessarily executed in the order represented by the arrow. Unless otherwise specified in the embodiment, the execution of these steps has no strict order limitation, and these steps can be executed in other orders. Moreover, at least part of the steps in the above flowcharts can include multiple steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or steps or stages in other steps.

[0178] The structure diagram of the data quality improvement device provided by the embodiment of the present disclosure is shown in FIG. 14. The data quality improvement device 1400 includes:

[0179] Figure 14 The output distance calculation unit 1410 is configured to input the data to be improved into a target neural network model to obtain an output distance, wherein the output distance represents the distance between the data to be improved and the implicit manifold corresponding to the target data to be determined, and the output distance is greater than 0;

[0180] The intermediate data calculation unit 1420 is configured to generate intermediate data according to the output distance, the data to be improved, the first gradient information and the pre-set gradient descent rate hyperparameter, wherein the first gradient information is used to represent the direction of the data to be improved to the nearest data point on the implicit manifold;

[0181] The target data determination unit 1430 is configured to update the data to be improved with the intermediate data, and send the data to be improved to the output distance calculation unit 1410 until a pre-set iteration stopping condition is met, and take the data to be improved as the target data.

[0182] Optionally, the data quality improvement device 1400 further includes:

[0183] Optionally, the data quality improvement device 1400 further includes:

[0184] ​The training data construction unit is configured to obtain an original data set and construct a training data set according to the original data set, wherein the training data set includes a plurality of high-quality data and a plurality of low-quality data.

[0185] The manifold construction unit is configured to construct an implicit manifold according to the plurality of high-quality data, wherein each high-quality data has a first distance of 0 with the implicit manifold.

[0186] The first calculation unit is configured to calculate a second distance of each low-quality data with the implicit manifold, wherein the second distance is greater than 0.

[0187] The neural network training unit is configured to input the training data set, the first distance and the second distance into a pre-set neural network model, train the pre-set neural network model, and obtain a target neural network model.

[0188] Optionally, the training data construction unit is specifically configured to:

[0189] obtain an original data set, wherein the original data set is a high-quality data set corresponding to to-be-improved data;

[0190] perform noise adding processing on the high-quality data set to obtain a low-quality data set;

[0191] merge the high-quality data set and the low-quality data set to obtain the training data set.

[0192] Optionally, the first calculation unit is specifically configured to:

[0193] for each low-quality data, determine a plurality of nearest neighbor data closest to the low-quality data from the high-quality data set by using a K nearest neighbor algorithm;

[0194] determine a distance value between the low-quality data and each nearest neighbor data;

[0195] obtain the second distance of the low-quality data according to an average value of the plurality of distance values.

[0196] Optionally, the neural network training unit is specifically configured to:

[0197] construct a first loss function of the pre-set neural network model according to an output distance corresponding to each high-quality data and a first distance corresponding to the high-quality data, and an output distance corresponding to each low-quality data and a second distance corresponding to the low-quality data.

[0198] Optionally, the neural network training unit is specifically configured to:

[0199] determine second gradient information of each low-quality data according to a derivative of the low-quality data and an output distance corresponding to the low-quality data;

[0200] According to the second gradient information of each low-quality data, a second loss function of a preset neural network model is constructed.

[0201] Optionally, the intermediate data calculation unit 1420 is specifically configured to:

[0202] According to the derivative of the to-be-promoted data and the output distance, the first gradient information is determined.

[0203] According to the product of the first gradient information, the gradient descent rate hyperparameter, and the output distance, the gradient descent value is determined.

[0204] According to the difference between the to-be-promoted data and the gradient descent value, the intermediate data is determined.

[0205] Referring to Figure 15 , Figure 15 A structure block diagram of a terminal for implementing part of the data quality promotion method of the embodiments of the present disclosure is shown. The terminal includes a radio frequency (RF) circuit 1510, a memory 1515, an input unit 1530, a display unit 1540, a sensor 1550, an audio circuit 1560, a wireless fidelity (WiFi) module 1570, a processor 1580, and a power supply 1590, and the like. Those skilled in the art can understand that Figure 15 The terminal structure shown does not constitute a limitation on a mobile phone or a computer, and can include more or fewer components than shown, or combine certain components, or have a different arrangement of components.

[0206] The RF circuit 1510 can be used for receiving and sending signals in the process of receiving or calling, especially receiving the downlink information of the base station and processing it by the processor 1580, and sending the uplink data to the base station.

[0207] The memory 1515 can be used to store software programs and modules, and the processor 1580 can execute various functions and data processing of the content terminal by running the software programs and modules stored in the memory 1515.

[0208] The input unit 1530 can be used to receive input digital or character information, and generate key signal input related to the setting and function control of the content terminal. Specifically, the input unit 1530 can include a touch panel 1531 and other input devices 1532.

[0209] The display unit 1540 can be used to display input information or provided information and various menus of the content terminal. The display unit 1540 can include a display panel 1515.

[0210] The audio circuit 1560, the speaker 1561, and the microphone 1562 can provide an audio interface.

[0211] In this embodiment, the processor 1580 included in the terminal can perform the data quality improvement method of the preceding embodiments.

[0212] The terminal of the embodiments of the present disclosure includes but is not limited to a mobile phone, a computer, a smart voice interaction device, a smart home appliance, a vehicle-mounted terminal, an aircraft, and the like. The embodiments of the present disclosure can be applied to various scenarios, including but not limited to content recommendation, data screening, and the like.

[0213] Figure 16 A structural block diagram of a part of a server for implementing the data quality improvement method of the embodiments of the present disclosure. The server can have a large difference due to different configurations or performances, and can include one or more central processing units (CPUs) 1622 (for example, one or more processors) and a memory 1632, one or more storage media 1630 (for example, one or more mass storage devices) storing an application program 1616 or data 1644. Among them, the memory 1632 and the storage medium 1630 can be temporary storage or persistent storage. The program stored in the storage medium 1630 can include one or more modules (not shown in the figure), each of which can include a series of instruction operations in the server. Further, the central processing unit 1622 can be configured to communicate with the storage medium 1630 and execute a series of instruction operations in the storage medium 1630 on the server.

[0214] The server can also include one or more power supplies 1626, one or more wired or wireless network interfaces 1650, one or more input / output interfaces 1658, and / or one or more operating systems 1641, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, and the like.

[0215] The central processing unit 1622 in the server can be used to execute the data quality improvement method of the embodiments of the present disclosure.

[0216] The embodiments of the present disclosure also provide a computer-readable storage medium for storing program code, the program code being used to execute the data quality improvement method of the preceding embodiments.

[0217] The embodiments of the present disclosure also provide a computer program product including a computer program. The processor of the computer device reads the computer program and executes it, so that the computer device executes the data quality improvement method as described above.

[0218] The terms "first", "second", "third", "fourth", and the like in the description of the disclosure and the above drawings, if any, are used to distinguish similar objects, and do not have to be used to describe a particular order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the disclosure described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "contain" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device containing a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0219] It should be understood that in the present disclosure, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" is used to describe the relationship between the associated content, which means that there can be three relationships, for example, "A and / or B" can represent three cases: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated content. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0220] It should be understood that in the description of the embodiments of the present disclosure, the meaning of multiple (or multiple items) is two or more, greater than, less than, more than, etc. are not included in the number, above, below, etc. are understood to include the number.

[0221] In several embodiments provided by the present disclosure, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be omitted or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed mutual ones can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0222] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0223] In addition, each functional unit in various embodiments of the present disclosure can be integrated into one processing unit, or each unit can exist physically, or two or more units can be integrated into one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0224] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical scheme of the present disclosure essentially or the part that contributes to the prior art, or the whole or part of the technical scheme can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the various embodiments of the method of the present disclosure. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0225] It should also be understood that the various embodiments provided by the present disclosure can be combined in any way to achieve different technical effects.

[0226] The above is a specific description of the embodiments of the present disclosure, but the present disclosure is not limited to the above embodiments. Those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the present disclosure, and these equivalent modifications or replacements are all included in the scope defined by the claims of the present disclosure.

Claims

1. A data quality improvement method, characterized by, The method comprises: obtaining an original data set, and constructing a training data set according to the original data set, wherein the training data set comprises a plurality of high-quality data and a plurality of low-quality data; constructing an implicit manifold according to the plurality of high-quality data, wherein the first distance of each high-quality data to the implicit manifold is 0; calculating the second distance of each low-quality data to the implicit manifold, wherein the second distance is greater than 0; inputting the training data set, the first distance and the second distance into a pre-set neural network model, training the pre-set neural network model, and obtaining a target neural network model; inputting to-be-upgraded data into the target neural network model to obtain an output distance, wherein the output distance represents the minimum distance between the to-be-upgraded data and an implicit manifold, and the implicit manifold is obtained based on a high-quality data set; generating intermediate data according to the output distance, the to-be-upgraded data, first gradient information and a pre-set gradient descent rate hyperparameter, wherein the first gradient information is used to represent the direction of the to-be-upgraded data to the nearest data point on the implicit manifold; updating the to-be-upgraded data according to the intermediate data, returning the step of inputting the to-be-upgraded data into the target neural network model to obtain the output distance, and iteratively updating the to-be-upgraded data until a pre-set iteration stopping condition is met, and taking the to-be-upgraded data meeting the pre-set iteration stopping condition as target data.

2. The method of claim 1, wherein, The obtaining of the original data set and the construction of the training data set according to the original data set comprise: obtaining the original data set, wherein the original data set is a high-quality data set corresponding to the to-be-upgraded data; performing noise addition processing on the high-quality data set to obtain a low-quality data set; merging the high-quality data set and the low-quality data set to obtain the training data set.

3. The method of claim 1, wherein, The calculation of the second distance of each low-quality data to the implicit manifold comprises: for each low-quality data, determining a plurality of nearest neighbor data to the low-quality data from the high-quality data set by using a K-nearest neighbor algorithm; determining the distance value between the low-quality data and each nearest neighbor data; obtaining the second distance of the low-quality data according to the average value of a plurality of distance values.

4. The method according to claim 1 or 3, characterized in that, Before the training data set, the first distance and the second distance are inputted into a pre-set neural network model, the pre-set neural network model is trained, and a target neural network model is obtained, the method further comprises: constructing a first loss function of the pre-set neural network model according to the output distance corresponding to each high-quality data and the first distance corresponding to each high-quality data, and the output distance corresponding to each low-quality data and the second distance corresponding to each low-quality data.

5. The method of claim 4, wherein, Before the training data set, the first distance and the second distance are inputted into a pre-set neural network model, the pre-set neural network model is trained, and a target neural network model is obtained, the method further comprises: determine second gradient information of each of the low-quality data according to derivatives of the low-quality data and the output distance corresponding to the low-quality data; construct a second loss function of the pre-set neural network model according to the second gradient information of each of the low-quality data.

6. The method of claim 1, wherein, The generating intermediate data according to the output distance, the to-be-promoted data, first gradient information and a pre-set gradient descent rate hyperparameter comprises: determining the first gradient information according to derivatives of the to-be-promoted data and the output distance; determining a gradient descent value according to a product of the first gradient information, the gradient descent rate hyperparameter and the output distance; determining the intermediate data according to a difference between the to-be-promoted data and the gradient descent value.

7. A data quality improvement apparatus characterized by comprising: The method comprises a training data construction unit configured to obtain an original data set and construct a training data set according to the original data set, wherein the training data set comprises a plurality of high-quality data and a plurality of low-quality data; a manifold construction unit configured to construct an implicit manifold according to the plurality of high-quality data, wherein a first distance between each of the high-quality data and the implicit manifold is 0; a first calculation unit configured to calculate a second distance between each of the low-quality data and the implicit manifold, wherein the second distance is greater than 0; a neural network training unit configured to input the training data set, the first distance and the second distance into a pre-set neural network model, train the pre-set neural network model and obtain a target neural network model; an output distance calculation unit configured to input to-be-promoted data into the target neural network model and obtain an output distance, wherein the output distance represents a minimum distance between the to-be-promoted data and an implicit manifold, and the implicit manifold is obtained based on a high-quality data set; an intermediate data calculation unit configured to generate intermediate data according to the output distance, the to-be-promoted data, first gradient information and a pre-set gradient descent rate hyperparameter, wherein the first gradient information is used to represent a direction of the to-be-promoted data to a nearest data point on the implicit manifold; a target data determination unit configured to update the to-be-promoted data according to the intermediate data, send the to-be-promoted data to the output distance calculation unit until a preset iteration stopping condition is met, and take the to-be-promoted data satisfying the preset iteration stopping condition as target data.

8. An electronic device comprising: A memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the data quality promotion method in any one of claims 1 to 6 when executing the computer program.

9. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer executable instructions, and the computer executable instructions implement the data quality promotion method in any one of claims 1 to 6.

10. A computer program product comprising computer programs or computer instructions, characterized in that, The computer program or the computer instruction is stored in a computer readable storage medium, and a processor of a computer device reads the computer program or the computer instruction from the computer readable storage medium. The processor executes the computer program or the computer instruction, so that the computer device executes the data quality improvement method as claimed in any one of claims 1 to 6.

Images