A method and apparatus for filling missing data

Abstract

The application discloses a kind of filling method and device of missing data, the method includes: obtaining target data;The target data is preprocessed, and preprocessed target data is obtained;The preprocessed target data is filled, and filling result data is obtained.The present application uses singular value threshold to analyze and process the characteristics of data matrix, effectively solves the problems of large error, complex calculation, poor adaptability and other problems in the existing missing data filling technology, improves the accuracy and efficiency of missing data filling.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and apparatus for filling missing data. The method utilizes a singular value threshold iteration algorithm to effectively fill missing data. When applied to the processing of large-scale datasets, filling in the missing parts of the data can improve data analysis and prediction, thereby enabling more accurate analysis. It can be widely used in various scenarios that require data integrity processing, such as data analysis, data mining, machine learning, and artificial intelligence. Background Technology

[0002] In actual data acquisition and processing, data loss is very common due to various reasons, such as sensor malfunction, data transmission errors, and human error. Missing data can seriously affect subsequent data analysis, model training, and decision-making, potentially leading to biased analysis results, degraded model performance, or even failure to function properly.

[0003] Currently, common methods for imputing missing data based on singular value thresholds include mean / median imputation, multiple imputation, and K-nearest neighbor imputation. Mean / median imputation is simple and direct, filling missing values ​​with the mean or median of the variable. However, this method ignores the correlation between data points, potentially introducing significant errors and reducing the accuracy and reliability of the data. Multiple imputation, through multiple simulations, can account for data uncertainty to some extent, but it has high computational complexity, requiring substantial computational resources and time. K-nearest neighbor imputation finds the K most similar neighbors to the missing data sample based on similarity metrics and uses their observations to fill the missing value. However, this method is sensitive to data distribution and characteristics, and its imputation effect is poor in high-dimensional data or cases with uneven data distribution. Therefore, a more efficient and accurate method for imputing missing data is urgently needed. Summary of the Invention

[0004] The technical problem to be solved by the present invention is that it provides a method and apparatus for filling missing data. By using singular value thresholding to perform characteristic analysis and processing on the data matrix, it effectively solves the problems of large error, complex calculation and poor adaptability in existing missing data filling techniques, improves the accuracy and efficiency of missing data filling, and provides a reliable data foundation for subsequent data analysis and processing.

[0005] To address the aforementioned technical problems, a first aspect of the present invention discloses a method for filling in missing data, the method comprising: S1, Obtain target data; S2, preprocess the target data to obtain preprocessed target data; The preprocessed target data includes a target data matrix and a target data indicator matrix; S3, perform filling processing on the preprocessed target data to obtain filled result data.

[0006] As an optional implementation, in a first aspect of the present invention, the preprocessing of the target data to obtain preprocessed target data includes: S21, construct a target data matrix using the target data; S22, set the known data element values ​​in the target data matrix to 1 and the missing data element values ​​to 0 to construct the target data indicator matrix.

[0007] As an optional implementation, in the first aspect of the present invention, the step of filling the preprocessed target data to obtain filled result data includes: S31, process the preprocessed target data to obtain a low-rank approximate data matrix; S32, perform data filling processing on the low-rank approximate data matrix to obtain the filled result data.

[0008] As an optional implementation, in a first aspect of the present invention, processing the preprocessed target data to obtain a low-rank approximate data matrix includes: S311, decompose the preprocessed target data to obtain the target data matrix and the target data indicator matrix; S312, Multiply the target data matrix and the target data indicator matrix element by element to obtain known data elements; S313, Perform singular value decomposition on the known data elements to obtain singular value decomposition result data; The singular value decomposition processing expression is: ； ； in, Represents the weight matrix; Represents the first element in the target data matrix. Line 1 List the importance weight values ​​of each element; This represents the row index of the target data matrix; Represents the column index of the target data matrix; Represents the target data matrix; Represents the regularization parameter; This indicates element-wise multiplication; Represents the left singular value vector matrix; Represents a right singular value vector matrix; Represents a singular value diagonal matrix; S314, The singular value decomposition result data is processed into a low-rank approximate data matrix.

[0009] As an optional implementation, in the first aspect of the present invention, the data filling process performed on the low-rank approximate data matrix to obtain filled result data includes: S321, Initialize the target data matrix to obtain an initial matrix; Initialize the Lagrange multipliers, penalty parameters, and initial threshold; Preset maximum number of iterations; S322, for the target data matrix, the first... The data is updated to obtain the updated target matrix data and target element data; S323, using the Lagrange multipliers and the penalty parameter, the target element data is decomposed to obtain the first singular value data; S324, using the singular value threshold adjustment model, the first singular value data is processed to obtain the singular value update threshold; Based on the singular value update threshold, the first singular value data is truncated to obtain the second singular value data. S235, Update the model using Lagrange multipliers, and update the Lagrange multipliers; Update the model using the penalty parameters, and update the penalty parameters accordingly; Increase the number of iterations by 1; S326, determine whether the number of iterations is less than the maximum number of iterations, and obtain the iteration number determination result; S327, When the result of the iteration count determination is yes, execute S328; When the result of the iteration count is negative, the target matrix data is updated to the filled result data; S328, The second singular value data is judged and processed to obtain the singular value relative error; When the singular value relative error is less than the singular value update threshold, the updated target matrix data is updated to the filled result data; When the singular value relative error is greater than or equal to the singular value update threshold, execute S322.

[0010] As an optional implementation, in the first aspect of the present invention, the expression for the singular value threshold adjustment model is: ； in, Indicates the first The singular value threshold for the next iteration; Indicates the first The singular value threshold for the next iteration; Indicates the first The median of the first singular value data in the next iteration; Indicates the smoothing factor; The expression for the Lagrange multiplier update model is: ; ; ; in, Indicates the first Lagrange multipliers in the next iteration; Indicates the first The singular value threshold for the next iteration; This represents the sum of the first singular value data; This represents a known data element in the target data matrix; This indicates a known data element in the target data indicator matrix; Indicates the first projection operator; This represents the singular value threshold operator.

[0011] As an optional implementation, in a first aspect of the present invention, the step of truncating the first singular value data based on the singular value update threshold to obtain the second singular value data includes: S3241, determine whether the first singular value data is greater than the singular value update threshold, and obtain the singular value judgment result; S3242, when the singular value determination result is yes, execute S3243; When the singular value determination result is negative, the second singular value data is updated to 0; S3243, The first singular value data is processed using the singular value shrinkage model to obtain the second singular value data; The singular value contraction model is expressed as follows: ; ; in, Represents the singular value diagonal matrix of the first... Each data element.

[0012] A second aspect of the present invention discloses a data filling device, the device comprising: Data acquisition module, data preprocessing module, and data filling module; The data acquisition module is used to acquire target data; The data preprocessing module is used to preprocess the target data to obtain preprocessed target data; The preprocessed target data includes a target data matrix and a target data indicator matrix; The data filling processing module is used to fill the preprocessed target data to obtain the filled result data.

[0013] A third aspect of the present invention discloses a data filling device, the device comprising: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute some or all of the steps in the missing data filling method disclosed in the first aspect of the present invention.

[0014] The fourth aspect of the present invention discloses a computer-readable storage medium, characterized in that the computer-readable storage medium stores computer instructions, which, when invoked, execute some or all of the steps in the missing data filling method disclosed in the first aspect of the present invention.

[0015] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: This invention utilizes a singular value thresholding algorithm to process data matrices, fully uncovering the potential relationships and structural information between data points. Compared to traditional imputation methods, it can more accurately estimate the values ​​of missing data, effectively improving the accuracy of data imputation. This method employs singular value decomposition and other computational methods, and through reasonable iterative strategies and parameter settings, it controls computational complexity while ensuring imputation effectiveness. Compared to some complex multiple imputation methods, computational efficiency is significantly improved, enabling the completion of large-scale missing data imputation tasks in a shorter time, making it suitable for large-scale data processing scenarios. This invention is insensitive to data distribution and characteristics, exhibiting good imputation results regardless of whether the data is low-dimensional or high-dimensional, or whether the data distribution is uniform or uneven. It has broad applicability and can be applied to various types of data processing scenarios. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram illustrating an application scenario of the missing data filling device provided in this embodiment of the invention; Figure 2 This is a flowchart illustrating a method for filling in missing data disclosed in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a missing data filling device disclosed in an embodiment of the present invention; Figure 4 This is a schematic diagram of another missing data filling device disclosed in an embodiment of the present invention. Detailed Implementation

[0018] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this invention are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, apparatus, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0020] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0021] In this application, the term "exemplary" is used to mean "used as an example, illustration, or description." Any embodiment described as "exemplary" in this application is not necessarily to be construed as being more preferred or advantageous than other embodiments. The following description is provided to enable any person skilled in the art to make and use this application. Details are set forth in the following description for purposes of explanation. It should be understood that those skilled in the art will recognize that this application can be made without using these specific details. In other instances, well-known structures and processes are not described in detail to avoid obscuring the description of this application with unnecessary detail. Therefore, this application is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in this application.

[0022] It should be noted that since the method in this application embodiment is executed in a computer device, the processing objects of each computer device exist in the form of data or information, such as time, which is essentially time information. It is understood that if size, quantity, position, etc. are mentioned in subsequent embodiments, they are all corresponding data that exist so that the computer device can process them. Specific details will not be elaborated here.

[0023] This application provides a method, system, apparatus, computer device, and computer-readable storage medium for filling in missing data, which will be described in detail below.

[0024] Please see Figure 1 , Figure 1 This is a schematic diagram illustrating a scenario in which the missing data filling device provided in this application is applied in a data processing system. The data processing system may include a computer device 100, which integrates a missing data filling device, such as... Figure 1 Computer equipment in the country.

[0025] In this embodiment, the computer device 100 can be a standalone server, a server network, or a server cluster. For example, the computer device 100 described in this embodiment includes, but is not limited to, a computer, a network host, a single network server, a set of multiple network servers, or a cloud server composed of multiple servers. The cloud server is composed of a large number of computers or network servers based on cloud computing.

[0026] It is understood that the computer device 100 used in the embodiments of this application can be a device that includes both receiving and transmitting hardware, that is, a device having receiving and transmitting hardware capable of performing bidirectional communication on a bidirectional communication link. Such a device may include: cellular or other communication devices having a single-line display, a multi-line display, or a cellular or other communication device without a multi-line display. Specifically, the computer device 100 may be a desktop terminal or a mobile terminal, and may also be one of a mobile phone, tablet computer, laptop computer, etc.

[0027] Those skilled in the art will understand that Figure 1 The application environment shown is merely one application scenario of the solution in this application and does not constitute a limitation on the application scenario of the solution in this application. Other application environments may include those that are more specific to this application. Figure 1 The number of computer devices shown is more or less, for example Figure 1 Only one computer device is shown in the diagram. It is understood that the system may also include one or more other services, which are not limited here.

[0028] In addition, such as Figure 1 As shown, the data processing system may also include a memory 200 for storing sensor-acquired data, processing result data, etc.

[0029] It should be noted that, Figure 1 The schematic diagram illustrating the application scenario of the missing data filling device is merely an example. The data processing system and scenario described in the embodiments of this application are intended to more clearly illustrate the technical solutions of the embodiments of this application and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of data processing systems and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0030] This invention discloses a method and apparatus for filling missing data. It utilizes a singular value thresholding algorithm to analyze and process the characteristics of a data matrix, effectively solving the problems of large errors, computational complexity, and poor adaptability in existing missing data filling techniques. This improves the accuracy and efficiency of missing data filling, providing a reliable data foundation for subsequent data analysis and processing. Detailed descriptions follow.

[0031] Example 1 Please see Figure 2 , Figure 2 This is a flowchart illustrating a method for filling in missing data disclosed in an embodiment of the present invention. Figure 2 The described method for filling in missing data is applied in data processing systems, such as local servers or cloud servers used in data processing systems, and is not limited to these applications in this embodiment of the invention. Figure 2 As shown, the method for filling in the missing data can include the following operations: S1, Obtain target data; It should be noted that the target data refers to sensor data that is missing due to network communication interruption; S2, preprocess the target data to obtain preprocessed target data; The preprocessed target data includes a target data matrix and a target data indicator matrix; S3, perform filling processing on the preprocessed target data to obtain filled result data.

[0032] As can be seen, the missing data filling method described in the embodiments of the present invention utilizes the singular value thresholding algorithm to analyze, process, and fill the characteristics of the data matrix, effectively solving the problems of large errors, computational complexity, and poor adaptability of existing missing data filling methods, and improving the accuracy and efficiency of missing data filling.

[0033] Optionally, the preprocessing of the target data to obtain preprocessed target data includes: S21, construct a target data matrix using the target data; S22, set the known data element values ​​in the target data matrix to 1 and the missing data element values ​​to 0 to construct the target data indicator matrix.

[0034] As can be seen, by using the missing data filling method described in the embodiments of the present invention, the target data is preprocessed to obtain preprocessed target data, which provides data support for subsequent characteristic analysis, processing and filling of the data matrix using the singular value thresholding algorithm. This effectively solves the problems of large error, computational complexity and poor adaptability of existing missing data filling methods, and improves the accuracy and efficiency of missing data filling.

[0035] Optionally, the step of filling the preprocessed target data to obtain filled result data includes: S31, process the preprocessed target data to obtain a low-rank approximate data matrix; S32, perform data filling processing on the low-rank approximate data matrix to obtain the filled result data.

[0036] As can be seen, the missing data filling method described in the embodiments of the present invention utilizes the singular value thresholding algorithm to analyze and process the characteristics of the data matrix, thereby achieving the filling process of the preprocessed target data and obtaining the filled result data. This effectively solves the problems of large error, complex calculation, and poor adaptability of existing missing data filling methods, and improves the accuracy and efficiency of missing data filling.

[0037] Optionally, the step of processing the preprocessed target data to obtain a low-rank approximate data matrix includes: S311, decompose the preprocessed target data to obtain the target data matrix and the target data indicator matrix; S312, Multiply the target data matrix and the target data indicator matrix element by element to obtain known data elements; S313, Perform singular value decomposition on the known data elements to obtain singular value decomposition result data; Optionally, the singular value decomposition processing expression is: ； ； in, Represents the weight matrix; Represents the first element in the target data matrix. Line 1 List the importance weight values ​​of each element; This represents the row index of the target data matrix; Represents the column index of the target data matrix; Represents the target data matrix; Represents the regularization parameter; This indicates element-wise multiplication; Represents the left singular value vector matrix; Represents a right singular value vector matrix; Represents a singular value diagonal matrix; It should be noted that the target data matrix , representing the coordinates of the sampled element A set; Optionally, the singular value decomposition process is a weighted singular value decomposition model, and the expression of the weighted singular value decomposition model is: , , in, Represents the weight matrix; Represents the first element in the target data matrix. Line 1 List the importance weight values ​​of each element; This represents the row index of the target data matrix; Represents the column index of the target data matrix; Represents the target data matrix; This indicates element-wise multiplication; Represents the left singular value vector matrix; Represents a right singular value vector matrix; Represents a singular value diagonal matrix; For the first OK Distance coefficient between column sample coordinates and missing sample coordinates; It should be noted that the weighted singular value decomposition model dynamically generates element-differentiated weights through distance coefficients. It can distinguish the importance level of data based on the spatial correlation between samples and missing samples. The weight matrix can automatically amplify the contribution weight of highly reliable and effective observation data in singular value decomposition, while simultaneously reducing the interference of noise and outlier sampling points on the matrix decomposition results. Unlike the traditional equal-weighted singular value decomposition, which can only treat all matrix elements uniformly and cannot distinguish between effective information and invalid noise, the weighted singular value decomposition model can complete noise suppression and effective feature enhancement during the decomposition stage. This makes the output singular value decomposition results more consistent with the inherent low-rank structure of the original data, reducing the low-rank approximate matrix reconstruction bias from the source. It significantly improves the missing value filling distortion problem in high-dimensional and unevenly distributed datasets, providing high-precision basic decomposition data for subsequent singular value thresholding iteration filling, and ultimately improving filling accuracy and iteration convergence speed simultaneously. S314, Perform low-rank processing on the singular value decomposition result data to obtain a low-rank approximate data matrix; It should be noted that in this embodiment, the preprocessed target data is processed by weighted singular value decomposition, and a weight matrix is ​​introduced. Different weights are used for different data to improve the reconstruction accuracy of key elements and improve the accuracy of data filling. As can be seen, by using the missing data filling method described in the embodiments of the present invention to process the preprocessed target data, a low-rank approximate data matrix is ​​obtained, which provides data support for subsequent characteristic analysis, processing and filling of the data matrix using the singular value thresholding algorithm. This effectively solves the problems of large error, computational complexity and poor adaptability of existing missing data filling methods, and improves the accuracy and efficiency of missing data filling.

[0038] Optionally, the step of performing data filling processing on the low-rank approximate data matrix to obtain filled result data includes: S321, Initialize the target data matrix to obtain an initial matrix; Initialize the Lagrange multipliers, penalty parameters, and initial threshold; Preset maximum number of iterations; S322, for the target data matrix, the first... The data is updated to obtain the updated target matrix data and target element data; Optionally, the update processing expression is: ; ; in, Indicates the first The target element data; express; This represents the singular value threshold operator; Indicates the first element in the target data indication matrix One element; Indicates the first element in the target data indication matrix One element; Indicates the first penalty parameter; Indicates the first projection operator; Indicates the second projection operator; Represents the target data matrix; Indicates the first The singular value threshold for the next iteration; It should be noted that the first penalty parameter As a step size factor in iterative algorithms, it is related to the matrix size: It should be noted that, in this embodiment, the value of the first penalty parameter is: ; It should be noted that the first projection operator Used to store the data matrix to be processed in the target data matrix All elements except those in the set element are set to 0, while the remaining elements remain unchanged. It should be noted that the second projection operator , representing the target data matrix obtained after calculation by the first projection operator; It should be noted that the first... Singularity threshold for the next iteration , obtained from the singular value threshold adjustment model; S323, using the Lagrange multipliers and the penalty parameter, the target element data is decomposed to obtain the first singular value data; S324, using the singular value threshold adjustment model, the first singular value data is processed to obtain the singular value update threshold; Based on the singular value update threshold, the first singular value data is truncated to obtain the second singular value data. S235, Update the model using Lagrange multipliers, and update the Lagrange multipliers; Update the model using the penalty parameters, and update the penalty parameters accordingly; Increase the number of iterations by 1; S326, determine whether the number of iterations is less than the maximum number of iterations, and obtain the iteration number determination result; S327, When the result of the iteration count determination is yes, execute S328; When the result of the iteration count is negative, the target matrix data is updated to the filled result data; S328, The second singular value data is judged and processed to obtain the singular value relative error; When the singular value relative error is less than the singular value update threshold, the updated target matrix data is updated to the filled result data; when the singular value relative error is greater than or equal to the singular value update threshold, S322 is executed.

[0039] It should be noted that during the iteration process Always maintain sparsity, while a larger threshold makes It maintains low rank and can be represented as the product of two singular vector matrices of significantly reduced size using the singular value shrinkage operator. These properties greatly reduce memory usage during computation, thus making it suitable for large-scale matrix filling calculations.

[0040] As can be seen, by using the missing data filling method described in the embodiments of the present invention to fill the low-rank approximate data matrix, the filled result data is obtained. The missing data filling is achieved by using the singular value thresholding algorithm, which effectively solves the problems of large error, computational complexity and poor adaptability of existing missing data filling methods, and improves the accuracy and efficiency of missing data filling.

[0041] Optionally, the initialization process of the target data matrix to obtain an initial matrix includes: S3211, Set initial values ​​for iteration ; S3212, Set the initial value of the iteration count. To satisfy ; S3213, Process the target data matrix to obtain an initial matrix. ; The processing expression is: ； in, This represents the initial matrix for the iterative process; This represents the initial value of the number of iterations; This represents the initial penalty parameter; This represents the second projection operator; Represents the target data matrix; It should be noted that, in this embodiment, the initial penalty parameter The value can be: ; It should be noted that the missing data filling method described in the embodiments of the present invention can skip [the steps described]. Step directly from Iterative calculations were initiated, which effectively improved the efficiency of data filling. As can be seen, by using the missing data filling method described in the embodiments of the present invention, the target data matrix is ​​initialized to obtain an initial matrix, which provides a data foundation for subsequent missing data filling processing using the singular value thresholding algorithm. This effectively solves the problems of large error, computational complexity, and poor adaptability of existing missing data filling methods, and improves the accuracy and efficiency of missing data filling.

[0042] Optionally, the singular value threshold adjustment model expression is: ； in, Indicates the first The singular value threshold for the next iteration; Indicates the first The singular value threshold for the next iteration; Indicates the first The median of the first singular value data in the next iteration; Represents the smoothing factor; It should be noted that the range of values ​​for the smoothing factor is [insert range here]. ; Optionally, the singular value threshold adjustment model expression is: , in, Indicates the first The singular value threshold for the next iteration; Indicates the first The singular value threshold for the next iteration; Indicates the first The median of the first singular value data in the next iteration; This represents the smoothing factor, with a value range of [value range missing]. This is used to control the smoothness of threshold updates; For the first The first singular value is the diagonal matrix of singular values.

[0043] It should be noted that existing singular value thresholding algorithms all use manually set fixed thresholds, which cannot adapt to datasets with different ranks and distribution characteristics, and are prone to underfilling and overfitting. The adaptive dynamic threshold update model, in each iteration, smoothly updates the threshold based on the median of the current singular value distribution. In the early stage of the iteration, the singular value dispersion is high, and the threshold is adaptively amplified to filter meaningless noise singular values. In the later stage of the iteration, the singular values ​​gradually converge, and the threshold is automatically reduced to retain subtle data correlation features, balancing filling accuracy and iteration stability, and solving the technical pain point of poor adaptability of fixed thresholds. Optionally, the expression for the Lagrange multiplier update model is: ; in, Indicates the first Lagrange multipliers in the next iteration; Indicates the first Lagrange multipliers in the next iteration; Indicates the first The target data matrix for the next iteration; Indicates the first Singular value data for the next iteration; Optionally, the expression for the Lagrange multiplier update model is: ; ; ; in, Indicates the first Lagrange multipliers in the next iteration; Indicates the first The singular value threshold for the next iteration; This represents the sum of the first singular value data; This represents a known data element in the target data matrix; This indicates a known data element in the target data indicator matrix; Indicates the third projection operator; This indicates a specific element in the target data indicator matrix; This represents the singular value threshold operator; It should be noted that the third projection operator refers to projecting the target data indicator matrix onto... All other elements are set to 0; It should be noted that, ,in This represents the computation of the trace of a matrix; Optionally, the expression for the Lagrange multiplier update model is: , , , in, , They represent the first , Round-rotation Lagrange multiplier matrix; , They represent the first , Round penalty parameters; This is the upper limit of the penalty parameter; This is the amplification factor for the penalty parameter; Indicates the relative error of singular values; Represents the Frobenius norm of a matrix; It should be noted that the traditional augmented Lagrange matrix filling algorithm is not adapted to the pre-process of weighted singular value decomposition, and the residual calculation does not introduce a weight matrix, resulting in poor fitting effect of iterative residuals. The proposed Lagrange multiplier update model introduces a weight matrix to correct the multiplier update residuals, which is fully matched with the pre-processed weighted singular value decomposition logic. The penalty parameter is dynamically amplified to accelerate the iterative convergence speed. The relative error of singular values ​​is used instead of the traditional global matrix residual as the convergence judgment criterion, which is more sensitive to the convergence identification of low-rank sparse data, can terminate invalid iterations in advance, and effectively reduce the overall computing power consumption. As can be seen, the missing data filling method described in the embodiments of the present invention utilizes the singular value threshold adjustment model and the Lagrange multiplier update model to perform data filling processing on the low-rank approximate data matrix, obtain the filled result data, and uses the singular value threshold algorithm to perform missing data filling processing, effectively solving the problems of large error, computational complexity, and poor adaptability of existing missing data filling methods, and improving the accuracy and efficiency of missing data filling.

[0044] Optionally, the step of truncating the first singular value data based on the singular value update threshold to obtain the second singular value data includes: S3241, determine whether the first singular value data is greater than the singular value update threshold, and obtain the singular value judgment result; S3242, when the singular value determination result is yes, execute S3243; When the singular value determination result is negative, the second singular value data is updated to 0; S3243, The first singular value data is processed using the singular value shrinkage model to obtain the second singular value data; The singular value contraction model is expressed as follows: ; ; in, Represents the singular value diagonal matrix of the first... One data element; It should be noted that the missing data imputation method described in the embodiments of the present invention only involves the singular value decomposition of the matrix and skips the... Step, directly from Iterative calculations were initiated, which effectively improved the efficiency of data filling. It should be noted that, in this invention, for complex noisy data imputation problems, the following parameters are set before step S2: ， ， Noise = ×White Gaussian noise; ； ； Where tol represents the iteration termination threshold; The initial singular value threshold is represented by OS; OS represents the oversampling rate. The iteration step size is represented by std; the standard deviation of the white Gaussian noise is represented by std. It should be noted that the oversampling rate OS is the ratio of the actual number of observed samples to the theoretical minimum number of observed samples, which is used to improve the stability of the filling. It should be noted that the iteration step size This is used to control the convergence speed of the algorithm.

[0045] It should be noted that the standard deviation (std) is used to measure the noise intensity or the degree of dispersion in the distribution. As can be seen, the missing data filling method described in the embodiments of the present invention truncates the first singular value data to obtain the second singular value data, providing data support for filling the low-rank approximate data matrix and obtaining the filling result data. The missing data filling process is realized by using the singular value threshold algorithm, which effectively solves the problems of large error, computational complexity and poor adaptability of existing missing data filling methods, and improves the accuracy and efficiency of missing data filling.

[0046] Example 2 Please see Figure 3 , Figure 3 This is a schematic diagram of a missing data filling device disclosed in an embodiment of the present invention. Figure 3The described missing data filling device is applied in data processing systems, such as local servers or cloud servers used in data systems, and the embodiments of the present invention are not limited thereto. Figure 3 As shown, the device may include: Data acquisition module 101, data preprocessing module 102, and data filling processing module 103; The data acquisition module 101 is used to acquire target data; The data preprocessing module 102 is used to preprocess the target data to obtain preprocessed target data; The preprocessed target data includes a target data matrix and a target data indicator matrix; The data filling processing module 103 is used to fill the preprocessed target data to obtain the filled result data; The device provided in this embodiment has the same implementation principle and technical effect as the aforementioned method embodiment. For the sake of brevity, any parts not mentioned in the device embodiment can be referred to the corresponding content in the aforementioned method embodiment.

[0047] Example 3 Please see Figure 4 , Figure 4 This is a schematic diagram of a missing data filling device disclosed in an embodiment of the present invention. Figure 4 The described apparatus can be applied to data processing systems that fill in missing data, such as local servers or cloud servers used in data processing systems, and the embodiments of the present invention are not limited thereto. Figure 4 As shown, the device may include: Memory 201 storing executable program code; Processor 202 coupled to memory 201; The processor 202 calls the executable program code stored in the memory 201 to execute the steps in the missing data filling method described in Embodiment 1.

[0048] Example 4 This invention discloses a computer-readable storage medium that stores a computer program for electronic data interchange, wherein the computer program causes a computer to perform the steps in the missing data filling method described in Embodiment 1.

[0049] Example 5 This invention discloses a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and the computer program is operable to cause a computer to perform the steps in the missing data filling method described in Embodiment 1.

[0050] It should be noted that all calculation expressions or mathematical functions in the embodiments of the present invention have undergone dimensionless processing of the variables involved before calculation.

[0051] It should be noted that in all the calculation expressions or mathematical functions in the embodiments of the present invention, the values ​​of the input independent variables all meet the reasonable requirements of the input value range of the calculation expression or mathematical function, and can ensure that the calculation expression or mathematical function can be calculated smoothly without violating physical laws or mathematical rules.

[0052] The device embodiments described above are merely illustrative. The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0053] Through the detailed description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, including read-only memory (ROM), random access memory (RAM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), once programmable read-only memory (OTPROM), electronically erasable rewritable read-only memory (EEPROM), read-only optical disc (CD-ROM) or other optical disc storage, disk storage, magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0054] Finally, it should be noted that the method, system, and apparatus for filling missing data disclosed in the embodiments of the present invention are merely preferred embodiments of the present invention and are only used to illustrate the technical solutions of the present invention, not to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for filling missing data, characterized in that, The method includes: S1, Obtain target data; S2, preprocess the target data to obtain preprocessed target data; The preprocessed target data includes a target data matrix and a target data indicator matrix; S3, perform filling processing on the preprocessed target data to obtain filled result data.

2. The method for filling missing data according to claim 1, characterized in that, The preprocessing of the target data to obtain preprocessed target data includes: S21, construct a target data matrix using the target data; S22, set the known data element values ​​in the target data matrix to 1 and the missing data element values ​​to 0 to construct the target data indicator matrix.

3. The method for filling missing data according to claim 1, characterized in that, The step of filling the preprocessed target data to obtain filled result data includes: S31, process the preprocessed target data to obtain a low-rank approximate data matrix; S32, perform data filling processing on the low-rank approximate data matrix to obtain the filled result data.

4. The method for filling missing data according to claim 3, characterized in that, The process of processing the preprocessed target data to obtain a low-rank approximate data matrix includes: S311, decompose the preprocessed target data to obtain the target data matrix and the target data indicator matrix; S312, Multiply the target data matrix and the target data indicator matrix element by element to obtain known data elements; S313, Perform singular value decomposition on the known data elements to obtain singular value decomposition result data; The singular value decomposition processing expression is: ； ； in, Represents the weight matrix; Represents the first element in the target data matrix. Line 1 List the importance weight values ​​of each element in the book; This represents the row index of the target data matrix; Represents the column index of the target data matrix; Represents the target data matrix; Represents the regularization parameter; This indicates element-wise multiplication; Represents the left singular value vector matrix; Represents a right singular value vector matrix; Represents a singular value diagonal matrix; S314, The singular value decomposition result data is processed into a low-rank approximate data matrix.

5. The method for filling missing data according to claim 3, characterized in that, The data filling process performed on the low-rank approximate data matrix to obtain the filled result data includes: S321, Initialize the target data matrix to obtain an initial matrix; Initialize the Lagrange multipliers, penalty parameters, and initial threshold; Preset maximum number of iterations; S322, for the target data matrix, the first... The data is updated to obtain the updated target matrix data and target element data; S323, using the Lagrange multipliers and the penalty parameter, the target element data is decomposed to obtain the first singular value data; S324, using the singular value threshold adjustment model, the first singular value data is processed to obtain the singular value update threshold; Based on the singular value update threshold, the first singular value data is truncated to obtain the second singular value data. S235, Update the model using Lagrange multipliers, and update the Lagrange multipliers; Update the model using the penalty parameters, and update the penalty parameters accordingly; Increase the number of iterations by 1; S326, determine whether the number of iterations is less than the maximum number of iterations, and obtain the iteration number determination result; S327, When the result of the iteration count determination is yes, execute S328; When the result of the iteration count is negative, the target matrix data is updated to the filled result data; S328, The second singular value data is judged and processed to obtain the singular value relative error; When the singular value relative error is less than the singular value update threshold, the updated target matrix data is updated to the filled result data; When the singular value relative error is greater than or equal to the singular value update threshold, execute S322.

6. The method for filling missing data according to claim 5, characterized in that, The expression for the singular value threshold adjustment model is: ； in, This represents the singular value threshold for the (k+1)th iteration; The singular value threshold represents the threshold value for the k-th iteration. This represents the median of the first singular value data in the k-th iteration; Indicates the smoothing factor; The expression for the Lagrange multiplier update model is: ; ; ; in, Denotes the Lagrange multiplier in the k-th iteration; Let represent the singular value threshold for the k-th iteration; This represents the sum of the first singular value data; X This represents a known data element in the target data matrix; Y This indicates a known data element in the target data indicator matrix; Indicates the first projection operator; This represents the singular value threshold operator.

7. The method for filling missing data according to claim 5, characterized in that, The step of truncating the first singular value data based on the singular value update threshold to obtain the second singular value data includes: S3241, determine whether the first singular value data is greater than the singular value update threshold, and obtain the singular value judgment result; S3242, when the singular value determination result is yes, execute S3243; When the singular value determination result is negative, the second singular value data is updated to 0; S3243, The first singular value data is processed using the singular value shrinkage model to obtain the second singular value data; The singular value contraction model is expressed as follows: ; ; in, Represents the singular value diagonal matrix of the first... i Each data element.

8. A device for filling missing data, characterized in that, The device includes: Data acquisition module, data preprocessing module, and data filling module; The data acquisition module is used to acquire target data; The data preprocessing module is used to preprocess the target data to obtain preprocessed target data; The preprocessed target data includes a target data matrix and a target data indicator matrix; The data filling processing module is used to fill the preprocessed target data to obtain the filled result data.

9. A device for filling missing data, characterized in that, The device includes: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the missing data filling method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions, which, when invoked, are used to perform the missing data filling method as described in any one of claims 1-7.

Images