A data processing method, apparatus, device, and medium

Abstract

Embodiments of the present application provide a data processing method, device and equipment and medium for determining missing data. In the embodiments of the present application, the electronic device supplements the data of the missing position in the first data matrix, processes the first data matrix, determines the first norm module of the difference matrix of the first data matrix related to the initial second data matrix and the second data matrix and the second norm module of the second data matrix, and until the value determined according to the first norm module and the second norm module is less than a preset threshold, the data corresponding to the missing position in the second data matrix is determined as the standard supplementary data, so that the target matrix containing the missing data as much as possible can be restored, and then the missing data can be accurately determined.

Description

Technical Field

[0001] This application relates to the fields of anomaly detection, network inference, and regression analysis, and in particular to a data processing method, apparatus, device, and medium. Background Technology

[0002] In today's era of rapid information technology development, massive amounts of network data are widely used across various fields to drive industry growth and technological advancement. However, during data acquisition and transmission, unavoidable objective or human factors can lead to data loss, potentially affecting the secure use of this data. Identifying lost data as pre-set data or determining its existence based on other data may result in data anomalies. Summary of the Invention

[0003] This application provides a data processing method, apparatus, device, and medium for determining missing data.

[0004] In a first aspect, embodiments of this application also provide a data processing method, the method comprising:

[0005] Acquire each data received at each time point within a preset time length, and determine each data corresponding to a first preset feature, wherein the first preset feature is the transaction amount or the transaction time interval;

[0006] A first data matrix is ​​constructed based on the determined data, and data at missing positions in the first data matrix is ​​supplemented. Data received at the same time point are located in the same row of the first data matrix, and data with the same feature are located in the same column.

[0007] Repeat the following steps: Perform Singular Value Decomposition (SVD) on the first data matrix to obtain a first matrix, a first diagonal matrix, and a second matrix; adjust the elements at preset positions in the first diagonal matrix to preset values; determine the second data matrix based on the first matrix, the adjusted first diagonal matrix, and the second matrix; supplement the missing data in the second data matrix to the corresponding missing positions in the first data matrix to generate a target matrix; determine the first normal form modulus of the difference matrix between the second data matrix and the initial second data matrix, and the second normal form modulus of the second data matrix; if the value determined by the first normal form modulus and the second normal form modulus is less than a preset threshold, then the data at the missing positions in the target matrix is ​​determined as standard supplementary data; otherwise, update the target matrix to the first data matrix.

[0008] Secondly, embodiments of this application also provide a data processing apparatus, the apparatus comprising:

[0009] The acquisition and determination module is used to acquire each data received at each time point within a preset time length, and determine each data corresponding to a first preset feature, wherein the first preset feature is the transaction amount or the transaction time interval;

[0010] The supplementation module is used to construct a first data matrix based on the determined data and supplement the missing data in the first data matrix, wherein data received at the same time point are located in the same row of the first data matrix and data with the same feature are located in the same column;

[0011] The processing module is used to repeatedly execute the following steps: performing SVD decomposition on the first data matrix to obtain a first matrix, a first diagonal matrix, and a second matrix; adjusting the elements at preset positions in the first diagonal matrix to preset values; determining a second data matrix based on the first matrix, the adjusted first diagonal matrix, and the second matrix; supplementing the data corresponding to the missing positions in the second data matrix to the corresponding missing positions in the first data matrix to generate a target matrix; determining the first normal form modulus of the difference matrix between the second data matrix and the initial second data matrix and the second normal form modulus of the second data matrix; if the value determined by the first normal form modulus and the second normal form modulus is less than a preset threshold, then the data at the missing positions in the target matrix is ​​determined as standard supplementary data; otherwise, the target matrix is ​​updated to the first data matrix.

[0012] Thirdly, embodiments of this application also provide an electronic device, which includes at least a processor and a memory, wherein the processor is used to execute the steps of any of the data processing methods described above when executing a computer program stored in the memory.

[0013] In this embodiment, the electronic device acquires each piece of data received at each time point within a preset time length and determines each piece of data corresponding to a first preset feature, wherein the first preset feature is the transaction amount or transaction time interval; constructs a first data matrix based on the determined data and supplements the missing data in the first data matrix, wherein data received at the same time point are located in the same row of the first data matrix, and data with the same feature are located in the same column; repeats the following steps: performs SVD decomposition on the first data matrix to obtain a first matrix, a first diagonal matrix, and a second matrix; adjusts the elements at preset positions in the first diagonal matrix to preset values; determines the second data matrix based on the first matrix, the adjusted first diagonal matrix, and the second matrix; supplements the data corresponding to the missing positions in the second data matrix to the corresponding missing positions in the first data matrix to generate a target matrix; determines the first normal form modulus of the difference matrix between the second data matrix and the initial second data matrix and the second normal form modulus of the second data matrix; if the value determined by the first normal form modulus and the second normal form modulus is less than a preset threshold, then the data at the missing positions in the target matrix is ​​determined as standard supplementary data; otherwise, the target matrix is ​​updated to the first data matrix. In this embodiment, the electronic device supplements the missing data in the first data matrix and processes the first data matrix to determine the first normal form modulus of the difference matrix between the first data matrix and the initial second data matrix, and the second normal form modulus of the second data matrix. This process continues until the value determined by the first normal form modulus and the second normal form modulus is less than a preset threshold. The data corresponding to the missing position in the second data matrix is ​​then determined as standard supplementary data. This allows for the restoration of a complete target matrix containing the missing data as much as possible, thereby accurately identifying the missing data. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This application provides a schematic diagram of a data processing process as illustrated in its embodiments.

[0016] Figure 2 This application provides a detailed schematic diagram illustrating a data processing procedure as described in an embodiment.

[0017] Figure 3 This is a schematic diagram of a data processing device structure provided in an embodiment of this application;

[0018] Figure 4This is a schematic diagram of an electronic device structure provided in an embodiment of this application. Detailed Implementation

[0019] The present application will now be described in further detail with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present application.

[0020] To accurately identify missing data, embodiments of this application provide a data processing method, apparatus, device, and medium.

[0021] The data processing method includes: an electronic device acquiring each piece of data received at each time point within a preset time length, and determining each piece of data corresponding to a first preset feature, wherein the first preset feature is transaction amount or transaction time interval; constructing a first data matrix based on the determined data, and supplementing the missing data positions in the first data matrix, wherein data received at the same time point are located in the same row of the first data matrix, and data with the same feature are located in the same column; repeatedly performing the following steps: performing SVD decomposition on the first data matrix to obtain a first matrix, a first diagonal matrix, and a second matrix; adjusting the elements at preset positions in the first diagonal matrix to preset values; determining the second data matrix based on the first matrix, the adjusted first diagonal matrix, and the second matrix; supplementing the missing data positions in the second data matrix to the corresponding missing positions in the first data matrix to generate a target matrix; determining the first normal form modulus of the difference matrix between the second data matrix and the initial second data matrix, and the second normal form modulus of the second data matrix; if the value determined by the first normal form modulus and the second normal form modulus is less than a preset threshold, then the data at the missing positions in the target matrix is ​​determined as standard supplementary data; otherwise, the target matrix is ​​updated to the first data matrix. This allows for accurate identification of missing data.

[0022] Example 1:

[0023] Figure 1 This application provides a schematic diagram of a data processing process, which includes the following steps:

[0024] S101: Obtain each data received at each time point within a preset time length, and determine each data corresponding to a first preset feature, wherein the first preset feature is the transaction amount or the transaction time interval.

[0025] The data processing method provided in this application is applied to an electronic device, which can be a smart device such as a PC or a server.

[0026] In order to accurately determine the missing data, the electronic device can first acquire the data received at each time point within a preset time length. The data can be sent to the electronic device by other devices after data processing, or it can be sent to the electronic device by other devices after acquiring the data.

[0027] After acquiring each piece of data, the electronic device can identify each piece of data that corresponds to a first preset feature. The first preset feature is the feature that the user is interested in identifying as missing data. This first preset feature can be sent by the user to the electronic device using their own device, allowing the electronic device to determine the missing data based on this feature. Alternatively, the first preset feature can be selected by the user on a preset page of the electronic device. The first preset feature can be the transaction amount or the transaction time interval, or other features that the user is interested in.

[0028] S102: Construct a first data matrix based on the determined data, and supplement the missing data in the first data matrix, wherein data received at the same time point are located in the same row of the first data matrix, and data with the same feature are located in the same column.

[0029] After determining each data point, the electronic device can construct a first data matrix based on each determined feature. Data received by the electronic device at the same time point are located in the same row of the first data matrix, and data with the same feature are located in the same column.

[0030] After constructing the first data matrix, there are missing positions in the first data matrix. The electronic device fills in the data at the missing positions in the first data matrix. Specifically, for each missing position, the electronic device can obtain any data from the column where the missing position is located and fill in the missing position with that data. It should be noted that the missing positions described in the embodiments of this application are relatively scattered in the first data matrix and account for a small proportion, generally 5%-10%.

[0031] S103: Repeat the following steps: Perform SVD decomposition on the first data matrix to obtain a first matrix, a first diagonal matrix, and a second matrix; adjust the elements at preset positions in the first diagonal matrix to preset values; determine the second data matrix based on the first matrix, the adjusted first diagonal matrix, and the second matrix; supplement the data corresponding to the missing positions in the second data matrix to the corresponding missing positions in the first data matrix to generate a target matrix; determine the first normal form modulus of the difference matrix between the second data matrix and the initial second data matrix and the second normal form modulus of the second data matrix; if the value determined by the first normal form modulus and the second normal form modulus is less than a preset threshold, then the data at the missing positions in the target matrix is ​​determined as standard supplementary data; otherwise, update the target matrix to the first data matrix.

[0032] After obtaining the first data matrix with the missing data filled in, the electronic device repeats the following steps:

[0033] The electronic device can perform SVD decomposition on the first data matrix to obtain a first matrix, a first diagonal matrix, and a second matrix. The specific method for performing SVD decomposition on a given matrix is ​​existing technology and will not be elaborated here. For example, performing SVD decomposition on matrix Y yields Y = LSR. T In this matrix, L is the first matrix, S is the first diagonal matrix, and R is the second matrix.

[0034] The electronic device can adjust the elements at preset positions in the first diagonal matrix to preset values, where the preset value can be 0. The electronic device can determine the second data matrix based on the first matrix, the adjusted first diagonal matrix, and the second matrix. Specifically, the electronic device can determine the matrix obtained by multiplying the first matrix and the adjusted first diagonal matrix, and determine the matrix obtained by multiplying the first matrix and the second matrix as the second data matrix.

[0035] After obtaining the second data matrix, the electronic device can supplement the missing data at the corresponding missing positions in the first data matrix to generate the target matrix. Specifically, for each missing position in the first data matrix, the electronic device can determine the row and column of the missing position, obtain the data of that row and column in the second data matrix, and supplement the missing data at the missing position. After obtaining the target matrix, the electronic device can determine whether the target matrix is ​​converged. Specifically, the electronic device can determine the first normal form modulus of the difference matrix between the second data matrix and the initial second data matrix, and the second normal form modulus of the second data matrix. A value is determined based on the first normal form modulus and the second normal form modulus. In one possible implementation, the electronic device can determine this value based on the difference between the first normal form modulus and the second normal form modulus. If this value is lower than a preset value, the target matrix is ​​determined to be converged; if the value is not lower than the preset value, the target matrix is ​​determined to be non-converged. If the value determined based on the first normal form modulus and the second normal form modulus is less than a preset threshold, the data at the missing positions in the target matrix is ​​determined as standard supplementary data; otherwise, the target matrix is ​​updated to the first data matrix. The process continues until the value determined by the first and second normal form moduloes is less than a preset threshold. At this point, the repeated steps end.

[0036] It should be noted that the first data matrix to fill in the missing data can also be called the response matrix Y. Specifically, the electronic device can first perform SVD decomposition on Y, Y = LSR. T S is a diagonal matrix with its diagonal elements arranged in descending order. The elements at predetermined positions in the diagonal matrix are adjusted to predetermined values ​​to obtain a diagonal matrix T(S,r). Let the second data matrix A = LT(S,r)R. T As a principal component decomposition of Y, the data corresponding to the missing positions in the second data matrix are added to the corresponding missing positions in the first data matrix. The above operation is repeated until the second data matrix A converges, thus obtaining the estimate of the data at the missing positions in the first data matrix Y.

[0037] Specifically, the electronic device can obtain the target matrix using the following formula:

[0038]

[0039] Where L is the newly obtained target matrix, Y is the first data matrix, A is the second data matrix, and M = {(i,j):Y} ij} is the set of observable elements of the first data matrix, that is, the set of elements at all positions except for the missing positions, M. c P is the set of missing positions in Y. For example, if the data in the first row and first column is not missing in the first data matrix, then the set M contains (1,1). M(Y) represents the projection of the first data matrix Y onto the set M, that is, the value Y when the element coordinates, i.e., the row and column (i,j) ∈ M of the element. ij Otherwise, the value is 0. Identify the second data matrix A in set M c The projection on the surface.

[0040] The method provided in this application is equivalent to using the sparsity of the first data matrix composed of features to be predicted during the data processing stage to supplement missing data through an iterative algorithm. This method can be applied to related fields such as data security and network data analysis.

[0041] In a real-world scenario, the SVD decomposition module of the electronic device's response variable matrix can perform SVD decomposition on the first data matrix, and the missing sample module of the electronic device can determine the data at the missing positions based on the decomposed first matrix, first diagonal matrix, and second matrix.

[0042] In this embodiment, the electronic device supplements the missing data in the first data matrix and processes the first data matrix to determine the first normal form modulus of the difference matrix between the first data matrix and the initial second data matrix, and the second normal form modulus of the second data matrix. This process continues until the value determined by the first normal form modulus and the second normal form modulus is less than a preset threshold. The data corresponding to the missing position in the second data matrix is ​​then determined as standard supplementary data. This allows for the restoration of a complete target matrix containing the missing data as much as possible, thereby accurately identifying the missing data.

[0043] Example 2:

[0044] To supplement the missing data in the first data matrix, based on the above embodiments, in this embodiment, the supplementation of the missing data in the first data matrix includes:

[0045] Fill the missing positions in the first data matrix with preset data; or...

[0046] For each missing location, determine the corresponding feature at that location, and then supplement the missing location in the first data matrix with the data corresponding to that feature; or,

[0047] For each missing position, the average value of each data point that has the same feature as the data at that missing position is determined, and the average value is added to the missing position in the first data matrix.

[0048] The electronic device can fill in the missing positions in the first data matrix with preset data.

[0049] The electronic device can also determine the feature corresponding to each missing position and supplement the missing position in the first data matrix with the data stored for that feature.

[0050] The electronic device can also determine the average value of each data point with the same feature as the data at the missing position for each missing position, that is, determine the average value of each data point in the column where the missing position is located, and fill the missing position in the first data matrix with the average value.

[0051] In order to accurately obtain the second data matrix, based on the above embodiments, in this embodiment, adjusting the elements at preset positions in the first diagonal matrix to preset values ​​includes:

[0052] Sort each element of the first diagonal matrix according to the row where the element is located, obtain each element after the preset number of elements in the sorting result, and adjust each obtained element to a preset value.

[0053] The electronic device can sort each element of the diagonal in the first diagonal matrix according to its row, obtain the sorting result, and then obtain each element after the preset number of elements in the sorted structure and adjust each element to a preset value. Specifically, it adjusts each element in the first diagonal matrix to a preset value. The electronic device is equivalent to keeping the first preset number of diagonal elements in the first diagonal matrix unchanged, and changing all other diagonal elements to a preset value, where the preset value can be 0.

[0054] In order to accurately obtain the second data matrix, based on the above embodiments, the preset quantity in this application embodiment is determined by the following formula:

[0055]

[0056] Where, τ n = (logn) -1 loglogn, k is a positive integer less than x, x is the number of rows in the first diagonal matrix, n is the number of rows in the first data matrix, q is the number of columns in the first data matrix, d k Let d be the k-th element on the diagonal of the first diagonal matrix. k+1 It is the (k+1)th element on the diagonal of the first diagonal matrix.

[0057] Electronic devices can determine the preset quantity using the following formula:

[0058]

[0059] Where, τn = (logn) -1 loglogn, k is a positive integer less than x, x is the number of rows in the first diagonal matrix, n is the number of rows in the first data matrix, q is the number of columns in the first data matrix, d k Let d be the k-th element on the diagonal of the first diagonal matrix. k+1 It is the (k+1)th element on the diagonal of the first diagonal matrix.

[0060] It should be noted that, This means that when 0 < k < x - 1, In this case, the maximum value of k is obtained. This is equivalent to selecting the optimal parameter r, i.e., selecting the optimal preset quantity, using the optimization method described in the embodiments of this application. In practical scenarios, the preset quantity can be determined by the optimal rank selection module of the electronic device.

[0061] To accurately determine the data at the missing location, based on the above embodiments, in this application embodiment, determining the numerical value according to the first paradigm modulus and the second paradigm modulus includes:

[0062] Determine the ratio of the first normal form module to the second normal form module, and determine the numerical value based on the ratio.

[0063] After acquiring the first normal form modulus and the second normal form modulus, the electronic device can determine the ratio of the first normal form modulus and the second normal form modulus, and determine the value based on the ratio. Specifically, the ratio can be determined as a numerical value.

[0064] Specifically, electronic devices can determine this value using the following formula:

[0065]

[0066] Among them, A new For the second data matrix, A old Let ||A be the initial second data matrix. new -A old || F For the first normal form module, ||A old || F It is a second normal form module.

[0067] Among them, When ε is a preset threshold as described in the embodiments of this application, the electronic device determines the data at the missing positions in the target matrix as standard supplementary data.

[0068] Example 3:

[0069] To improve the accuracy of data determination, based on the above embodiments, the method in this application embodiment further includes:

[0070] Determine other data in each of the data that correspond to the second preset feature; wherein, the second preset feature is a feature associated with the first preset feature, and the second preset feature is any one of company area, business volume, and company logo;

[0071] A third data matrix is ​​constructed based on the other data obtained, wherein the third data matrix is ​​constructed in the same way as the first data matrix;

[0072] When the values ​​determined by the first normal form module and the second normal form module are less than a preset threshold, the second data matrix is ​​decomposed by SVD to obtain a third matrix, a second diagonal matrix, and a fourth matrix; based on the generalized least squares method, the third data matrix, the preset identity matrix, and the number of rows of the first data matrix are processed to obtain matrix U;

[0073] The regression coefficient matrix and outlier matrix are determined based on the number of rows in the matrix U, the second diagonal matrix, the fourth matrix, and the first data matrix.

[0074] The standard data corresponding to the abnormal data is determined based on the difference between the data of each non-preset data in the outlier matrix and the corresponding data in the first data matrix.

[0075] In real-world scenarios, the received data with the first preset feature may contain anomalous data. In order to accurately determine the anomalous data, the electronic device can make a determination based on other data associated with the first preset feature.

[0076] Specifically, the electronic device can determine other data corresponding to a second preset feature in each acquired data set. The second preset feature is a feature associated with a first preset feature, and can be any one of the following: company area, company business volume, or company identifier. After acquiring each other data set, a data matrix can be constructed based on the acquired data. For ease of distinction, this data matrix can be called the third data matrix, and the construction method of the third data matrix is ​​the same as that of the first data matrix. After constructing the third data matrix, the electronic device can acquire a second data matrix when the data determined by the first and second normal form moduloes is less than a preset threshold, and perform SVD decomposition on this second data matrix to obtain a third matrix, a second diagonal matrix, and a fourth matrix. The specific method for performing SVD decomposition on the matrix is ​​existing technology and will not be elaborated here.

[0077] The electronic device can use the generalized least squares method to process the number of data points in each of the third data matrix, the third matrix, the preset identity matrix, and the first data matrix to obtain matrix U. Specifically, the electronic device can determine the outlier matrix C based on the high-dimensional sparse low-rank regression model Y = XB + C + E, where Y ∈ R. n*q , X∈R n*p , B∈R p*q , C∈R n *q , E∈R n*q , Y∈R n*q , X∈R n*p , B∈R p*q , C∈R n*q , E∈R n*q This refers to a matrix where Y, X, B, C, and E are all n-row, q-column matrices. Y is the first data matrix used to fill in missing data positions, and it is also the first data matrix constructed from the features of interest. X∈R n*p The predictor variables are the third data matrix constructed from known features, B∈R. p*q It is the regression coefficient matrix, C∈R n*q It is an outlier matrix, E∈R n*q Y is a random error matrix with each row being independent and identically distributed. Matrix B and matrix C are sparse matrices. It should be noted that the second data matrix when the data determined by the first and second normal form moduli is less than a preset threshold can be considered as the first data matrix after supplementing the missing data. Therefore, Y is the second data matrix when the data determined by the first and second normal form moduli is less than a preset threshold, as described in the embodiments of this application. E is ignored in the embodiments of this application. A slight modification to the regression model can yield the following result: Where n is the number of rows in the first data matrix, and In is the identity matrix with n rows. This transformed estimate of the regression coefficient matrix can automatically reconstruct the results of outlier detection and regression coefficient matrix estimation from the original problem. This transformation method is equivalent to regularizing the third data matrix X, making each column orthogonal and its L2 modulus uniform. The true regression coefficient matrix B is sparse and low-rank, while the outlier matrix C is row-sparse, meaning that only a small number of data points are outliers. Since both matrices B and C are sparse, the above formula can be directly applied to... Treat it as a new predictor variable. This is considered as the corresponding coefficient matrix. Electronic devices can... Perform SVD decomposition to obtain therefore Furthermore, performing SVD decomposition on the second data matrix yields the third matrix, the second diagonal matrix, and the fourth matrix. Therefore, the second data matrix is ​​equal to D′, and the fourth matrix is ​​equal to V′. Equal to the third matrix, therefore the electronic device can be based on Since it is equal to the third matrix, the generalized least squares method is used to obtain matrix U.

[0078] To improve computational efficiency, the regression coefficients for each column of the third matrix can be calculated in parallel. Specifically, assume that the second data matrix A is decomposed into SVD: A = ZDV. T Where Z is the third matrix, D is the second diagonal matrix, and V is the fourth matrix, let in, This is the column matrix corresponding to the first column of the fourth matrix. Let r be the column matrix corresponding to the r-th column of the fourth matrix, where r is the column number of the third matrix, the second diagonal matrix, and the fourth matrix. It is the first diagonal element of the second diagonal matrix. The r-th diagonal element in the second diagonal matrix. This is the column matrix corresponding to the first column of the third matrix. This is the column matrix corresponding to the r-th column in the third matrix.

[0079] based on Electronic devices can process the components of matrix U. The column matrix is ​​computed in parallel using the following steps:

[0080]

[0081] in, Let be the estimated value of the column matrix corresponding to the k-th column of matrix U. Let X be the column matrix corresponding to the k-th column in the third matrix, X be the third data matrix, n be the number of rows in the first data matrix, and λ be the column matrix corresponding to the k-th column in the third matrix. k For the k-th value in the generalized least squares method, the specific value of λ is... k No restrictions are placed on the specific numerical values ​​used here.

[0082] In real-world scenarios, the parallel computation feature vector module of an electronic device can be used to determine matrix U.

[0083] Electronic devices can obtain the column matrix corresponding to each column of matrix U in this way. Electronic devices can obtain the regression coefficient matrix and outlier matrix through the following steps:

[0084]

[0085] Among them, M k for The column matrix corresponding to the k-th column of this matrix It is the k-th diagonal element in the second diagonal matrix. This is the column matrix corresponding to the k-th column of the third matrix. The column matrix corresponding to the k-th column of the fourth matrix is ​​obtained. The electronic device can then obtain the k-th matrix of matrix B and the k-th matrix of matrix C respectively.

[0086] After obtaining each matrix corresponding to matrix B and matrix C, the electronic device can add each matrix corresponding to matrix B to obtain matrix B, and add each matrix corresponding to matrix C to obtain matrix C.

[0087] After obtaining matrix Y, the electronic device can... Matrix B and matrix C are obtained, which together form the regression coefficient matrix and the outlier matrix. The electronic device can derive these matrices from matrix U, the second diagonal matrix, the fourth matrix, and the number of rows in the first data matrix. The outlier matrix has the same number of rows and columns as the first data matrix, and any data in the outlier matrix that is not part of a preset set of data represents the outlier data in the first data matrix. This preset set of data is typically 0.

[0088] After obtaining the outlier matrix, the electronic device can determine the standard data corresponding to each outlier data point by comparing it with the corresponding data in the first data matrix. Specifically, the electronic device can determine the standard data corresponding to the outlier data by using the negative of the difference. In practical scenarios, this determination of outlier data can be performed by modules such as the outlier matrix calculation module and the regression coefficient estimation module.

[0089] In today's era of rapid information technology development, massive amounts of network-related data are widely used in various fields to promote industry development and technological progress, such as diffusion networks and recommender systems. These data are characterized by significant correlations. This application's embodiment is equivalent to using some known features to predict data for variables of interest; in this embodiment, it uses data with a second preset feature to predict data with a first preset feature. Sparse rank-reduced regression is an important tool and method for mining the network structure of large-scale response predictors. It uses a linear model with sparse coefficients to characterize the correlation between known features and the variable to be predicted. Therefore, this application's embodiment can accurately identify outlier data. This application's embodiment essentially provides a method for outlier detection and model parameter estimation for network-related data with missing data.

[0090] For the analysis and inference of sparse reduced-rank regression models, data quality is particularly important. However, the data used in practice often has many problems: during data acquisition and transmission, there are unavoidable issues such as outlier interference and missing response data caused by objective or human factors. This may affect the safe use of this data, causing model estimation failure and data invalidation. In addition, massive amounts of data also put a strain on computing resources. Existing research has proposed some targeted solutions to these problems, such as using model sparsity for outlier inference and using known data to estimate missing data. However, these methods are difficult to comprehensively consider the data quality issues in network inference and lack a systematic data processing and analysis workflow. To solve the data security problem in network data analysis, it is necessary to accurately identify outliers in the data. The method for outlier detection and parameter estimation of multi-response linear regression models with missing data proposed in this application comprehensively considers the problems of large computational load and missing response variables. This application embodiment is equivalent to obtaining the estimate of the outlier matrix C and the regression model coefficient matrix B from the third data matrix X and the first data matrix Y containing the data at the missing positions under certain assumptions.

[0091] In this way, electronic devices can identify outliers even when data is partially missing or outliers are present, and obtain estimates from the corresponding models.

[0092] For network-related data, this embodiment first abstracts them into a numerical matrix. Non-numerical features can be represented numerically using encoding or classification methods. Each row represents a received data instance, and each column corresponds to a feature. A linear model is established to characterize the correlation between known features and the features to be predicted. It is assumed that data gaps exist only in the variable to be predicted, and outliers are defined as sparse (i.e., only a small number of data points contain outliers) and fixed perturbations in the variable to be predicted. Obtaining the regression coefficient matrix is ​​equivalent to obtaining the relationship between known features and relevant features of interest in the case of partial data gaps and outliers. This embodiment uses a linear model to characterize the correlation between known features and relevant features of interest, assuming that data gaps exist only in the variable to be predicted, and defining outliers as sparse and fixed perturbations in the variable to be predicted. Based on this, this embodiment can accurately identify outlier data using the method described above.

[0093] The method proposed in this application addresses the problems of outlier interference and missing data in network data association mining. It presents a workflow for data processing, outlier identification, and model estimation for large-scale linear models. This solves the problems of existing multi-response high-dimensional sparse regression model estimation and inference methods, which cannot simultaneously handle computationally intensive data, outlier interference, and missing data in some predictors. It ensures the safe and full utilization of data, maintaining data availability and model estimation effectiveness even with poor data quality. The method proposed in this application is applicable to network data association mining with missing data and outlier interference, enabling the estimation and outlier detection of large-scale multi-response linear regression models. Furthermore, it utilizes parallel computing during the computation process, improving the algorithm's computational efficiency and speed.

[0094] Figure 2 This application provides a detailed schematic diagram of a data processing process, which includes the following steps:

[0095] S201: Obtain each data received at each time point within a preset time length, and determine each data corresponding to the first preset feature and each other data corresponding to the second preset feature.

[0096] S202: Construct a first data matrix based on the determined data, and supplement the missing data in the first data matrix.

[0097] S203: Based on the first data matrix after supplementing the missing data, determine the standard supplementary data in the first data matrix.

[0098] Specifically, the following steps can be used to determine the standard supplementary data, and the following steps can be repeated: Perform SVD decomposition on the first data matrix to obtain a first matrix, a first diagonal matrix, and a second matrix; adjust the elements at preset positions in the first diagonal matrix to preset values; determine the second data matrix based on the first matrix, the adjusted first diagonal matrix, and the second matrix; supplement the missing data in the second data matrix to the corresponding missing positions in the first data matrix to generate the target matrix; determine the first normal form modulus of the difference matrix between the second data matrix and the initial second data matrix, and the second normal form modulus of the second data matrix; if the value determined by the first normal form modulus and the second normal form modulus is less than a preset threshold, then the data at the missing positions in the target matrix is ​​determined as standard supplementary data; otherwise, the target matrix is ​​updated to the first data matrix.

[0099] S204: Add standard supplementary data to the missing positions in the first data matrix.

[0100] S205: Construct a third data matrix based on the other data obtained.

[0101] S206: Perform SVD decomposition on the second data matrix corresponding to the first normal form module and the second normal form module when the values ​​are less than a preset threshold to obtain the third matrix, the second diagonal matrix and the fourth matrix; based on the generalized least squares method, process the number of rows of the third data matrix, the third matrix, the preset identity matrix and the first data matrix to obtain matrix U.

[0102] Specifically, how to determine matrix U has been described in the above embodiments and will not be repeated here.

[0103] S207: Determine the regression coefficient matrix and outlier matrix based on the number of rows in matrix U, the second diagonal matrix, the fourth matrix, and the first data matrix.

[0104] S208: Determine the standard data corresponding to the abnormal data based on the difference between the data of each non-preset data in the outlier matrix and the corresponding data in the first data matrix.

[0105] It should be noted that in real-world scenarios, electronic devices may also determine the third data matrix first. Specifically, whether or not the electronic device determines the third data matrix first is not restricted here.

[0106] In one possible implementation, the regression model can be written as In the form of, where E is negligible, we need to find The estimate, when subjected to SVD decomposition, yields:

[0107]

[0108] in,

[0109] and,

[0110] Where, U∈R (p+n)*r ,V∈R q*r S = diag(σ1,…,σ) r )∈R r*r ,make Then we have n - 1 Z T Z = I r Where U∈R (p+n)*r This refers to U being a (p+n) row, r column matrix, where V∈R q*r and S∈R r*r This refers to V and S being r-row, r-column matrices.

[0111] Example 4:

[0112] To improve the accuracy of data determination, based on the above embodiments, the method in this application embodiment further includes:

[0113] If a prediction request carrying each data to be predicted received at each time point within the preset time length is received, then the data to be processed corresponding to the second preset feature in each data to be predicted is determined; a fourth data matrix is ​​constructed based on the obtained data to be processed; wherein, the construction process of the fourth data matrix is ​​the same as that of the first data matrix;

[0114] The matrix obtained by multiplying the fourth data matrix by the regression coefficient matrix is ​​determined as the target data matrix;

[0115] Based on the data contained in the target matrix, determine each target data in the data to be predicted that corresponds to the first preset feature.

[0116] After the electronic device obtains the regression coefficient matrix, it can obtain the correlation between the data of the first preset feature and the data of the second preset feature. Therefore, if the data of the second preset feature is known, the data of the first preset feature can be predicted according to the regression coefficient matrix. Then, based on the predicted data of the first preset feature and the received data of the first preset feature, the missing data in the data of the first preset feature can be determined directly.

[0117] Specifically, when a user needs to predict data with a first preset feature, they can select each piece of data to be predicted received at each time point within a preset time length on the preset page of their device and click a preset button, such as the "Predict" button. At this time, the electronic device can receive a prediction request carrying each piece of data to be predicted received at each time point within the preset time length. Upon receiving the prediction request, the electronic device can determine the data to be processed corresponding to the second preset feature in each piece of data to be predicted carried in the prediction request, and construct a fourth data matrix based on the obtained data to be processed. The construction process of the fourth data matrix is ​​the same as that of the first data matrix, and will not be described again here.

[0118] After obtaining the fourth data matrix, the electronic device can determine the target data matrix by multiplying the fourth data matrix by the regression coefficient matrix. Once the target data matrix is ​​obtained, the electronic device can determine each target data point in the data to be predicted that corresponds to the first preset feature, based on the data contained within the target data matrix.

[0119] To improve the accuracy of the predicted data, based on the above embodiments, the method in this application embodiment further includes:

[0120] The regression coefficient matrix is ​​updated by multiplying the inverse of the first data matrix (after adjusting for outliers and filling in missing data) and the third data matrix.

[0121] To improve the accuracy of determining the regression coefficient matrix, after acquiring abnormal and missing data, the electronic device can adjust the abnormal data and fill in the missing data, obtain the first data matrix after adjusting and filling in the abnormal data, and obtain the product of the first data matrix after adjusting and filling in the abnormal data and the inverse matrix of the third data matrix. The matrix corresponding to this product is used to update the regression coefficient matrix.

[0122] Specifically, the electronic device can let Y′=AC′ be the corrected response variable matrix, where A is the second data matrix when the values ​​determined by the first normal form modulus and the second normal form modulus are less than the threshold, which can be considered as the first data matrix after filling in the missing data, and C′ is the outlier matrix. Therefore, Y′ is the first data matrix after adjusting the outlier data and filling in the missing data as described in the embodiments of this application. Then, the regression coefficient matrix B is estimated by using the regression model Y′=XB+E that does not contain outliers. Specifically, E is the random error matrix with independent and identical distribution for each row, which is ignored here, i.e., Y′=XB. Therefore, the electronic device can redetermine the regression coefficient matrix B based on Y′ and X using the generalized least squares method, thereby improving the accuracy of the regression coefficient matrix determination.

[0123] Specifically, electronic devices can redetermine the regression coefficient matrix using the following formula:

[0124] Step 1: Adjust the first data matrix A after correcting outlier data and filling in missing data. re SVD decomposition yields Where 1 ≤ k ≤ x, k represents any column, and x is the rank of the first data matrix. For A re The column matrix corresponding to the k-th column of the first matrix obtained by SVD decomposition. For A re The k-th diagonal element of the diagonal matrix obtained by SVD decomposition. For A re The column matrix corresponding to the k-th column of the last matrix obtained by SVD decomposition.

[0125] Step 2: Find the optimal rank parameter

[0126] in, Where τ n = (logn) -1 loglogn, This refers to the k-th diagonal element of the diagonal matrix obtained by decomposing the first data matrix after adjusting for abnormal data and supplementing with accurate data.

[0127] Step 3: [The sentence is incomplete and requires more context.] of Each component is computed in parallel for the following steps:

[0128]

[0129]

[0130] in, This is the column matrix corresponding to the k-th column of the first matrix obtained by performing SVD decomposition on the regression coefficient matrix B. For A re The column matrix corresponding to the k-th column of the first matrix obtained by SVD decomposition, X is the third data matrix, n is the number of rows in the first data matrix, and λ k For the k-th value in the generalized least squares method, the specific value of λ is... k No restrictions are placed on the numerical values ​​used here. This is the k-th matrix of the regression coefficient matrix. For A re The k-th diagonal element of the diagonal matrix obtained by SVD decomposition. For A re The column matrix corresponding to the k-th column of the last matrix obtained by SVD decomposition.

[0131] After obtaining each matrix corresponding to the regression coefficient matrix, the electronic device can add up each matrix corresponding to the regression coefficient matrix to obtain the regression coefficient matrix.

[0132] Specifically, the regression coefficient matrix B is decomposed into SVD: B = U re D re V re T The first data matrix after adjusting for outliers and filling in missing data is Y′=XB, i.e., Y′=XU. re D re V re T , Y′ is decomposed into SVD, Y′=U y D y V y T XU re =U y D re =D y V re =V y Based on XU re =U yThe electronic device uses the above-mentioned generalized least squares method to determine U. re Each

[0133] To address the challenges of data quality and safe use in network data mining, such as the computational difficulty caused by large data volumes, outlier interference, and missing response data in some predictors, this application proposes a novel parameter estimation method for multi-response linear regression models. This method utilizes low-rank characteristics to estimate missing data, treating it as part of the sparse coefficients of the linear model. Furthermore, it employs parallel computing to improve computational efficiency, ultimately identifying outliers and estimating model coefficient parameters. This enables safe and full utilization of the data and establishes a complete workflow encompassing data processing, anomaly identification, and model estimation.

[0134] Example 5:

[0135] Figure 3 This application provides a schematic diagram of a data processing device structure, which includes:

[0136] The acquisition and determination module 301 is used to acquire each data received at each time point within a preset time length, and determine each data corresponding to a first preset feature, wherein the first preset feature is the transaction amount or the transaction time interval;

[0137] The supplementary module 302 is used to construct a first data matrix based on the determined data and supplement the missing data in the first data matrix, wherein data received at the same time point are located in the same row of the first data matrix and data with the same feature are located in the same column;

[0138] Processing module 303 is used to repeatedly execute the following steps: performing Singular Value Decomposition (SVD) on the first data matrix to obtain a first matrix, a first diagonal matrix, and a second matrix; adjusting the elements at preset positions in the first diagonal matrix to preset values; determining a second data matrix based on the first matrix, the adjusted first diagonal matrix, and the second matrix; supplementing the data corresponding to the missing positions in the second data matrix to the corresponding missing positions in the first data matrix to generate a target matrix; determining the first normal form modulus of the difference matrix between the second data matrix and the initial second data matrix and the second normal form modulus of the second data matrix; if the value determined based on the first normal form modulus and the second normal form modulus is less than a preset threshold, then the data at the missing positions in the target matrix is ​​determined as standard supplementary data; otherwise, the target matrix is ​​updated to the first data matrix.

[0139] In one possible implementation, the supplementation module 302 is specifically used to supplement preset data to the missing positions in the first data matrix; or, for each missing position, to determine the feature corresponding to the missing position and supplement the data stored for the feature to the missing position in the first data matrix; or, for each missing position, to determine the average value of each data that has the same feature as the data at the missing position and supplement the average value to the missing position in the first data matrix.

[0140] In one possible implementation, the processing module 303 is specifically used to sort each element of the diagonal in the first diagonal matrix according to the row where the element is located, obtain each element after the preset number of elements in the sorting result, and adjust each obtained element to a preset value.

[0141] In one possible implementation, the processing module 303 is further configured to determine the preset quantity using the following formula: Where, τ n = (logn) - 1 loglogn, k is a positive integer less than x, x is the number of rows in the first diagonal matrix, n is the number of rows in the first data matrix, q is the number of columns in the first data matrix, d k Let d be the k-th element on the diagonal of the first diagonal matrix. k+1 It is the (k+1)th element on the diagonal of the first diagonal matrix.

[0142] In one possible implementation, the processing module 303 is specifically used to determine the ratio of the first paradigm module to the second paradigm module, and to determine a numerical value based on the ratio.

[0143] In one possible implementation, the processing module 303 is further configured to determine other data corresponding to a second preset feature in each of the data; wherein the second preset feature is a feature associated with the first preset feature, and the second preset feature is any one of company area, business volume, and company logo; construct a third data matrix based on the obtained other data, wherein the third data matrix is ​​constructed in the same way as the first data matrix; perform SVD decomposition on the second data matrix corresponding to the value determined by the first normal form modulus and the second normal form modulus being less than a preset threshold to obtain a third matrix, a second diagonal matrix, and a fourth matrix; process the third data matrix, the third matrix, the preset identity matrix, and the number of rows of the first data matrix based on the generalized least squares method to obtain matrix U; determine a regression coefficient matrix and an outlier matrix based on matrix U, the second diagonal matrix, the fourth matrix, and the number of rows of the first data matrix; and determine the standard data corresponding to the outlier data based on the difference between the data of each non-preset data in the outlier matrix and the data corresponding to that data in the first data matrix.

[0144] In one possible implementation, the processing module 303 is further configured to, if receiving a prediction request carrying each data to be predicted received at each time point within the preset time length, determine the data to be processed corresponding to the second preset feature in each data to be predicted; construct a fourth data matrix based on the acquired data to be processed; wherein the construction process of the fourth data matrix is ​​the same as that of the first data matrix; determine the target data matrix by multiplying the fourth data matrix by the regression coefficient matrix; and determine each target data corresponding to the first preset feature in the data to be predicted based on the data contained in the target matrix.

[0145] In one possible implementation, the processing module 303 is further configured to update the regression coefficient matrix by multiplying the product of the first data matrix after adjusting for abnormal data and supplementing for missing data and the inverse matrix of the third data matrix.

[0146] Example 6:

[0147] Based on the above embodiments, this application also provides an electronic device, such as... Figure 4 As shown, it includes: processor 401, communication interface 402, memory 403 and communication bus 404, wherein processor 401, communication interface 402 and memory 403 communicate with each other through communication bus 404.

[0148] The memory 403 stores a computer program, which, when executed by the processor 401, causes the processor 401 to perform the following steps:

[0149] Acquire each data received at each time point within a preset time length, and determine each data corresponding to a first preset feature, wherein the first preset feature is the transaction amount or the transaction time interval;

[0150] A first data matrix is ​​constructed based on the determined data, and data at missing positions in the first data matrix is ​​supplemented. Data received at the same time point are located in the same row of the first data matrix, and data with the same feature are located in the same column.

[0151] Repeat the following steps: Perform SVD decomposition on the first data matrix to obtain a first matrix, a first diagonal matrix, and a second matrix; adjust the elements at preset positions in the first diagonal matrix to preset values; determine the second data matrix based on the first matrix, the adjusted first diagonal matrix, and the second matrix; supplement the missing data in the second data matrix to the corresponding missing positions in the first data matrix to generate a target matrix; determine the first normal form modulus of the difference matrix between the second data matrix and the initial second data matrix, and the second normal form modulus of the second data matrix; if the value determined by the first normal form modulus and the second normal form modulus is less than a preset threshold, then the data at the missing positions in the target matrix is ​​determined as standard supplementary data; otherwise, update the target matrix to the first data matrix.

[0152] Furthermore, the processor 401 is specifically configured to supplement the missing positions in the first data matrix with preset data; or,

[0153] For each missing location, determine the feature corresponding to that location, and then supplement the missing location in the first data matrix with the data corresponding to that feature; or,

[0154] For each missing position, the average value of each data point that has the same feature as the data at that missing position is determined, and the average value is added to the missing position in the first data matrix.

[0155] Furthermore, the processor 401 is specifically configured to sort each element of the diagonal in the first diagonal matrix according to the row in which the element is located, obtain each element after the preset number of elements in the sorting result, and adjust each obtained element to a preset value.

[0156] Furthermore, the processor 401 is specifically configured to determine the preset quantity using the following formula:

[0157]

[0158] Where, τ n = (logn) -1loglogn, k is a positive integer less than x, x is the number of rows in the first diagonal matrix, n is the number of rows in the first data matrix, q is the number of columns in the first data matrix, d k Let d be the k-th element on the diagonal of the first diagonal matrix. k+1 It is the (k+1)th element on the diagonal of the first diagonal matrix.

[0159] Furthermore, the processor 401 is specifically used to determine the ratio of the first paradigm module to the second paradigm module, and to determine a numerical value based on the ratio.

[0160] Furthermore, the processor 401 is also configured to determine other data in each of the data that correspond to the second preset feature; wherein, the second preset feature is a feature associated with the first preset feature, and the second preset feature is any one of company area, business volume, and company identifier;

[0161] A third data matrix is ​​constructed based on the other data obtained, wherein the third data matrix is ​​constructed in the same way as the first data matrix;

[0162] When the values ​​determined by the first normal form module and the second normal form module are less than a preset threshold, the second data matrix is ​​decomposed by SVD to obtain a third matrix, a second diagonal matrix, and a fourth matrix; based on the generalized least squares method, the third data matrix, the preset identity matrix, and the number of rows of the first data matrix are processed to obtain matrix U;

[0163] The regression coefficient matrix and outlier matrix are determined based on the number of rows in the matrix U, the second diagonal matrix, the fourth matrix, and the first data matrix.

[0164] The standard data corresponding to the abnormal data is determined based on the difference between the data of each non-preset data in the outlier matrix and the corresponding data in the first data matrix.

[0165] Furthermore, the processor 401 is also configured to, if it receives a prediction request carrying each data to be predicted received at each time point within the preset time length, determine the data to be processed corresponding to the second preset feature in each data to be predicted; and construct a fourth data matrix based on the acquired data to be processed; wherein the construction process of the fourth data matrix is ​​the same as that of the first data matrix;

[0166] The matrix obtained by multiplying the fourth data matrix by the regression coefficient matrix is ​​determined as the target data matrix;

[0167] Based on the data contained in the target matrix, determine each target data in the data to be predicted that corresponds to the first preset feature.

[0168] Furthermore, the processor 401 is also used to update the regression coefficient matrix by multiplying the product of the first data matrix after adjusting abnormal data and supplementing missing data and the inverse matrix of the third data matrix.

[0169] The communication bus mentioned in the above server can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus.

[0170] Communication interface 402 is used for communication between the above-mentioned electronic device and other devices.

[0171] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0172] The processors mentioned above can be general-purpose processors, including central processing units, network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits, field-programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0173] Example 7:

[0174] Based on the above embodiments, this application also provides a computer-readable storage medium storing a computer program executable by an electronic device. When the program is run on the electronic device, the electronic device performs the following steps:

[0175] The memory stores a computer program that, when executed by the processor, causes the processor to perform the following steps:

[0176] Acquire each data received at each time point within a preset time length, and determine each data corresponding to a first preset feature, wherein the first preset feature is the transaction amount or the transaction time interval;

[0177] A first data matrix is ​​constructed based on the determined data, and data at missing positions in the first data matrix is ​​supplemented. Data received at the same time point are located in the same row of the first data matrix, and data with the same feature are located in the same column.

[0178] Repeat the following steps: Perform SVD decomposition on the first data matrix to obtain a first matrix, a first diagonal matrix, and a second matrix; adjust the elements at preset positions in the first diagonal matrix to preset values; determine the second data matrix based on the first matrix, the adjusted first diagonal matrix, and the second matrix; supplement the missing data in the second data matrix to the corresponding missing positions in the first data matrix to generate a target matrix; determine the first normal form modulus of the difference matrix between the second data matrix and the initial second data matrix, and the second normal form modulus of the second data matrix; if the value determined by the first normal form modulus and the second normal form modulus is less than a preset threshold, then the data at the missing positions in the target matrix is ​​determined as standard supplementary data; otherwise, update the target matrix to the first data matrix.

[0179] In one possible implementation, the data used to supplement the missing positions in the first data matrix includes:

[0180] Fill the missing positions in the first data matrix with preset data; or...

[0181] For each missing location, determine the feature corresponding to that location, and then supplement the missing location in the first data matrix with the data corresponding to that feature; or,

[0182] For each missing position, the average value of each data point that has the same feature as the data at that missing position is determined, and the average value is added to the missing position in the first data matrix.

[0183] In one possible implementation, adjusting the element at a preset position in the first diagonal matrix to a preset value includes:

[0184] Sort each element of the first diagonal matrix according to the row where the element is located, obtain each element after the preset number of elements in the sorting result, and adjust each obtained element to a preset value.

[0185] In one possible implementation, the preset quantity is determined by the following formula:

[0186]

[0187] Where, τ n = (logn) -1loglogn, k is a positive integer less than x, x is the number of rows in the first diagonal matrix, n is the number of rows in the first data matrix, q is the number of columns in the first data matrix, d k Let d be the k-th element on the diagonal of the first diagonal matrix. k+1 It is the (k+1)th element on the diagonal of the first diagonal matrix.

[0188] In one possible implementation, determining the numerical value based on the first paradigm modulus and the second paradigm modulus includes:

[0189] Determine the ratio of the first normal form module to the second normal form module, and determine the numerical value based on the ratio.

[0190] In one possible implementation, the method further includes:

[0191] Determine other data in each of the data that correspond to the second preset feature; wherein, the second preset feature is a feature associated with the first preset feature, and the second preset feature is any one of company area, business volume, and company logo;

[0192] A third data matrix is ​​constructed based on the other data obtained, wherein the third data matrix is ​​constructed in the same way as the first data matrix;

[0193] When the values ​​determined by the first normal form module and the second normal form module are less than a preset threshold, the second data matrix is ​​decomposed by SVD to obtain a third matrix, a second diagonal matrix, and a fourth matrix; based on the generalized least squares method, the third data matrix, the preset identity matrix, and the number of rows of the first data matrix are processed to obtain matrix U;

[0194] The regression coefficient matrix and outlier matrix are determined based on the number of rows in the matrix U, the second diagonal matrix, the fourth matrix, and the first data matrix.

[0195] The standard data corresponding to the abnormal data is determined based on the difference between the data of each non-preset data in the outlier matrix and the corresponding data in the first data matrix.

[0196] In one possible implementation, the method further includes:

[0197] If a prediction request carrying each data to be predicted received at each time point within the preset time length is received, then the data to be processed corresponding to the second preset feature in each data to be predicted is determined; a fourth data matrix is ​​constructed based on the obtained data to be processed; wherein, the construction process of the fourth data matrix is ​​the same as that of the first data matrix;

[0198] The matrix obtained by multiplying the fourth data matrix by the regression coefficient matrix is ​​determined as the target data matrix;

[0199] Based on the data contained in the target matrix, determine each target data in the data to be predicted that corresponds to the first preset feature.

[0200] In one possible implementation, the method further includes:

[0201] The regression coefficient matrix is ​​updated by multiplying the inverse of the first data matrix (after adjusting for outliers and filling in missing data) and the third data matrix.

[0202] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0203] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0204] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0205] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0206] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A data processing method, characterized in that, The method includes: Acquire each data received at each time point within a preset time length, and determine each data corresponding to a first preset feature, wherein the first preset feature is the transaction amount or the transaction time interval; A first data matrix is ​​constructed based on the determined data, and data at missing positions in the first data matrix is ​​supplemented. Data received at the same time point are located in the same row of the first data matrix, and data with the same feature are located in the same column. Repeat the following steps: Perform Singular Value Decomposition (SVD) on the first data matrix to obtain a first matrix, a first diagonal matrix, and a second matrix; adjust the elements at preset positions in the first diagonal matrix to preset values; determine the second data matrix based on the first matrix, the adjusted first diagonal matrix, and the second matrix; supplement the missing data in the second data matrix to the corresponding missing positions in the first data matrix to generate a target matrix; determine the first normal form modulus of the difference matrix between the second data matrix and the initial second data matrix, and the second normal form modulus of the second data matrix; if the value determined by the first normal form modulus and the second normal form modulus is less than a preset threshold, then the data at the missing positions in the target matrix is ​​determined as standard supplementary data; otherwise, update the target matrix to the first data matrix.

2. The method according to claim 1, characterized in that, The data used to supplement the missing positions in the first data matrix includes: Fill the missing positions in the first data matrix with preset data; or... For each missing location, determine the feature corresponding to that location, and then supplement the missing location in the first data matrix with the data corresponding to that feature; or, For each missing position, the average value of each data point that has the same feature as the data at that missing position is determined, and the average value is added to the missing position in the first data matrix.

3. The method according to claim 1, characterized in that, The step of adjusting the element at a preset position in the first diagonal matrix to a preset value includes: Sort each element of the first diagonal matrix according to the row where the element is located, obtain each element after the preset number of elements in the sorting result, and adjust each obtained element to a preset value.

4. The method according to claim 3, characterized in that, The preset quantity is determined by the following formula: Where, τ n = (logn) -1 loglogn, k is a positive integer less than x, x is the number of rows in the first diagonal matrix, n is the number of rows in the first data matrix, q is the number of columns in the first data matrix, d k Let d be the k-th element on the diagonal of the first diagonal matrix. k+1 It is the (k+1)th element on the diagonal of the first diagonal matrix.

5. The method according to claim 1, characterized in that, Determining the numerical value based on the first normal form module and the second normal form module includes: Determine the ratio of the first normal form module to the second normal form module, and determine the numerical value based on the ratio.

6. The method according to claim 1, characterized in that, The method further includes: Determine other data in each of the data that correspond to the second preset feature; wherein, the second preset feature is a feature associated with the first preset feature, and the second preset feature is any one of company area, business volume, and company logo; A third data matrix is ​​constructed based on the other data obtained, wherein the third data matrix is ​​constructed in the same way as the first data matrix; When the values ​​determined by the first normal form module and the second normal form module are less than a preset threshold, the second data matrix is ​​decomposed by SVD to obtain a third matrix, a second diagonal matrix, and a fourth matrix; based on the generalized least squares method, the third data matrix, the preset identity matrix, and the number of rows of the first data matrix are processed to obtain matrix U; The regression coefficient matrix and outlier matrix are determined based on the number of rows in the matrix U, the second diagonal matrix, the fourth matrix, and the first data matrix. The standard data corresponding to the abnormal data is determined based on the difference between the data of each non-preset data in the outlier matrix and the corresponding data in the first data matrix.

7. The method according to claim 6, characterized in that, The method further includes: If a prediction request carrying each data to be predicted received at each time point within the preset time length is received, then the data to be processed corresponding to the second preset feature in each data to be predicted is determined; a fourth data matrix is ​​constructed based on the obtained data to be processed; wherein, the construction process of the fourth data matrix is ​​the same as that of the first data matrix; The matrix obtained by multiplying the fourth data matrix by the regression coefficient matrix is ​​determined as the target data matrix; Based on the data contained in the target matrix, determine each target data corresponding to the first preset feature in the data to be predicted.

8. The method according to claim 6 or 7, characterized in that, The method further includes: The regression coefficient matrix is ​​updated by multiplying the inverse of the first data matrix (after adjusting for outliers and filling in missing data) and the third data matrix.

9. A data processing apparatus, characterized in that, The device includes: The acquisition and determination module is used to acquire each data received at each time point within a preset time length, and determine each data corresponding to a first preset feature, wherein the first preset feature is the transaction amount or the transaction time interval; The supplementation module is used to construct a first data matrix based on the determined data and supplement the missing data in the first data matrix, wherein data received at the same time point are located in the same row of the first data matrix and data with the same feature are located in the same column; The processing module is used to repeatedly execute the following steps: performing Singular Value Decomposition (SVD) on the first data matrix to obtain a first matrix, a first diagonal matrix, and a second matrix; adjusting the elements at preset positions in the first diagonal matrix to preset values; determining a second data matrix based on the first matrix, the adjusted first diagonal matrix, and the second matrix; supplementing the missing data in the second data matrix to the corresponding missing positions in the first data matrix to generate a target matrix; determining the first normal form modulus of the difference matrix between the second data matrix and the initial second data matrix and the second normal form modulus of the second data matrix; if the value determined by the first normal form modulus and the second normal form modulus is less than a preset threshold, then the data at the missing positions in the target matrix is ​​determined as standard supplementary data; otherwise, the target matrix is ​​updated to the first data matrix.

10. An electronic device, characterized in that, The electronic device includes at least a processor and a memory, wherein the processor is configured to execute the steps of any of the data processing methods described in claims 1-8 when executing a computer program stored in the memory.

Images