Data processing method and device, electronic equipment, storage medium and program product

Abstract

The embodiment of the invention provides a data processing method and device, electronic equipment, a storage medium and a program product, and relates to the technical field of computers.The method comprises the steps that a to-be-processed data set is obtained, and the to-be-processed data set is derived from observation of high-dimensional response variables; based on the to-be-processed data set, model parameters are determined, and the model parameters are used for determining a factor model; the model parameters comprise a load matrix and a unique factor variance; a common factor score and a unique factor score are determined from the model parameters based on the dataset to be processed. According to the embodiment of the invention, the common factor score and the unique factor score corresponding to the data set can be determined from the data set.

Description

Technical Field

[0001] The present application belongs to the field of computer technology. Specifically, the present application relates to a data processing method, device, electronic device, storage medium and program product. Background Art

[0002] Factor models are powerful tools for simplifying and analyzing high-dimensional data, and play an important role in behavioral analysis, management, economics and finance, healthcare, and related big data processing fields. In the factor analysis process, common factors and unique factors are two important concepts that can help researchers understand the inherent structure of the data and conduct more in-depth analysis. In factor analysis, common factors refer to common factors shared by multiple observed variables, which can explain the common variance between these variables. Unique factors refer to special factors that are unique to each observed variable and cannot be explained by other variables. By identifying common factors and unique factors, researchers can better understand the structure of the data and the relationship between variables. For example, after obtaining the grades of students in a class in various subjects, analyzing the common problems and individual problems of students from the grades of various subjects will help to conduct a more in-depth analysis of the students in this class;

[0003] Therefore, how to obtain common factor scores and unique factor scores from the data becomes a key issue. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide a data processing method, device, electronic device, storage medium and program product that can obtain common factor scores and unique factor scores from data.

[0005] In order to achieve the above objectives, the technical solutions provided in the embodiments of the present application are as follows:

[0006] In a first aspect, a data processing method is provided, comprising:

[0007] Obtaining a data set to be processed, where the data set to be processed is derived from observations of a high-dimensional response variable;

[0008] Determining model parameters based on the data set to be processed, wherein the model parameters are used to determine a factor model; the model parameters include a loading matrix and unique factor variances;

[0009] Based on the data set to be processed, common factor scores and unique factor scores are determined by the model parameters.

[0010] In a possible implementation, determining model parameters based on the data set to be processed includes:

[0011] If the data set to be processed presents a normal distribution, determining the loading matrix and the unique factor variance based on the data set to be processed by maximum likelihood estimation, and determining the model based on the loading matrix and the unique factor variance;

[0012] If the data set to be processed has other distributions, the loading matrix and unique factor variance are determined based on the data set to be processed by generalized least squares estimation, and the model is determined based on the loading matrix and the unique factor variance.

[0013] In a possible implementation, determining the common factor scores and the unique factor scores by the model parameters based on the data set to be processed includes:

[0014] Constructing a loss function using the dataset to be processed, the loading matrix, and the unique factor variance;

[0015] Based on the loss function, common factor scores and unique factor scores are estimated.

[0016] In a possible implementation, determining the loading matrix and unique factor variance based on the data set to be processed by maximum likelihood estimation includes:

[0017] Based on the data set to be processed and using a first formula, a representation of a first covariance structure matrix is ​​determined; wherein the first formula is:

[0018]

[0019] in, represents the representation of the first covariance structure matrix, S is the sample covariance matrix generated by the data set to be processed; Represents a diagonal matrix whose diagonal elements are represented by vectors Elemental composition, region is a p-dimensional real vector space; p is the number of model response variables, m is the number of model common factors, i=1,2,…,p,is a matrix The eigenvalues ​​arranged from large to small are i=1,2,…,p,is its corresponding eigenvector, where I is the identity matrix;

[0020] Based on the representation of the first covariance structure matrix and using the second formula, a first objective function of maximum likelihood estimation is determined, wherein the second formula is:

[0021]

[0022] Where tr represents the trace of the matrix, and the first objective function of the maximum likelihood estimation is:

[0023] Based on the first objective function and using a third formula, the gradient vector of the first objective function is calculated; wherein the third formula is:

[0024] represents the gradient vector of the first objective function, diag(A) represents the vector consisting of the elements on the main diagonal of matrix A,

[0025] Based on the first objective function and using a fourth formula, the Hessian matrix of the first objective function is calculated; wherein the fourth formula is:

[0026]

[0027] in, The Hessian matrix characterizing the first objective function, “°” represents the Hadamard product of the matrix;

[0028] determining the unique factor variance of the model by a Newton iteration method based on the first objective function, the gradient vector of the first objective function, and the Hessian matrix of the first objective function;

[0029] Based on the unique factor variances, the loading matrix was determined.

[0030] In a possible implementation, the distribution form of the data to be processed is general, and the method of determining the loading matrix and the unique factor variance based on the data set to be processed by generalized least squares estimation includes:

[0031] Based on the data set to be processed and using the fifth formula, the representation of the second covariance structure matrix is ​​determined; wherein the fifth formula is:

[0032]

[0033] in, represents the representation of the second covariance structure matrix; the weight V is a given positive definite matrix or a random matrix that converges to a positive definite matrix according to probability, area i=1,2,…,p, is a matrix The eigenvalues ​​arranged from large to small are i=1,2,…,p,its corresponding eigenvector;

[0034] Based on the representation of the second covariance structure matrix and using the sixth formula, the second objective function of the generalized least squares estimation is determined; wherein the sixth formula is:

[0035]

[0036] Among them, the second objective function of the generalized least squares estimation is:

[0037]

[0038] Based on the second objective function, the gradient vector of the second objective function is determined by the seventh formula; wherein: the seventh formula is:

[0039]

[0040] in, represents the gradient vector of the second objective function,

[0041] Based on the second objective function and using the eighth formula, the Hessian matrix of the second objective function is calculated; wherein the eighth formula is:

[0042]

[0043] in

[0044] Obtaining a unique factor variance of the model by a Newton iteration method based on the second objective function, the gradient vector of the second objective function, and the Hessian matrix of the second objective function;

[0045] Based on the unique factor variances, determine the loading matrix of the model.

[0046] In a possible implementation, determining the common factor scores and the unique factor scores by the model parameters based on the data set to be processed includes:

[0047] Centralizing the data set to be processed to obtain a centralized data set;

[0048] Based on the minimization of the loss function, the common factor scores and the unique factor scores are determined.

[0049] In another possible implementation, based on the adjustment matrix, the common factor scores and the unique factor scores are adjusted when necessary to assist in finding appropriate common factors and unique factors.

[0050] In a possible implementation, determining the common factor scores and the unique factor scores by the model parameters based on the data set to be processed includes:

[0051] Centralizing the data set to be processed to obtain a centralized data set;

[0052] Based on the centralized data set, a loss function is constructed, where the loss function is:

[0053]

[0054] Where X is the centered data set to be processed, Λ is the loading matrix, Ψ is the diagonal matrix composed of unique factor variances, N is the sample size, F and EΨ are the undetermined common factor scores and unique factor scores respectively;

[0055] Based on the data set to be processed after the centralization process, make a matrix The singular value decomposition of obtains the p+m order orthogonal matrix and N-1 order orthogonal matrix

[0056]

[0057] in, is a diagonal matrix composed of positive singular values, the matrix Q is the following N×(N-1) order auxiliary matrix:

[0058]

[0059] Based on the orthogonal matrix and Determine the common factor score F and the unique factor score EΨ, where F and E are obtained by the following formula:

[0060]

[0061] in, for the reason The first p columns of Depend on The remaining m columns of for the reason The first p columns of Depend on The remaining Np-1 columns of K are composed of any (Np-1)×m-order standard orthogonal matrix, that is, K′K=I.

[0062] In a second aspect, a data processing device is provided, the device comprising:

[0063] An acquisition module, configured to acquire a data set to be processed, where the data set to be processed is derived from observations of a high-dimensional response variable;

[0064] A first determination module is configured to determine model parameters based on the data set to be processed, wherein the model parameters are used to determine a factor model; the model parameters include a loading matrix and a unique factor variance;

[0065] The second determination module is configured to determine the common factor scores and the unique factor scores by using the model parameters based on the data set to be processed.

[0066] In a possible implementation, the distribution form of the data to be processed is general, and when the first determination module determines the loading matrix and the unique factor variance based on the data set to be processed by generalized least squares estimation, it is specifically configured to:

[0067] Based on the data set to be processed and using the fifth formula, the representation of the second covariance structure matrix is ​​determined; wherein the fifth formula is:

[0068]

[0069] in, represents the representation of the second covariance structure matrix; the weight V is a given positive definite matrix or a random matrix that converges to a positive definite matrix according to probability, area i=1,2,…,p, is a matrix The eigenvalues ​​arranged from large to small are i=1,2,…,p,its corresponding eigenvector;

[0070] Based on the representation of the second covariance structure matrix and using the sixth formula, the second objective function of the generalized least squares estimation is determined; wherein the sixth formula is:

[0071]

[0072] Among them, the second objective function of the generalized least squares estimation is:

[0073]

[0074] Based on the second objective function, the gradient vector of the second objective function is determined by the seventh formula; wherein: the seventh formula is:

[0075]

[0076] in, represents the gradient vector of the second objective function,

[0077] Based on the second objective function and using the eighth formula, the Hessian matrix of the second objective function is calculated; wherein the eighth formula is:

[0078]

[0079] in

[0080] Obtaining a unique factor variance of the model by a Newton iteration method based on the second objective function, the gradient vector of the second objective function, and the Hessian matrix of the second objective function;

[0081] Based on the unique factor variances, determine the loading matrix of the model.

[0082] In a possible implementation, the second determination module determines the common factor scores and the unique factor scores based on the data set to be processed and the model parameters, specifically for:

[0083] Centralizing the data set to be processed to obtain a centralized data set;

[0084] Based on the centralized data set, a loss function is constructed, where the loss function is:

[0085]

[0086] Where X is the centered data set to be processed, Λ is the loading matrix, Ψ is the diagonal matrix composed of unique factor variances, N is the sample size, F and EΨ are the undetermined common factor scores and unique factor scores respectively;

[0087] Based on the data set to be processed after the centralization process, make a matrix The singular value decomposition of obtains the p+m order orthogonal matrix and N-1 order orthogonal matrix

[0088]

[0089] in, is a diagonal matrix composed of positive singular values, the matrix Q is the following N×(N-1) order auxiliary matrix:

[0090]

[0091] Based on the orthogonal matrix and Determine the common factor score F and the unique factor score EΨ, where F and E are obtained by the following formula:

[0092]

[0093] in, for the reason The first p columns of Depend on The remaining m columns of for the reason The first p columns of Depend on The remaining Np-1 columns of K are composed of any (Np-1)×m-order standard orthogonal matrix, that is, K′K=I.

[0094] In a third aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the data processing method provided by any possible implementation of the first aspect.

[0095] In a fourth aspect, an embodiment of the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the data processing method provided by any possible implementation of the first aspect.

[0096] In a fifth aspect, an embodiment of the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the data processing method provided by any possible implementation of the first aspect.

[0097] The beneficial effects of the technical solution provided by the embodiments of the present application are as follows:

[0098] Embodiments of the present application provide a method, apparatus, electronic device, storage medium, and program product for data processing. In embodiments of the present application, after obtaining a data set to be processed that originates from observations of a high-dimensional response variable, model parameters are determined based on the data set to be processed. The model parameters include a loading matrix and a unique factor variance. Based on the data set to be processed, common factor scores and unique factor scores are determined by the model parameters, thereby determining common factor scores and unique factor scores from the data set to be processed. BRIEF DESCRIPTION OF THE DRAWINGS

[0099] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for describing the embodiments of the present application.

[0100] Figure 1 This is a flow chart of a data processing method in an embodiment of the present application;

[0101] Figure 2Schematic diagram of a process for determining a loading matrix and unique factor variance by maximum likelihood estimation in an embodiment of the present application;

[0102] Figure 3 This is a flow chart of determining the loading matrix and unique factor variance by generalized least squares estimation in an embodiment of the present application;

[0103] Figure 4 This is a flow chart of determining common factor scores and unique factor scores from model parameters in an embodiment of the present application;

[0104] Figure 5 In the embodiment of this application Figure 2 A flowchart of the Newton iteration method for determining the unique factor variance of the model by maximum likelihood estimation;

[0105] Figure 6 In one embodiment of the present application, Figure 5 Schematic diagram of the process of selecting the initial vector in the iterative process;

[0106] Figure 7 In the embodiment of this application Figure 3 A flowchart of the Newton iteration method for determining the unique factor variance of the model by generalized least squares estimation;

[0107] Figure 8 In one embodiment of the present application, Figure 7 Schematic diagram of the process of selecting the initial vector in the iterative process;

[0108] Figure 9 A schematic diagram of a process for obtaining a minimum value of a loss function in an embodiment of the present application;

[0109] Figure 10 This is a schematic diagram of the score distribution of the first factor and the second factor in the embodiment of the present application;

[0110] Figure 11 Schematic diagram of the change between the number of common factors and the second objective function and loss function in the embodiment of the present application;

[0111] Figure 12 This is a schematic diagram of the structure of a data processing device in an embodiment of the present application;

[0112] Figure 13 This is a structural diagram of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0113] The following describes the embodiments of the present application in conjunction with the accompanying drawings. It should be understood that the embodiments described below in conjunction with the accompanying drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions of the embodiments of the present application.

[0114] Those skilled in the art will understand that, unless otherwise stated, the singular forms "a", "an", "said", and "the" used herein may also include plural forms. It should be further understood that the terms "including" and "comprising" used in the embodiments of the present application mean that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements, and / or components, but do not exclude implementation as other features, information, data, steps, operations, elements, components, and / or combinations thereof supported by the present technical field. It should be understood that when we say that an element is "connected" or "coupled" to another element, the element can be directly connected or coupled to the other element, or it can refer to the element and the other element establishing a connection relationship through an intermediate element. In addition, the "connection" or "coupling" used here can include wireless connection or wireless coupling. The term "and / or" used here indicates at least one of the items defined by the term, for example, "A and / or B" can be implemented as "A", or as "B", or as "A and B". When describing multiple (two or more) items, if the relationship between the multiple items is not clearly defined, the multiple items may refer to one, multiple or all of the multiple items. For example, the description of "parameter A includes A1, A2, A3" can be implemented as parameter A including A1 or A2 or A3, and can also be implemented as parameter A including at least two of the three items A1, A2, and A3.

[0115] For p-dimensional response variables (y1, y2, ..., y p )′, and obtain a dataset X=(x ti ), where x ti is the i-th variable y i The observation value at t (i=1,2,…,p,t=1,2,…,N). The factor model decomposes the data set X into the following:

[0116] X-μ X ≈FΛ′+EΨ 1 / 2 (1)

[0117] in is the mean matrix of X, F is an N×m matrix, E is an N×p matrix, Λ is a p×m matrix, and Ψ is a p-order non-negative diagonal matrix. F and E satisfy F′E=0 and F and EΨ are called common factors (scores) and unique factors (scores), respectively. Λ is called the loading matrix of the model, and the elements on the diagonal of Ψ are denoted as is called the unique factor variance. m is a positive integer much smaller than p, called the number of common factors. Here is an N-dimensional vector, I m and I N are the identity matrices of order m and N.

[0118] To build a factor model, we first need to estimate the parameter matrices Λ and Ψ. Then, we need to estimate the factor scores F and E based on the established model.

[0119] There are two widely used methods for estimating model parameters. The first is maximum likelihood estimation under normal distribution conditions, and the second is generalized least squares estimation under general distribution conditions. Regarding maximum likelihood estimation, the related art methods have two major flaws. First, it does not guarantee the convergence of the iterations; second, the unique factor variance must be greater than a certain positive number for the iterations to proceed. Currently, the factor analysis modules in almost all data processing software set the default value of this positive number to 0.01 or 0.05. In fact, to date, the correct algorithms for these two estimation methods have not been found.

[0120] These methods express the parameter matrix with covariance structure as Σ=ΛΛ′+Ψ, that is, Σ is a p×(m+1)-element matrix function with variables Λ and Ψ. The complexity of the domain of definition of this function may be the reason why the currently used parameter estimation methods cannot guarantee iterative convergence. This application uses a new method to express the covariance structure matrix. Under such a representation, Σ is a p-element matrix function, so that the objective function is a p-element function, which makes it easy to prove the convergence of the algorithm.

[0121] For the estimation of factor scores, the methods currently used only estimate the common factor scores. These methods regard unique factors as errors or completely ignore unique factors, and use regression methods and least squares methods, such as Thompson regression method and Bartlett weighted least squares method. This approach actually goes against the original intention of the factor model, because unique factors are another important factor affecting the response variable. A new method for estimating common factor scores and unique factor scores is provided in the embodiments of the present application. In the embodiments of the present application, not only a reasonable method is provided for estimating common factor scores, but also the analysis of unique factors is expanded from the previous analysis that only cares about common factors.

[0122] Furthermore, the present invention provides a method for data processing. Figure 1 As shown, in an embodiment of the present application, the data processing method may be executed by an electronic device, and the method may include:

[0123] Step S101: Obtain a data set to be processed.

[0124] Among them, the data set to be processed comes from the observation of high-dimensional response variables, and the collection methods include scientific experiments, social surveys, etc.

[0125] For the embodiment of the present application, the data set to be processed may include simulated scores of various subjects of multiple students. For example, the data set to be processed may include: simulated scores of various subjects of 60 students (N=60); such as, it may include: simulated data of mathematics (x1), physics (x2), chemistry (x3), Chinese (x4), history (x5) and English (x6). The data set to be processed is denoted as X, as shown in Table 1.

[0126] Table 1

[0127]

[0128]

[0129] Step S102: Determine model parameters based on the data set to be processed.

[0130] Among them, model parameters are used to determine the factor model; model parameters include the loading matrix and unique factor variance.

[0131] Step S103: Based on the data set to be processed, determine the common factor scores and unique factor scores using the model parameters.

[0132] For the embodiment of the present application, the common factor score may include at least one common factor score, and the special factor score may also include at least one special factor score. In the embodiment of the present application, the analysis result for the dataset to be processed is determined based on the common factor score and the special factor score.

[0133] Specifically, if the data set to be processed is the score of each subject in each subject; the analysis result of the data set to be processed includes at least one of the following: the relationship between each common factor and each subject; the preference of each subject in the data set to be processed for each subject.

[0134] Specifically, a method of obtaining the dataset to be processed may include: obtaining the dataset to be processed input by the target object, or obtaining the dataset to be processed from other devices, or obtaining the dataset to be processed from local storage.

[0135] Specifically, in an embodiment of the present application, in step S102, model parameters are determined based on the data set to be processed, which may specifically include: if the data set to be processed presents a normal distribution, the loading matrix and unique factor variance are determined based on the data set to be processed by maximum likelihood estimation, and the model parameters are determined based on the loading matrix and unique factor variance; if the distribution of the data set to be processed is unknown, the loading matrix and unique factor variance are determined based on the data set to be processed by generalized least squares estimation, and the model parameters are determined based on the loading matrix and unique factor variance.

[0136] Specifically, in the embodiment of the present application, it is known that the data to be processed is normally distributed, and the data is analyzed according to the maximum likelihood estimation method: the number m of common factors, the loading matrix and the variance of the unique factors are determined.

[0137] Specifically, based on the above embodiment, the load matrix and unique factor variance are determined based on the data set to be processed by maximum likelihood estimation, which may include: step Sa1, step Sb1, step Sc1, step Sd1, step Se1 and step Sf1, as shown in FIG. Figure 2 As shown,

[0138] Step Sa1: Based on the data set to be processed and using the first formula, determine the representation of the first covariance structure matrix.

[0139] Among them, the first formula is:

[0140] in, Represents the representation of the first covariance structure matrix, S is the sample covariance matrix generated by the data set to be processed, that is, S is reversible; Represents a diagonal matrix whose diagonal elements are represented by vectors Elemental composition, region is a p-dimensional real vector space; p is the number of model response variables, m is the number of common factors in the model, i=1,2,…,p,is a matrix The eigenvalues ​​arranged from large to small are i=1,2,…,p,is its corresponding eigenvector, where I is the identity matrix.

[0141] Step Sb1: Based on the representation of the first covariance structure matrix and using the second formula, determine the first objective function of maximum likelihood estimation:

[0142] Among them, the second formula is:

[0143] Where tr represents the trace of the matrix, and the first objective function of the maximum likelihood estimation is:

[0144] For the embodiments of this application, exist If is differentiable arbitrarily, the gradient vector of the first objective function is calculated as follows.

[0145] Step Sc1, based on the first objective function and using the third formula, calculating the gradient vector of the first objective function;

[0146] Among them, the third formula is: represents the gradient vector of the first objective function, diag(A) represents the vector consisting of the elements on the main diagonal of matrix A, in, here

[0147] Step Sd1: Based on the first objective function and using the fourth formula, calculate the Hessian matrix of the first objective function.

[0148] Among them, the fourth formula is:

[0149]

[0150] in, The Hessian matrix representing the first objective function, Represents the Hadamard product of matrices.

[0151] Step Se1: Based on the first objective function, the gradient vector of the first objective function and the Hessian matrix of the first objective function, the unique factor variance of the model is determined by the Newton iteration method.

[0152] For the embodiments of this application, according to The gradient vector and the Hessian matrix Use the steepest descent algorithm, such as Newton's iterative method, to find In the area The minimum point in The maximum likelihood estimate of the model covariance structure matrix is

[0153]

[0154] Among them, the unique factor variance estimate of the model maximum likelihood estimate is: Specifically, the Newton iteration method described in step Se1 can be based on Figure 5 The process is implemented, and the selection of the initial vector can be based on Figure 6 Process obtained.

[0155] Among them, such as Figure 5 As shown, the Newton iteration process can be as follows:

[0156] Se11. Input: sample covariance matrix S generated by the data to be processed, number of common factors m;

[0157] Se12, select the initial vector Set the stopping error ε=10 -6 , let k = 0;

[0158] Se13, calculate the first objective function based on the second formula and the third formula The function value and gradient vector of g are equal to the first objective function in The magnitude of the gradient vector of ;

[0159] Se14, if g < ε, output unique factor variance Stop iteration;

[0160] Se15, calculate the first objective function based on the fourth formula The Hessian matrix of →H;

[0161] Se16. Calculate the minimum eigenvalue of H → λ. If λ≤0: H + (g-λ)I → H, let v = -H -1 ×The first objective function is The gradient vector of

[0162] Se17, select n to make

[0163] Se18, select n so that the first objective function is The value is less than The value of v / 2 n →v;

[0164] S19, k+1→k, go to Se13.

[0165] Among them, such as Figure 6 As shown, Se12 may specifically include:

[0166] Se121, based on the sample covariance S generated by the data to be processed, let

[0167] Se122, make a matrix spectral decomposition of ;

[0168] Se123, if Stop iteration and output the initial vector

[0169] Se124, Transfer to Se122.

[0170] Step Sf1: Determine the loading matrix based on the unique factor variance.

[0171] Among them, the loading matrix of the maximum likelihood estimation of the model is estimated as:

[0172] Similarly, based on the above embodiment, the loading matrix and the unique factor variance can be determined based on the data set to be processed by generalized least squares estimation, which can specifically include: step Sa2, step Sb2, step Sc2, step Sd2, step Se2 and step Sf2, as shown in FIG. Figure 3 As shown,

[0173] Step Sa2: Based on the data set to be processed and using the fifth formula, determine the representation of the second covariance structure matrix.

[0174] Among them, the fifth formula is:

[0175] in, represents the representation of the second covariance structure matrix; the weight V is a given positive definite matrix or a random matrix that converges to a positive definite matrix according to probability, area i=1,2,…,p, is a matrix The eigenvalues ​​arranged from large to small are i=1,2,…,p,its corresponding eigenvector;

[0176] Specifically, Make a matrix Spectral decomposition of , obtaining eigenvalues ​​arranged from large to small i=1,2,…,p, and their corresponding standard orthogonal eigenvectors.

[0177] Step Sb2: Based on the representation of the second covariance structure matrix and using the sixth formula, determine the second objective function of the generalized least squares estimation.

[0178] Among them, the sixth formula is: Among them, the second objective function of the generalized least squares estimation is: in, exist Arbitrarily differentiable.

[0179] Step Sc2: Based on the second objective function and using the seventh formula, determine the gradient vector of the second objective function.

[0180] Among them: The seventh formula is: in, represents the gradient vector of the second objective function and is The gradient vector of

[0181] Step Sd2: Based on the second objective function and using the eighth formula, calculate the Hessian matrix of the second objective function.

[0182] Among them, the eighth formula is:

[0183] in

[0184] Step Se2: Based on the second objective function, the gradient vector of the second objective function and the Hessian matrix of the second objective function, the unique factor variance of the model is obtained by Newton iteration method.

[0185] For the embodiments of this application, according to The gradient vector and the Hessian matrix For example, the Newton iteration method uses the steepest descent algorithm to obtain the function exist The minimum point on

[0186] Among them, the generalized least squares estimate of the model parameter matrix is

[0187]

[0188] Among them, the unique factor variance estimate of the generalized least squares estimator of the model is:

[0189] Specifically, the Newton iteration method described in step Se2 can be based on Figure 7 The process is implemented, in which the initial vector selection of step Se22 can be achieved by Figure 8 process implementation.

[0190] Among them, such as Figure 7 As shown, the Newton iteration process is as follows Figure 7 Specifically, it may include:

[0191] Se21, Input: Sample covariance matrix S generated by the data to be processed, number of common factors m;

[0192] Se22, select the initial vector Set the stopping error ε=10 -6 , let k = 0;

[0193] Se23, calculate the second objective function based on the sixth formula and the seventh formula The function value and gradient vector of g are equal to the second objective function in The magnitude of the gradient vector of ;

[0194] Se24, if g < ε, output unique factor variance Stop iteration;

[0195] Se25, calculate the second objective function based on the eighth formula The Hessian matrix of →H;

[0196] Se26. Calculate the minimum eigenvalue of H → λ. If λ≤0: H + (g-λ) I → H, let v = -H -1 ×The second objective function is The gradient vector of

[0197] Se27, select n to make

[0198] Se28, select n so that the second objective function is The value is less than The value of v / 2 n →v;

[0199] Se29, k+1→k, enter Se23.

[0200] Specifically, Se22 may include Se221-Se224, such as Figure 8 As shown:

[0201] Se22 may specifically include:

[0202] Se221, based on the sample covariance S generated by the data to be processed, let

[0203] Se222, make a matrix spectral decomposition of ;

[0204] Se223, if Stop iteration and output the initial vector

[0205] Se224, Transfer to Se222.

[0206] f2. Determine the model loading matrix based on the unique factor variance.

[0207] Specifically, the loading matrix of the generalized least squares estimation of the model is estimated as:

[0208] Specifically, in step S103, based on the dataset to be processed, the common factor scores and unique factor scores are determined from the model parameters, which may specifically include: constructing a loss function through the dataset to be processed, the loading matrix, and the unique factor variances; and estimating the common factor scores and unique factor scores based on the loss function by minimizing the loss function.

[0209] Specifically, based on the dataset to be processed, the common factor scores and unique factor scores are determined from the model parameters, which may specifically include: step S1, step S2, and step S3, as Figure 4 shown, where

[0210] Step S1: Centralize the dataset to be processed to obtain the centralized dataset.

[0211] For the embodiments of this application, the dataset to be processed is centralized in the following manner to obtain the centralized dataset:

[0212]

[0213] where N is the sample size, is an N-dimensional vector. If the sample size N < p + m + 1, the sample size is increased using the bootstrap method or other methods to make N ≥ p + m + 1. For example, as shown in Table 1, N = 60, p = 6, m = 2, which already meets the requirements.

[0214] Step S2: Based on the centralized dataset, the loading matrix of the model, and the unique factor variances, construct a loss function.

[0215] The loss function is:

[0216]

[0217] where X is the centralized dataset to be processed, Λ is the loading matrix, Ψ is the diagonal matrix composed of unique factor variances, tr represents the trace of the matrix, and F and E are the to-be-determined common factor scores and unique factor scores respectively.

[0218] Step S3: Based on the loss function, obtain to minimize the loss function, where is the common factor score, is the unique factor score.

[0219] Step S3 is implemented through the Figure 9 process. Specifically as follows.

[0220] Step S31: Input the dataset to be processed X, the loading matrix unique factor variances The diagonal matrix

[0221] Step S32: Check whether the sample size N satisfies N≥number of variables (p)+number of common factors (m)+1. If not, use the bootstrap sampling method to increase the sample size.

[0222] Step S33: construct an auxiliary matrix Q to centralize the data set X to be processed.

[0223] The auxiliary matrix is:

[0224]

[0225] Step S34: Based on the centralized data set X, make a matrix The singular value decomposition of Q obtains the p+m order orthogonal matrix and N-1 order orthogonal matrix So that:

[0226]

[0227] in, is a diagonal matrix of positive singular values, i=1,…,p, are singular values, and the matrix

[0228] Step S35: Orthogonal matrix and Divide into blocks and in p column, m columns, p column, Np-1 columns.

[0229] Step S36 : Create a factor adjustment matrix K: a (Np-1)×m-order orthogonal matrix (i.e., satisfying K′K=I).

[0230] Step S37: Calculate the loss function value

[0231] Step S38: Output common factor scores and unique factor scores and the loss function value.

[0232] For the embodiments of the present application, the number of common factors m=1, 2, and 3 are respectively taken. Based on the above-mentioned maximum likelihood estimation process Sa1-Sf1 and factor scoring process S1-S3, the calculated index values ​​are listed in Table 2.

[0233] According to the p-value and loss function value in Table 2, the number of common factors of the model should be determined to be 2.

[0234] When m=2, the load matrix is ​​estimated and unique factor variance (vector) estimates Listed in Table 3.

[0235] The two common factor scores F1 and F2 are listed in Table 4, and the unique factor scores are omitted. At this point, the implementation process of this application is completed.

[0236] Further data analysis depends on the research objectives. If you are only interested in the impact of common factors, you only need to study the two-dimensional data in Table 1. If you are studying the characteristics and influence of individual factors, you can analyze the unique factor scores.

[0237] The following example examines student preferences for arts and science and their performance evaluation. It should be understood that the data is fictitious, and the analysis results should be considered fictitious. The analysis methods used are specific to this case and do not limit the scope of this application.

[0238] Table 2. Maximum likelihood estimation index values ​​for different number of factors

[0239]

[0240] Table 3. Factor loadings and unique factor variances (m = 2)

[0241]

[0242] Table 4. Common factor scores

[0243]

[0244]

[0245] Based on the rotational invariance of the common factors in the model and the research objectives, the data in Table 4 can be rotated to highlight the characteristics of the data. The specific method is as follows.

[0246] Based on the common factor scores and loading matrix, the common factors that affect the response variables (scores in each subject) are found and explained by rotating the coordinate axes.

[0247] After the maximum variance orthogonal rotation, the first common factor is used as the vertical axis and the second common factor is used as the horizontal axis to rotate 225 degrees forward. The rotation matrix is:

[0248]

[0249] Common factor scores after rotation and Listed in Table 5, the factor loadings after rotation Listed in Table 6.

[0250] Table 5. Common factor scores after rotation

[0251]

[0252]

[0253] Table 6. Rotated factor loadings (m=2)

[0254]

[0255] Furthermore, the correlation coefficients between the rotated common factors and the response variables were calculated, and the results are listed in Table 7.

[0256] Table 7. Correlation coefficients between the rotated common factors and each variable

[0257]

[0258] From Table 7, we know that the first factor It is negatively correlated with science subjects (mathematics x1, physics x2, chemistry x3) and positively correlated with liberal arts subjects (Chinese x4, history x5, English x6). Its magnitude reflects the students' preference for liberal arts and science: negative direction is biased towards science, positive direction is biased towards liberal arts, and there is no bias near 0, which can be called the liberal arts index; the second factor It is positively correlated with each subject, and its size reflects the overall level of students' scores in the six courses. It can be called performance index or grade point. The common factor (first factor and second factor) score and The distribution of Figure 10 .

[0259] The evaluation of students' learning situation should be based on the liberal arts index Divided into three parts: biased towards theory, non-biased and biased towards literature, based on the performance index For example, among students who are biased towards science, No. 12 should be ranked first, among students who are not biased towards any subject, No. 54 should be ranked first, and among students who are biased towards liberal arts, No. 36 should be ranked first; No. 32 is heavily biased towards science, and No. 13 is heavily biased towards liberal arts.

[0260] From Table 3, the unique factor variance of physics course is the largest. This indicates that the students' academic performance in this course is most affected by individual factors. Among them, the factor with the largest contribution to the variance is No. 28, with a unique factor score of 2.6505, indicating that the physics performance of this course will fluctuate greatly. On the contrary, the absolute value of the unique factor score of No. 54 is the smallest, at 0.0205, indicating that its physics performance is relatively stable.

[0261] The student liberal arts index and performance index can be measured using the following methods.

[0262] The first common factor ( Arts and Sciences Index) and the second common factor ( The performance index (f1) is regressed with the response variable to obtain a regression equation, which is then standardized: the liberal arts index f1 is adjusted to the range of -10 to +10. That is, if f1 = -10, it is completely biased towards science, f1 = +10, it is completely biased towards liberal arts, and f1 ≈ 0 indicates that there is basically no bias; the performance index f2 is adjusted between 0 and 10. If the score in each subject is 0, f2 = 0, and if the score in each subject is full, f2 = 10. The measurement formula is as follows:

[0263] f1=(-6.66x1-1.06x2-1.33x3+8.49x4+1.72x5+0.74x6-95.00) / 100;

[0264] f2 = (3.23x1 + 0.34x2 + 0.55x3 + 4.83x4 + 0.84x5 + 0.21x6) / 100. For example, a student's grades are: Math 91.0, Physics 84.0, Chemistry 92.5, Chinese 78.5, History 84.0, English 85.5. Their liberal arts index f1 = -0.39, and their performance index f1 = 8.41. Evaluation: This student has a slight bias towards science, but overall good grades.

[0265] The data processing method shown in the embodiment of the present application is described below using a specific example. The data in Table 8 are derived from anthropometric and physical health measurements of 50 male applicants. A total of 15 variables were used (Gunst and Mason, 1980), represented by (xk), where k = 1, 2, ..., 15.

[0266] These variables include:

[0267] 1. Reaction time to visual stimulus in seconds (x1);

[0268] 2. Applicant's height in centimeters (x2);

[0269] 3. Applicant's weight in kilograms (x3);

[0270] 4. Applicant's shoulder width in centimeters (x4);

[0271] 5. Applicant's pelvic width in centimeters (x4);

[0272] 6. Applicant's minimum chest circumference in centimeters (x6);

[0273] 7. Applicant's thigh skinfold thickness in millimeters (x7);

[0274] 8. Applicant's resting pulse rate (x8);

[0275] 9. Applicant's diastolic blood pressure (x9);

[0276] 10. Number of pull-ups the applicant can complete (x10);

[0277] 11. Applicant's maximum breathing capacity (in liters) (x11);

[0278] 12. Applicant's pulse rate after 5 minutes of recovery from treadmill running (x12);

[0279] 13. Applicant's maximum treadmill speed (x13);

[0280] 14. Applicant's treadmill endurance time (in minutes) (x14);

[0281] 15. Applicant's total body fat measurement (x15);

[0282] Table 8

[0283]

[0284]

[0285] We now use factor models to simplify data and use relatively few common factors to evaluate applicants' physical and health conditions.

[0286] Due to the different scales of the data, in order to reduce calculation errors, the data is preprocessed, that is, the scale is adjusted. A common processing method is to convert the data of each variable into the "standard score" of the variable. The processing method is as follows:

[0287] (xk) i The value of the kth variable of the applicant with serial number i, i = 1, 2, ..., 50, k = 1, 2, ..., 15, is expressed as Replace (xk) i ,in and s k represent the mean and standard deviation of the kth variable respectively.

[0288] The dataset X below refers to the dataset after adjusting the original dataset.

[0289] Because the distribution of the dataset X is unknown, the model parameters are estimated using generalized least squares. To emphasize reaction speed and upper limb strength, the weight matrix V is a diagonal matrix: the first and tenth elements on the diagonal are 0.1, and the remaining elements are 1.

[0290] Based on the data set to be processed X, the number of common factors m = 1, 2, ..., 8, from steps Sa2 to Sf2, the model loading matrix and unique factor variance (see Figure 3 ), where the iterative process of step Se2 is completed by steps Se21 to Se29 (see Figure 7 ), where the initial vector is completed by steps Se221 to Se224 (see Figure 8 The common factor scores, unique factor scores and loss function values ​​are completed by steps S31 to S38 (see Figure 9 ). The relevant calculation indicators are shown in Table 9. According to the indicators in Table 9 and Figure 11 , the appropriate number of common factors is 3 or 4. To be conservative, take the number of common factors m = 4. Figure 11 When the number of common factors makes the function values ​​of the two functions very small and the amplitude of further decrease is very small, the number of common factors can be selected as the common factors of the model. It can also be determined by referring to the indicators AIC and BIC.

[0291] Table 9. Maximum likelihood estimation index values ​​for different numbers of common factors

[0292]

[0293] Based on the number of common factors (m = 4) and the aforementioned steps, we obtained the model's loading matrix, unique factor variances, and common factor scores. Similar to the previous example, after performing a varimax rotation, the loading matrix and common factor scores are listed in Tables 10 and 11, respectively. Thus, the 15-dimensional dataset X, under the factor model determined in Table 10, has been compressed into the 4-dimensional data shown in Table 11.

[0294] Table 10. Factor loadings and unique factor variances (m = 4)

[0295]

[0296] Table 11

[0297]

[0298]

[0299] In this case, the meaning of the common factors can be further analyzed and the applicant's physical condition can be evaluated accordingly.

[0300] The correlation coefficients between the four common factors and the 15 response variables were calculated, and the results are listed in Table 12.

[0301] Based on the loading matrix in Table 10 and the correlation coefficients in Table 12, the first common factor, F1, can be used as a measure of reaction speed; the second common factor, F2, can be used as a measure of upper limb strength and obesity; the third common factor, F3, can be used as a measure of skeletal structure or physique; and the fourth common factor, F4, can be used as an auxiliary measure of cardiovascular health. For example, based on these four indicators, applicants No. 20 and No. 30 ranked at the top of the comprehensive index.

[0302] Table 12. Correlation coefficients between variables and common factors

[0303] F1 F2 F3 F4 x1 0.9478 0.3152 0.0172 -0.0063 x2 0.2094 0.0094 0.7850 -0.1536 x3 0.2020 -0.4826 0.7824 -0.0924 x4 -0.0738 -0.1266 0.7668 -0.0697 x5 -0.0275 -0.0994 0.7234 -0.1279 x6 0.0817 -0.1855 0.3421 0.0343 x7 0.3632 -0.6816 0.2390 0.1623 x8 0.1131 0.1987 -0.1683 0.6675 x9 0.1269 0.0673 -0.0762 0.3100 x10 -0.4025 0.7183 -0.2040 0.0154 x11 0.2021 -0.1322 0.5138 0.0314 x12 -0.0765 -0.1479 -0.0630 0.7721 x13 -0.2548 0.2884 0.1789 -0.5004 x14 -0.1094 0.1995 -0.3403 -0.0779 x15 0.3953 -0.6619 0.4566 0.0820

[0304] Furthermore, a regression model can be used to establish the relationship between these four common factors and various physical examination indicators, which can be used as a measurement formula for physical and health conditions, but is omitted here.

[0305] Because this model has zero unique factor variance, existing data processing software cannot be directly applied. One approach is to force the unique factor variance to be greater than a positive number (usually 0.05) to create an approximate 5-factor model. Another approach is to remove three variables and create a 4-factor model. However, these approaches are too crude and imprecise.

[0306] Based on the above embodiments, the present application provides a data processing device, such as Figure 12 As shown, the data processing device 100 may include: an acquisition module 1001, a first determination module 1002 and a second determination module 1003, wherein:

[0307] An acquisition module 1001 is used to acquire a data set to be processed, where the data set to be processed is derived from observations of a high-dimensional response variable;

[0308] The first determination module 1002 is used to determine model parameters based on the data set to be processed, where the model parameters are used to determine the model parameters of the factor model; the model parameters include the loading matrix and the unique factor variance;

[0309] The second determining module 1003 is configured to determine the common factor scores and the unique factor scores based on the data set to be processed and the model parameters.

[0310] In another possible implementation of the embodiment of the present application, when determining the model parameters based on the data set to be processed, the first determination module 1002 is specifically used to: if the data set to be processed presents a normal distribution, determine the loading matrix and the unique factor variance based on the data set to be processed by maximum likelihood estimation, and determine the model based on the loading matrix and the unique factor variance; determine the loading matrix and the unique factor variance based on the data set to be processed by generalized least squares estimation, and determine the model based on the loading matrix and the unique factor variance.

[0311] In another possible implementation of the embodiment of the present application, when the second determination module 1003 determines the common factor scores and unique factor scores by the model parameters based on the data set to be processed, it is specifically used to: construct a loss function through the data set to be processed, the loading matrix and the unique factor variance; and estimate the common factor scores and unique factor scores based on the loss function.

[0312] In another possible implementation of the embodiment of the present application, when the first determining module 1002 determines the loading matrix and the unique factor variance based on the data set to be processed by maximum likelihood estimation, it is specifically configured to:

[0313] Based on the data set to be processed and using the first formula, the representation of the first covariance structure matrix is ​​determined; wherein the first formula is:

[0314]

[0315] in, represents the representation of the first covariance structure matrix, S is the sample covariance matrix generated by the data set to be processed; Represents a diagonal matrix whose diagonal elements are represented by vectors Elemental composition, region is a p-dimensional real vector space; p is the number of model response variables, m is the number of common factors in the model, i=1,2,…,p,is a matrix The eigenvalues ​​arranged from large to small are i=1,2,…,p,is its corresponding eigenvector, where I is the identity matrix;

[0316] Based on the representation of the first covariance structure matrix and through the second formula, the first objective function of the maximum likelihood estimation is determined, wherein the second formula is:

[0317]

[0318] Where tr represents the trace of the matrix, and the first objective function of the maximum likelihood estimation is:

[0319] Based on the first objective function and using the third formula, the gradient vector of the first objective function is calculated; wherein the third formula is:

[0320] represents the gradient vector of the first objective function, diag(A) represents the vector consisting of the elements on the main diagonal of matrix A,

[0321] Based on the first objective function and using the fourth formula, the Hessian matrix of the first objective function is calculated; wherein the fourth formula is:

[0322]

[0323] in, The Hessian matrix representing the first objective function, represents the Hadamard product of matrices;

[0324] determining the unique factor variance of the model by a Newton iteration method based on the first objective function, the gradient vector of the first objective function, and the Hessian matrix of the first objective function;

[0325] Based on the unique factor variances, the loading matrix was determined.

[0326] In another possible implementation of the embodiment of the present application, when the first determining module 1002 determines the loading matrix and the unique factor variance based on the data set to be processed by generalized least squares estimation, it is specifically configured to:

[0327] Based on the data set to be processed and using the fifth formula, the representation of the second covariance structure matrix is ​​determined; wherein the fifth formula is:

[0328]

[0329] in, represents the representation of the second covariance structure matrix; the weight V is a given positive definite matrix or a random matrix that converges to a positive definite matrix according to probability, area i=1,2,…,p, is a matrix The eigenvalues ​​arranged from large to small are i=1,2,…,p,its corresponding eigenvector;

[0330] Based on the representation of the second covariance structure matrix and through the sixth formula, the second objective function of the generalized least squares estimation is determined; wherein the sixth formula is:

[0331]

[0332] Among them, the second objective function of the generalized least squares estimation is:

[0333]

[0334] Based on the second objective function, the gradient vector of the second objective function is determined by the seventh formula; wherein: the seventh formula is:

[0335]

[0336] in, represents the gradient vector of the second objective function,

[0337] Based on the second objective function and using the eighth formula, the Hessian matrix of the second objective function is calculated; wherein the eighth formula is:

[0338]

[0339] in

[0340] Based on the second objective function, the gradient vector of the second objective function and the Hessian matrix of the second objective function, the unique factor variance of the model is obtained by the Newton iteration method;

[0341] Based on the unique factor variances, determine the loading matrix of the model.

[0342] In another possible implementation of the embodiment of the present application, when the second determining module 1003 determines the common factor score and the unique factor score based on the model parameters based on the dataset to be processed, it is specifically configured to:

[0343] Centralize the data set to be processed to obtain a centralized data set;

[0344] Based on the centralized data set, a loss function is constructed, where the loss function is:

[0345]

[0346] Where X is the centered data set to be processed, Λ is the loading matrix, Ψ is the diagonal matrix composed of unique factor variances, N is the sample size, F and EΨ are the undetermined common factor scores and unique factor scores respectively;

[0347] Based on the centralized data set to be processed, make a matrix The singular value decomposition of obtains the p+m order orthogonal matrix and N-1 order orthogonal matrix

[0348]

[0349] in, is a diagonal matrix composed of positive singular values, the matrix Q is the following N×(N-1) order auxiliary matrix:

[0350]

[0351] Based on orthogonal matrix and Determine the common factor score F and the unique factor score EΨ, where F and E are obtained by the following formula:

[0352]

[0353] in, for the reason The first p columns of Depend on The remaining m columns of for the reason The first p columns of Depend on The remaining Np-1 columns of K are composed of any (Np-1)×m-order standard orthogonal matrix, that is, K′K=I.

[0354] Figure 13 A schematic diagram of the structure of an electronic device applicable to the embodiment of the present application is shown in FIG. Figure 13 As shown, for example, the electronic device may be a server or a user terminal, and the electronic device may be used to implement the method provided in any embodiment of the present application.

[0355] like Figure 13 As shown in FIG, the electronic device 2000 may mainly include at least one processor 2001 ( Figure 13 1 ), memory 2002, communication module 2003 and input / output interface 2004 and other components, optionally, the components can be connected and communicated through bus 2005. It should be noted that, Figure 13 The structure of the electronic device 2000 shown in the figure is merely illustrative and does not constitute a limitation on the electronic devices to which the method provided in the embodiments of the present application is applicable.

[0356] The memory 2002 can be used to store an operating system and application programs, etc. The application program may include a computer program that implements the method described in the embodiment of the present application when called by the processor 2001, and may also include programs for implementing other functions or services. The memory 2002 can be a ROM (Read Only Memory) or other type of static storage device that can store static information and instructions, a RAM (Random Access Memory) or other type of dynamic storage device that can store information and computer programs, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0357] The processor 2001 is connected to the memory 2002 via the bus 2005 and implements corresponding functions by calling the application program stored in the memory 2002. Among them, the processor 2001 can be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof, which can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor 2001 can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0358] The electronic device 2000 can be connected to a network via a communication module 2003 (which may include, but is not limited to, components such as a network interface) to communicate with other devices (such as a user terminal or a server) via the network to implement data interaction, such as sending data to or receiving data from other devices. The communication module 2003 may include a wired network interface and / or a wireless network interface, etc., that is, the communication module may include at least one of a wired communication module and a wireless communication module.

[0359] The electronic device 2000 can be connected to the required input / output devices, such as a keyboard, a display device, etc., through the input / output interface 2004. The electronic device 2000 itself can have a display device, and can also be connected to other external display devices through the interface 2004. Optionally, a storage device, such as a hard disk, can also be connected through the interface 2004, so that data in the electronic device 2000 can be stored in the storage device, or data in the storage device can be read, and data in the storage device can also be stored in the memory 2002. It can be understood that the input / output interface 2004 can be a wired interface or a wireless interface. Depending on the actual application scenario, the device connected to the input / output interface 2004 can be a component of the electronic device 2000, or it can be an external device connected to the electronic device 2000 when needed.

[0360] Bus 2005, used to connect the various components, may include a path for transmitting information between the components. Bus 2005 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. Depending on their function, bus 2005 may be categorized as an address bus, a data bus, a control bus, or the like.

[0361] Optionally, for the solution provided in the embodiment of the present application, the memory 2002 can be used to store a computer program for executing the solution of the present application, and be run by the processor 2001. When the processor 2001 runs the computer program, the actions of the method or device provided in the embodiment of the present application are implemented.

[0362] Based on the same principle as the method provided in the embodiment of the present application, the embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the corresponding content of the aforementioned method embodiment can be implemented.

[0363] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the corresponding content of the aforementioned method embodiment can be implemented.

[0364] It should be noted that the terms "first," "second," "third," "fourth," "1," "2," etc. (if any) in the specification and claims of this application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than that shown or described in the drawings.

[0365] It should be understood that, although each operation step is indicated by arrows in the flowchart of the embodiment of the present application, the order of implementation of these steps is not limited to the order indicated by the arrows. Unless otherwise clearly stated herein, in some implementation scenarios of the embodiment of the present application, the implementation steps in each flowchart can be performed in other orders according to demand. In addition, some or all of the steps in each flowchart can include multiple sub-steps or multiple stages based on actual implementation scenarios. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage in these sub-steps or stages can also be executed at different times respectively. Under different scenarios at the execution time, the execution order of these sub-steps or stages can be flexibly configured according to demand, and the embodiment of the present application does not limit this.

[0366] The above description is only an optional implementation method for some implementation scenarios of this application. It should be pointed out that for ordinary technicians in this technical field, without departing from the technical concept of the solution of this application, the use of other similar implementation methods based on the technical ideas of this application also falls within the protection scope of the embodiments of this application.

Claims

1. A data processing method, characterized in that: The method comprises: Obtaining a data set to be processed, where the data set to be processed is derived from observations of a high-dimensional response variable; Determining model parameters based on the data set to be processed, wherein the model parameters are used to determine a factor model; the model parameters include a loading matrix and unique factor variances; Based on the data set to be processed, common factor scores and unique factor scores are determined by the model parameters.

2. The method according to claim 1, characterized in that The determining of model parameters based on the data set to be processed includes: If the data set to be processed presents a normal distribution, determining the loading matrix and the unique factor variance based on the data set to be processed by maximum likelihood estimation, and determining the model parameters based on the loading matrix and the unique factor variance; The loading matrix and the unique factor variance are determined based on the data set to be processed by generalized least squares estimation, and the model parameters are determined based on the loading matrix and the unique factor variance.

3. The method according to claim 1 or 2, characterized in that The determining of the common factor scores and the unique factor scores by the model parameters based on the data set to be processed includes: Constructing a loss function using the dataset to be processed, the loading matrix, and the unique factor variance; Based on the loss function, common factor scores and unique factor scores are estimated.

4. The method according to claim 2, characterized in that The step of determining the loading matrix and the unique factor variance based on the data set to be processed by maximum likelihood estimation includes: Based on the data set to be processed and using a first formula, a representation of a first covariance structure matrix is ​​determined; wherein the first formula is: in, represents the representation of the first covariance structure matrix, S is the sample covariance matrix generated by the data set to be processed; Represents a diagonal matrix whose diagonal elements are represented by vectors Elemental composition, region is a p-dimensional real vector space; p is the number of model response variables, m is the number of model common factors, is a matrix The eigenvalues ​​arranged from large to small are Its corresponding eigenvector, where I is the identity matrix; Based on the representation of the first covariance structure matrix and using the second formula, a first objective function of maximum likelihood estimation is determined, wherein the second formula is: Where tr represents the trace of the matrix, and the first objective function of the maximum likelihood estimation is: Based on the first objective function and using a third formula, the gradient vector of the first objective function is calculated; wherein the third formula is: represents the gradient vector of the first objective function, diag(A) represents the vector consisting of the elements on the main diagonal of matrix A, Based on the first objective function and using a fourth formula, the Hessian matrix of the first objective function is calculated; wherein the fourth formula is: in, The Hessian matrix characterizing the first objective function, "°" represents the Hadamard product of the matrix; determining the unique factor variance of the model by a Newton iteration method based on the first objective function, the gradient vector of the first objective function, and the Hessian matrix of the first objective function; Based on the unique factor variances, the loading matrix was determined.

5. The method according to claim 2, characterized in that The method of estimating the loading matrix and the unique factor variance based on the data set to be processed by generalized least squares includes: Based on the data set to be processed and using the fifth formula, the representation of the second covariance structure matrix is ​​determined; wherein the fifth formula is: in, represents the representation of the second covariance structure matrix; the weight V is a given positive definite matrix or a random matrix that converges to a positive definite matrix according to probability, area is a matrix The eigenvalues ​​arranged from large to small are is its corresponding eigenvector; Based on the representation of the second covariance structure matrix and using the sixth formula, the second objective function of the generalized least squares estimation is determined; wherein the sixth formula is: Among them, the second objective function of the generalized least squares estimation is: Based on the second objective function, the gradient vector of the second objective function is determined by the seventh formula; wherein: the seventh formula is: in, represents the gradient vector of the second objective function, Based on the second objective function and using the eighth formula, the Hessian matrix of the second objective function is calculated; wherein the eighth formula is: in Obtaining a unique factor variance of the model by a Newton iteration method based on the second objective function, the gradient vector of the second objective function, and the Hessian matrix of the second objective function; Based on the unique factor variances, determine the loading matrix of the model.

6. The method according to claim 3, characterized in that The determining of the common factor scores and the unique factor scores by the model parameters based on the data set to be processed includes: Centralizing the data set to be processed to obtain a centralized data set; Based on the centralized data set, a loss function is constructed, where the loss function is: Where X is the centered data set to be processed, Λ is the loading matrix, Ψ is the diagonal matrix composed of unique factor variances, N is the sample size, F and EΨ are the undetermined common factor scores and unique factor scores respectively; Based on the data set to be processed after the centralization process, make a matrix The singular value decomposition of obtains the p+m order orthogonal matrix and N-1 order orthogonal matrix in, is a diagonal matrix composed of positive singular values, the matrix Q is the following N×(N-1) order auxiliary matrix: Based on the orthogonal matrix and Determine the common factor score F and the unique factor score EΨ, where F and E are obtained by the following formula: in, for the reason The first p columns of Depend on The remaining m columns of for the reason The first p columns of Depend on The remaining Np-1 columns of K are composed of any (Np-1)×m-order standard orthogonal matrix, and K′K=I.

7. A data processing device, characterized in that: The device comprises: An acquisition module, configured to acquire a data set to be processed, where the data set to be processed is derived from observations of a high-dimensional response variable; A first determination module is configured to determine model parameters based on the data set to be processed, wherein the model parameters are used to determine model parameters of a factor model; the model parameters include a loading matrix and a unique factor variance; The second determination module is configured to determine the common factor scores and the unique factor scores by using the model parameters based on the data set to be processed.

8. An electronic device, characterized in that: The electronic device includes a memory and a processor, wherein a computer program is stored in the memory, and the processor executes the data processing method according to any one of claims 1 to 6 when running the computer program.

9. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and when the computer program is executed by a processor, the data processing method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the data processing method according to any one of claims 1 to 6 are implemented.

Images