Sparse matrix modeling method and device, computer device and medium

Abstract

Embodiments of the present application are suitable for the technical field of machine learning, and provide a sparse matrix modeling method and device, computer equipment and medium, the method comprising: dimension reduction on a sparse matrix used for modeling, to obtain a low-rank matrix after dimension reduction; splicing the low-rank matrix and the sparse matrix to obtain a splicing matrix, the splicing matrix having a plurality of features; redundancy analysis on a plurality of features to obtain redundant features; deleting the redundant features in the splicing matrix to obtain a feature matrix used for modeling training; and using the feature matrix and a target algorithm for modeling training to obtain a target model. Through the above method, the sparse matrix modeling method can have migration.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of machine learning, and particularly relates to a sparse matrix modeling method and device, a computer device, and a medium. BACKGROUND

[0002] Modeling is an important means and premise for studying a system. Modeling by using sample data can obtain a target model for solving a corresponding problem.

[0003] There are more zero data in each dimension of the data of the sparse matrix. When modeling by using the sparse matrix, the sparse matrix generally needs to be processed before modeling. Existing processing methods for the sparse matrix can include compressing sparse rows or sparse columns according to a business, but this method needs to determine the compressed sparse rows or sparse columns according to the business. Dimension reduction processing can also be performed by using a dimension reduction method such as Principal Component Analysis (PCA) or Linear Discriminant Analysis (LDA). However, these dimension reduction methods either do not contain category information in the dimension reduction content or need to specify category information each time the dimension reduction is performed, and the data processing is relatively complex.

[0004] It can be seen that, when modeling by using the sparse matrix, specific processing needs to be performed according to a specific business, the modeling process is complex and highly targeted, and the migration is poor. SUMMARY

[0005] Therefore, an embodiment of the present application provides a sparse matrix modeling method, device, computer device, and medium, to make the sparse matrix modeling method have migration.

[0006] A first aspect of an embodiment of the present application provides a sparse matrix modeling method, comprising:

[0007] Dimension reduction is performed on a sparse matrix used for modeling, to obtain a low-rank matrix after dimension reduction;

[0008] The low-rank matrix and the sparse matrix are spliced to obtain a spliced matrix, the spliced matrix has a plurality of features;

[0009] Redundancy analysis is performed on the plurality of features to obtain redundant features;

[0010] The redundant features in the spliced matrix are deleted to obtain a feature matrix used for modeling training;

[0011] Modeling training is performed by using the feature matrix and a target algorithm to obtain a target model.

[0012] A second aspect of an embodiment of the present application provides a sparse matrix modeling device, comprising:

[0013] a dimension reduction module, configured to perform dimension reduction on the sparse matrix for modeling to obtain a low-rank matrix after dimension reduction;

[0014] a splicing module, configured to splice the low-rank matrix and the sparse matrix to obtain a spliced matrix, the spliced matrix having a plurality of features;

[0015] a calculation module, configured to delete redundant features in the spliced matrix to obtain the redundant features;

[0016] a deletion module, configured to delete target features with a redundancy higher than a preset redundancy threshold in the spliced matrix to obtain a feature matrix for modeling training;

[0017] a modeling module, configured to perform modeling training by using the feature matrix and a target algorithm to obtain a target model.

[0018] A third aspect of the embodiment of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method in the first aspect above when executing the computer program.

[0019] A fourth aspect of the embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program is executable by a processor to implement the method in the first aspect above.

[0020] A fifth aspect of the embodiment of the present application provides a computer program product, which, when executed on a computer device, causes the computer device to perform the method in the first aspect above.

[0021] Compared with the prior art, the embodiment of the present application has the following advantages:

[0022] In the embodiment of the present application, when using a sparse matrix for modeling, dimension reduction operation is performed on the sparse matrix to obtain a low-rank matrix; the low-rank matrix and the sparse matrix are spliced to obtain a spliced matrix; the features in the spliced matrix are analyzed for redundancy to determine the redundant features; the redundant features in the spliced matrix are deleted to obtain a feature matrix for modeling; and the feature matrix and a target algorithm are used for modeling to obtain a target model. In the embodiment, when using a sparse matrix for modeling, a method combining dimension reduction and redundancy analysis is used, so that the processed sparse matrix can retain the information in the original data and improve the representation ability, thereby retaining the representation ability of the matrix for business data after processing the sparse matrix, so that the method can be directly applied to a machine learning model as an embedded module, and the migration of the sparse matrix modeling method is enhanced. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart illustrating the steps of a sparse matrix modeling method provided in an embodiment of this application;

[0025] Figure 2 This is a flowchart illustrating an improved LDA algorithm provided in an embodiment of this application;

[0026] Figure 3 This is a flowchart illustrating the steps of another sparse matrix modeling method provided in this application embodiment;

[0027] Figure 4 This is a flowchart illustrating the steps of another sparse matrix modeling method provided in this application embodiment;

[0028] Figure 5 This is a schematic diagram of a sparse matrix modeling device provided in an embodiment of this application;

[0029] Figure 6 This is a schematic diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0030] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0031] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0032] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0033] As used in this specification and claims, the term “if’ can be construed to mean “when” or “once,” or “in response to a determination” or “in response to a detection” that a described condition or event occurs. Similarly, the phrase “if it is determined” or “if [a described condition or event] is detected” can be construed to mean “once it is determined” or “in response to a determination” or “once [a described condition or event] is detected” or “in response to the detection” of [a described condition or event].

[0034] In addition, in the description of the present application and the appended claims, the terms “first”, “second”, “third”, etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0035] Reference in the specification to “one embodiment” or “some embodiments” means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. The appearances of the phrases “in one embodiment”, “in some embodiments”, “in other embodiments”, “in additional embodiments”, and so on, in various places in the specification are not necessarily all referring to the same embodiment, unless otherwise specifically stated. The terms “comprising”, “including”, “having” and their variants mean “including but not limited to”, unless otherwise specifically stated.

[0036] The technical solutions of the present application are described below through specific embodiments.

[0037] Referring to Figure 1 , a step flow diagram of a sparse matrix modeling method provided by an embodiment of the present application is shown, which can specifically include the following steps:

[0038] S101, dimension reduction is performed on the sparse matrix for modeling to obtain a low-rank matrix after dimension reduction.

[0039] The execution subject of the present embodiment is a computer device. The method in the present embodiment can be applied to all systems or terminals that need to deploy machine learning models online. For example, it can be applied to face recognition systems, speech recognition systems, vehicle recognition systems, and text processing systems.

[0040] Since there are multiple zero values in the sparse matrix, the information density is low, and direct model training with the sparse matrix may not be efficient, therefore, dimension reduction can be performed on the sparse matrix. In the present embodiment, an improved LDA algorithm can be used to perform dimension reduction operation on the sparse matrix. The traditional LDA algorithm has the following disadvantages: when the original data has K categories, it can be reduced to at most K-1 dimensions, which is not flexible enough. The improved LDA algorithm can be reduced to any specified dimension.

[0041] Figure 2 is a flowchart of an improved LDA algorithm provided by an embodiment of the present application. As shown in the figure, when dimensionality reduction is performed using the improved LDA algorithm, first, data preparation is performed. It is assumed that the input sparse matrix is X, and the data contains i categories, which are y1, y2,..., yi respectively. Then, data preprocessing is performed. The data preprocessing mainly processes outliers and missing values in the sample data. The data preprocessing can also include re-encoding of enumerated Chinese features in the sample data, such as recording gender male as 1 and gender female as 2. Figure 2

[0042] After the data preprocessing, the sample data in the sparse matrix is divided according to the categories, and the sparse matrix is divided into a first sample set,..., and an i-th sample set. For each sample set, a PCA model can be constructed under a specified dimensionality reduction target K. i PCA models can be established corresponding to the i sample sets. After the i PCA models are constructed, the feature matrices after dimensionality reduction of the i PCA models are determined, which are X1, X2,..., Xi respectively.

[0043] The within-class scatter S w of the i dimensionally reduced feature matrices is calculated, and the between-class scatter S b between the i feature matrices is calculated.

[0044] The calculation formula of the within-class scatter is:

[0045]

[0046] where K is the Kth category, D k is the Kth sample set, x is a sample in the Kth sample set, N k is the number of samples in the Kth sample set, and m k is the mean vector of the samples after dimensionality reduction of the Kth sample set.

[0047] The calculation formula of the between-class scatter is:

[0048]

[0049] where i and j represent different categories, and m i and m j represent the mean vectors of the samples of different categories.

[0050] The current value of K, the value of the within-class scatter, and the value of the between-class scatter are output. Then, by adjusting K, the maximum between-class scatter and the minimum within-class scatter are obtained, and the K at this time is the optimal K. The feature matrices corresponding to the optimal K are the low-rank matrices finally output by the LDA algorithm. ​

[0051] The improved LDA algorithm in the improved LDA algorithm is equivalent to combining the PCA algorithm and the LDA algorithm, can be reduced to any dimension, and can be classified and reduced.

[0052] Figure 2 The improved PDA algorithm in the improved PDA algorithm can be used as an embedded module in the sparse matrix modeling process. The input of the embedded module is a sparse matrix, and the output is a low-rank matrix and an optimal K. The hyperparameter K represents the best dimension reduction dimension of the sparse matrix, and the dimension of the low-rank matrix is K.

[0053] The improved LDA algorithm in the improved LDA algorithm can be used as an embedded module in the sparse matrix modeling process. The input of the embedded module is a sparse matrix, and the output is a low-rank matrix and an optimal K. The hyperparameter K represents the best dimension reduction dimension of the sparse matrix, and the dimension of the low-rank matrix is K. Figure 2 The improved LDA algorithm in the improved LDA algorithm can be used as an embedded module in the sparse matrix modeling process. The input of the embedded module is a sparse matrix, and the output is a low-rank matrix and an optimal K. The hyperparameter K represents the best dimension reduction dimension of the sparse matrix, and the dimension of the low-rank matrix is K.

[0054] Each row of the sparse matrix can represent a sample data, and each column can represent a feature data. The low-rank matrix obtained by dimension reduction of the sparse matrix is consistent with the number of rows of the sparse matrix, so they can be spliced.

[0055] In the spliced matrix, the data of the sparse matrix is the original data, and the representation ability of the original data can be preserved; the data in the low-rank matrix is the data after dimension reduction, and the information density is greater, and it has stronger feature representation ability. Therefore, the spliced matrix can cover more data information on the one hand, and can include more feature information on the other hand. Using the data in the spliced matrix for subsequent training can improve the accuracy of the model obtained by training.

[0056] S103, redundancy analysis is performed on a plurality of the features to obtain redundant features.

[0057] The low-rank matrix is a dense matrix obtained by dimension reduction of the sparse matrix, so there may be the same features in the low-rank matrix and the sparse matrix. That is, the spliced matrix may have two identical features, but in the model training, each feature in the sample data should be irrelevant. Therefore, there are redundant features in the spliced matrix.

[0058] In determining the redundant features in the spliced matrix, two features with strong correlation can be determined first. Specifically, the feature correlation between each two features in the spliced matrix can be calculated; if the feature correlation between two features is greater than a preset redundancy threshold, it means that there is a redundant feature in the two features. At this time, only one of the two features needs to be retained. When calculating the feature correlation, the Pearson coefficient method can be used to calculate the correlation between each two features.

[0059]

[0060] ​For two features with strong feature correlation, a feature with small information gain can be selected as a redundant feature. Specifically, when the feature correlation is greater than a preset redundancy threshold, two target features corresponding to the feature correlation are determined; information gain values of the two target features are respectively determined; and a target feature with a small information gain value is selected as a redundant feature.

[0061] In this embodiment, the feature with a larger information gain value is selected as a redundant feature from the two features with strong correlation, so that more information can be included in the remaining features.

[0062] S104, the redundant features in the splicing matrix are deleted to obtain a feature matrix for modeling training.

[0063] Specifically, one feature in the splicing matrix can correspond to one column in the splicing matrix, the column corresponding to the redundant feature is determined, and then the data in the column is deleted.

[0064] Deleting the data corresponding to the redundant features in the splicing matrix can avoid semantic conflicts between the sparse matrix and the low-rank matrix in the splicing matrix, so that the feature matrix after deleting the redundant data can have stronger representation ability, which is beneficial to modeling accuracy.

[0065] S105, a target model is obtained by using the feature matrix and a target algorithm for modeling training.

[0066] The target algorithm can be an algorithm trained using sample data of the sparse matrix. For example, the sparse matrix can be image data, and the target algorithm can be an image recognition algorithm. The data in the sparse matrix can have different characteristics according to different application scenarios. For example, when the method is applied to image processing, the data in the sparse matrix can be image feature data, and the corresponding target model is an image processing model; when the method is applied to text processing, the data in the sparse matrix can be text feature data, and the corresponding target model is a text processing model.

[0067] The sparse matrix is the original data, and after training the target algorithm using the sparse matrix, the accuracy and recall rate before embedding can be obtained. After training the target algorithm using the method in this embodiment, the accuracy and recall rate after embedding can be obtained.

[0068] If the accuracy and recall rate after embedding are both improved compared to the accuracy and recall rate before embedding, it can be determined that the evaluation passes, and it can be put online.

[0069] If the target model does not pass the evaluation, the target model needs to be retrained. When retraining, the steps of S102-S105 can be retrained.

[0070] In the embodiment of the present application, when the sparse matrix is processed, the corresponding service information does not need to be determined, only the data processing is needed. The sparse matrix and the low-rank matrix are spliced to preserve the information in the data, so that the data has corresponding representation ability for the service, thereby specific processing for the service is not needed, and the migration of the sparse modeling method is improved.

[0071] Referring to Figure 3 , another step flow diagram of the sparse matrix dimension reduction method provided by the embodiment of the present application is shown, which can specifically include the following steps:

[0072] S301, based on a plurality of preset dimensions, dimension reduction operation is respectively performed on a plurality of sample sets corresponding to the sparse matrix to obtain a sub-matrix set under each dimension, and the sub-matrix set includes a plurality of one-to-one corresponding sub-matrices of the sample set.

[0073] The execution subject of the embodiment is a computer device, and the method in the embodiment can be applied to a system or terminal that needs to perform machine learning model training.

[0074] The plurality of dimensions are dimensions lower than the rank of the sparse matrix. For example, the rank of the sparse matrix is 10, and the plurality of dimensions can include 1-9. Each sample set corresponds to a category of data, and each sample data set includes a plurality of sample data.

[0075] The PCA method can be used for dimension reduction operation on each sample set. For a specified dimension reduction dimension, each sample set can correspond to a sub-matrix after dimension reduction; and the plurality of sub-matrices corresponding to the plurality of sample sets form the above-mentioned sub-matrix set.

[0076] For the plurality of preset dimensions, a plurality of sub-matrix sets can be corresponded. For example, there are 9 dimensions of 1-9, then 9 times of dimension reduction operation can be performed on each sample data set, and the data dimension of the sample data set is reduced to one dimension of 1-9; under each dimension, one sample set can correspond to one sub-matrix. In this way, the 9 dimensions of 1-9 can correspond to one sub-matrix set respectively.

[0077] S302, the intra-class scatter of the sub-matrix set under each dimension is calculated.

[0078] Specifically, the intra-class scatter of the plurality of sub-matrices under each dimension can be calculated by using the following formula:

[0079]

[0080] Wherein, K is the Kth category, D k is the Kth sub-matrix, x is the sample data of the Kth sub-matrix, and Nk is the number of sample data of the Kth sub-matrix, m k is the mean vector of sample data in the Kth sub-matrix, S w is the intra-class dispersion.

[0081] The intra-class dispersion can be used to measure the dispersion within the class. When the intra-class dispersion is low, it indicates that the corresponding data within the class is more concentrated and can be classified into one class.

[0082] S303, calculate the inter-class dispersion of the sub-matrix set in each dimension.

[0083] Specifically, the inter-class dispersion of the plurality of sub-matrices in each dimension can be calculated by the following formula:

[0084]

[0085] where i is the ith class, j is the jth class, mi is the mean vector of sample data of the sub-matrix corresponding to the ith class, mi is the mean vector of sample data of the sub-matrix corresponding to the ith class, S b is the inter-class dispersion.

[0086] The inter-class dispersion can be used to measure the correlation between classes. The greater the inter-class dispersion, the more distant the classes are, and the more accurate the classification result is.

[0087] S304, determine the low-rank matrix based on the intra-class dispersion and the inter-class dispersion.

[0088] Specifically, a plurality of dimensions can correspond to a plurality of sub-matrix sets, and the intra-class dispersion and the inter-class dispersion of each sub-matrix set can be calculated. A target sub-matrix set with the maximum inter-class dispersion and the minimum intra-class dispersion is determined from the plurality of sub-matrix sets. Then, the dimension corresponding to the target sub-matrix set is taken as the target dimension of the low-rank matrix. Then, the dimension corresponding to the target sub-matrix set is taken as the target dimension of the low-rank matrix. Then, the dimension corresponding to the target sub-matrix set is taken as the target dimension of the low-rank matrix.

[0089] In one possible implementation, when there is no sub-matrix set with the maximum inter-class dispersion and the minimum intra-class dispersion, the sub-matrix set with the maximum inter-class dispersion or the sub-matrix set with the minimum intra-class dispersion can be determined as the target sub-matrix set.

[0090] In one possible implementation, the sample data in each sub-matrix in the target sub-matrix set can be arranged according to the arrangement manner in the original sparse matrix to obtain the low-rank matrix.

[0091] S305, splice the low-rank matrix and the sparse matrix to obtain a spliced matrix, and the spliced matrix has a plurality of features.

[0092] S306, redundancy analysis is performed on the plurality of features to obtain redundant features.

[0093] Specifically, for the plurality of features in the spliced matrix, feature correlation between each two features can be calculated respectively; when the feature correlation is greater than a preset redundancy threshold, two target features corresponding to the feature correlation are determined; information gain values of the two target features are determined respectively; and a target feature with a smaller information gain value is determined as a redundant feature.

[0094] In a possible implementation, information entropy of the two target features can be calculated respectively, and a target feature with smaller information entropy is determined as a redundant feature.

[0095] In a possible implementation, when the two target features include one belonging to a sparse matrix and one belonging to a low-rank matrix, the target feature belonging to the sparse matrix is determined as a redundant feature.

[0096] S307, the redundant features in the spliced matrix are deleted to obtain a feature matrix for modeling training.

[0097] S308, the feature matrix and a target algorithm are used for modeling training to obtain a target model.

[0098] Specifically, the method in the embodiment of the present application can be embedded as a module into the target algorithm, and in the process of training the target algorithm using the sparse matrix, the method in the embodiment also participates in the training. For the method in the embodiment, the hyperparameters can include an optimal dimension and a redundancy threshold. Through continuous training of the target algorithm, if the inter-class dispersion of the low-rank matrix after dimension reduction is maximum, the intra-class dispersion is minimum, and the prediction accuracy of the target algorithm is improved, it is indicated that the module corresponding to the method in the embodiment is trained, and the optimal dimension and the redundancy threshold are obtained. The trained module can be grafted into different algorithms for use, and has strong migration.

[0099] The method in the embodiment combines the LDA algorithm and the PCA algorithm for dimension reduction operation, which can classify and reduce the dimension of data on one hand, and can reduce the dimension of data to any dimension on the other hand. Through such dimension reduction, the obtained low-rank matrix is no longer limited in dimension, thereby enhancing the feature expression of the low-rank matrix, and making the target model obtained by modeling the sparse matrix more accurate.

[0100] It should be noted that the sequence numbers of the steps in the above embodiments do not mean the execution sequence, and the execution sequence of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0101] The following uses an example of a target algorithm to illustrate the implementation of the method in the embodiment of the present application. Figure 4The sparse matrix modeling method shown further illustrates the method in the application. Figure 4 The sparse matrix modeling method shown is an embedded sparse matrix modeling method based on an improved LDA algorithm and redundancy analysis. As shown in the formula (1), the method can include: Figure 4

[0102] First, data preparation is performed. It is assumed that a plurality of sample data for training form a sparse matrix X, and the sample data can include i categories, y1, y2,..., yi, respectively.

[0103] Then, data preprocessing is performed on the sparse matrix. For example, outlier correction, missing value filling, etc. can be performed. For recoding of enumerated Chinese features, the enumerated features are converted into digital features.

[0104] The sparse matrix after data preprocessing can be subjected to dimension reduction operation by the improved LDA algorithm, and output a low-rank matrix L after dimension reduction and a hyperparameter K of the improved LDA algorithm.

[0105] The low-rank matrix L after dimension reduction and the original sparse matrix S are spliced to obtain a spliced matrix. Then, redundancy analysis is performed on the spliced matrix, and features with redundancy exceeding a redundancy threshold r are deleted. The redundancy threshold r is a newly added second hyperparameter, and participates in training together with the hyperparameter K of the improved LDA algorithm.

[0106] After training, the embedded modeling of the sparse matrix is completed.

[0107] Then, effect evaluation can be performed. According to the effect accuracy and recall rate of the target algorithm before embedding the improved LDA algorithm and redundancy analysis, and the effect accuracy and recall rate after embedding the improved LDA algorithm and redundancy analysis, it is determined whether the effect accuracy and recall rate after embedding the improved LDA algorithm and redundancy analysis reach a predetermined improvement value. If the predetermined improvement value is reached, it can be put online. If the predetermined improvement value is not reached, the values of the hyperparameters K and r are changed, and training is continued until the inter-class dispersion of the low-rank matrix is maximum, the intra-class dispersion is minimum, the effect accuracy is improved, and the recall rate is improved, which indicates that the training is completed. The sparse matrix modeling method after training can be put online in the form of a module and embedded into a machine learning algorithm for use.

[0108] ​In this embodiment, the spliced matrix of the low-rank matrix L and the sparse matrix S can be used to perform representation enhancement based on the low-rank matrix while preserving the representation ability of the original matrix to the greatest extent. Redundancy analysis on the spliced matrix can maximize the avoidance of semantic conflicts between the L and S matrices and avoid affecting the accuracy of the machine learning model when the L and S matrices are applied to the machine learning model. The method in this application can directly be embedded into a machine learning model without the need to set the number of categories in advance, and has wide application scenarios and strong migration.

[0109] Referring to Figure 5 , a schematic diagram of a sparse matrix modeling device provided by an embodiment of the application is shown, which can specifically include a dimension reduction module 51, a splicing module 52, an analysis module 53, a deletion module 54, and a modeling module 55, wherein:

[0110] The dimension reduction module 51 is configured to perform dimension reduction on a sparse matrix used for modeling to obtain a low-rank matrix after dimension reduction.

[0111] The splicing module 52 is configured to splice the low-rank matrix and the sparse matrix to obtain a spliced matrix, wherein the spliced matrix has a plurality of features.

[0112] The analysis module 53 is configured to perform redundancy analysis on the plurality of features to obtain redundant features.

[0113] The deletion module 54 is configured to delete the redundant features in the spliced matrix to obtain a feature matrix used for modeling training.

[0114] The modeling module 55 is configured to perform modeling training on the feature matrix and a target algorithm to obtain a target model.

[0115] In a possible implementation, the above-mentioned dimension reduction module 51 includes:

[0116] A division sub-module is configured to divide the sparse matrix into a plurality of sample sets according to a plurality of preset categories, and each sample set includes a plurality of sample data.

[0117] A dimension reduction operation sub-module is configured to perform dimension reduction operation on each sample set based on a plurality of preset dimensions to obtain a sub-matrix set corresponding to each sample set in each dimension, and the sub-matrix set includes a plurality of sub-matrices.

[0118] An intra-class scatter calculation sub-module is configured to calculate the intra-class scatter of the sub-matrix set in each dimension.

[0119] An inter-class scatter calculation sub-module is configured to calculate the inter-class scatter of the sub-matrix set in each dimension.

[0120] A determining sub-module is configured to determine the low-rank matrix based on the intra-class divergence and the inter-class divergence.

[0121] In a possible implementation, the intra-class divergence calculation sub-module calculates the intra-class divergence of the plurality of sub-matrices in each dimension according to the following formula:

[0122]

[0123] wherein K represents the Kth category, D k represents the Kth sub-matrix, x represents sample data of the Kth sub-matrix, N k represents the number of sample data of the Kth sub-matrix, m k represents a mean vector of sample data in the Kth sub-matrix, S w represents the intra-class divergence.

[0124] In a possible implementation, the inter-class divergence calculation sub-module calculates the inter-class divergence of the plurality of sub-matrices in each dimension according to the following formula:

[0125]

[0126] wherein i represents the ith category, j represents the jth category, mi represents a mean vector of sample data of a sub-matrix corresponding to the ith category, mj represents a mean vector of sample data of a sub-matrix corresponding to the jth category, S b represents the inter-class divergence.

[0127] In a possible implementation, the determining sub-module includes:

[0128] A target sub-matrix set determining unit is configured to determine a target sub-matrix set with the maximum inter-class divergence and the minimum intra-class divergence.

[0129] A target dimension determining unit is configured to determine a dimension corresponding to the target sub-matrix set as a target dimension of the low-rank matrix.

[0130] A low-rank matrix determining unit is configured to perform dimension reduction operation on the sparse matrix according to the target dimension, to obtain the low-rank matrix.

[0131] In a possible implementation, the analysis module 53 includes:

[0132] A feature correlation calculation sub-module is configured to calculate feature correlation between each two features.

[0133] The judgment submodule is used to determine the two target features corresponding to the feature correlation when the feature correlation is greater than a preset redundancy threshold.

[0134] A determination submodule is used to determine the information gain values ​​of the two target features respectively;

[0135] The redundancy feature determination submodule is used to identify the target feature with the smaller information gain value as the redundancy feature.

[0136] In one possible implementation, the above-mentioned device further includes:

[0137] Calculate the accuracy and recall of the target model;

[0138] If the accuracy is greater than the preset accuracy and the recall is less than the preset recall, then the target model will be deployed online.

[0139] As the apparatus embodiments are basically similar to the method embodiments, they are described in a relatively simple manner. For relevant details, please refer to the description in the method embodiment section.

[0140] Figure 6 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Figure 6 As shown, the computer device 6 of this embodiment includes: at least one processor 60 ( Figure 6 (Only one is shown) a processor, a memory 61, and a computer program 62 stored in the memory 61 and executable on the at least one processor 60, wherein the processor 60 executes the computer program 62 to implement the steps in any of the above method embodiments.

[0141] The computer device 6 can be a desktop computer, laptop, handheld computer, or cloud computing device, etc. This computer device may include, but is not limited to, a processor 60 and a memory 61. Those skilled in the art will understand that... Figure 6 The computer device 6 is merely an example and does not constitute a limitation on the computer device 6. It may include more or fewer components than shown, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0142] The processor 60 can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or the like. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0143] The memory 61 can be an internal storage unit of the computer device 6 in some embodiments, for example, a hard disk or a memory of the computer device 6. The memory 61 can also be an external storage device of the computer device 6 in other embodiments, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, or the like. Further, the memory 61 can include both an internal storage unit and an external storage device of the computer device 6. The memory 61 is used to store an operating system, an application program, a boot loader, data, and other programs, for example, program codes of the computer program, and the like. The memory 61 can also be used to temporarily store data that has been output or is to be output.

[0144] The embodiments of the present application further provide a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the steps in the above various method embodiments.

[0145] The embodiments of the present application provide a computer program product. When the computer program product is run on a computer device, the computer device is enabled to implement the steps in the above various method embodiments.

[0146] In the above embodiments, the description of each embodiment has its own focus, and the parts not described or recorded in detail in a certain embodiment can be referred to the relevant description of other embodiments.

[0147] Those skilled in the art can understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0148] In the embodiments provided by the present application, it should be understood that the disclosed apparatus / computer device and method can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely schematic. The division of the modules or units is merely a logical function division, and there can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical, mechanical or in other forms.

[0149] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e. can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0150] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than limit them. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A method of sparse matrix modeling, comprising: The method comprises the following steps: dimension reduction is performed on a sparse matrix used for modeling to obtain a low-rank matrix after dimension reduction, the sparse matrix being image feature data or text feature data, each row of the sparse matrix representing a sample data, each column representing a feature data, the low-rank matrix obtained after dimension reduction of the sparse matrix being consistent with the number of rows of the sparse matrix; the low-rank matrix and the sparse matrix are spliced to obtain a spliced matrix, the spliced matrix having a plurality of features; redundancy analysis is performed on the plurality of features to obtain redundant features; the redundant features in the spliced matrix are deleted to obtain a feature matrix used for modeling training; a target model is obtained by using the feature matrix and a target algorithm for modeling training, the target model being an image processing model or a text processing model; the dimension reduction of the sparse matrix used for modeling to obtain the low-rank matrix after dimension reduction comprises the following steps: the sparse matrix is divided into a plurality of sample sets according to a plurality of preset categories, each sample set comprising a plurality of sample data; dimension reduction operations are respectively performed on the plurality of sample sets based on a plurality of preset dimensions to obtain a sub-matrix set under each dimension, the sub-matrix set comprising a plurality of sub-matrices corresponding to the sample sets one by one; intra-class scatter of the sub-matrix set under each dimension is calculated; inter-class scatter of the sub-matrix set under each dimension is calculated; the low-rank matrix is determined based on the intra-class scatter and the inter-class scatter; the determination of the low-rank matrix based on the intra-class scatter and the inter-class scatter comprises the following steps: a target sub-matrix set with maximum inter-class scatter and minimum intra-class scatter is determined; a dimension corresponding to the target sub-matrix set is taken as a target dimension of the low-rank matrix; dimension reduction operations are performed on the sparse matrix according to the target dimension to obtain the low-rank matrix.

2. The method of claim 1, wherein, The intra-class scatter of the plurality of sub-matrix sets under each dimension is calculated by using the following formula: where k is the kth of the categories, D k is the kth of the sub-matrices, x is the sample data of the kth of the sub-matrices, N k is the number of sample data of the kth of the sub-matrices, m k is the mean vector of the sample data in the kth of the sub-matrices, S w is the intra-category dispersion.

3. The method of claim 2, wherein, The inter-class scatter of the plurality of sub-matrix sets under each dimension is calculated by using the following formula: where i is the ith category, j is the jth category, m i is the mean vector of the sample data of the sub-matrix corresponding to the ith category, m j is the mean vector of the sample data of the sub-matrix corresponding to the jth category, S b is the inter-class scatter.

4. The method of claim 1, wherein, the redundancy analysis of the plurality of features to obtain the redundant features comprises the following steps: feature correlations between two features are respectively calculated; when the feature correlation is greater than a preset redundancy threshold, two target features corresponding to the feature correlation are determined; information gain values of the two target features are respectively determined; the target feature with a smaller information gain value is taken as the redundant feature.

5. The method of claim 1, wherein, The method further comprises the following steps: accuracy and recall rate of the target model are calculated; if the accuracy is greater than a preset accuracy and the recall rate is greater than a preset recall rate, the target model is put online.

6. A sparse matrix modeling device, characterized by, The method comprises the following steps: a dimension reduction module is configured to perform dimension reduction on a sparse matrix used for modeling to obtain a low-rank matrix after dimension reduction, the sparse matrix being image feature data or text feature data, each row of the sparse matrix representing a sample data, each column representing a feature data, the low-rank matrix obtained after dimension reduction of the sparse matrix being consistent with the number of rows of the sparse matrix; The splicing module is configured to splice the low-rank matrix and the sparse matrix to obtain a spliced matrix, the spliced matrix having a plurality of features. The analysis module is configured to perform redundancy analysis on the plurality of features to obtain redundant features. The deletion module is configured to delete the redundant features in the spliced matrix to obtain a feature matrix for modeling training. The modeling module is configured to perform modeling training on the feature matrix and a target algorithm to obtain a target model, the target model being an image processing model or a text processing model. The dimension reduction module comprises: The division submodule is configured to divide the sparse matrix into a plurality of sample sets according to a plurality of preset categories, each sample set comprising a plurality of sample data. The dimension reduction operation submodule is configured to perform dimension reduction operation on each of the plurality of sample sets based on a plurality of preset dimensions to obtain a submatrix set under each dimension, the submatrix set comprising a plurality of submatrices corresponding to the sample set one by one. The intra-class scatter calculation submodule is configured to calculate the intra-class scatter of the submatrix set under each dimension. The inter-class scatter calculation submodule is configured to calculate the inter-class scatter of the submatrix set under each dimension. The determination submodule is configured to determine the low-rank matrix based on the intra-class scatter and the inter-class scatter. The determination submodule comprises: The target submatrix set determination unit is configured to determine a target submatrix set having the maximum inter-class scatter and the minimum intra-class scatter. The target dimension determination unit is configured to determine the dimension corresponding to the target submatrix set as a target dimension of the low-rank matrix. The low-rank matrix determination unit is configured to perform dimension reduction operation on the sparse matrix according to the target dimension to obtain the low-rank matrix.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the method of any one of claims 1-5.

8. A computer-readable storage medium storing a computer program, the computer-readable storage medium comprising: The computer program is executed by the processor to implement the method of any one of claims 1-5.

Images