Neural network-based complex data dimension reduction method and system

Abstract

The invention discloses a complex data dimension reduction method and system based on a neural network, and belongs to the technical field of complex data processing. A feature recognition model is established, target data needing to be analyzed by a user is acquired, the target data is analyzed through the feature recognition model, and a multi-dimensional body is generated; identifying a data processing demand of a user, analyzing the multi-dimensional volume according to the data processing demand, and determining a unit demand classification corresponding to the data processing demand; marking unit classifications corresponding to the unknown data features as unit unknown classifications, and evaluating the unit unknown classifications according to the unit demand classifications and the data processing demands to obtain corresponding unknown influence values; adjusting the multi-dimensional body according to the demand classification of each unit to obtain a multi-dimensional demand body; carrying out dimensionality reduction on the target data according to the multi-dimensional demand body to obtain target dimensionality-reduced data; and analyzing the target dimension reduction data according to the data processing demand to obtain an analysis result, and displaying the analysis result to a user.

Description

Technical Field

[0001] The present invention belongs to the technical field of complex data processing, and specifically relates to a complex data dimensionality reduction method and system based on a neural network. Background Art

[0002] In the current field of data processing and analysis, individual users face many challenges, especially when dealing with complex high-dimensional data. Traditional data processing methods often divide different types of analysis tasks (such as text analysis, image analysis, etc.) into different systems, and each system may have different functional requirements (such as face recognition, quantity recognition, and text recognition in images). When users perform data analysis, they need to find suitable tools in systems on different platforms based on the type of data (such as text, images, etc.) and functional requirements (such as face recognition, quantity recognition, etc.). This processing method not only increases the complexity of the user's operation, but may also cause errors and losses in the transmission and conversion of data between different systems, causing users to spend a lot of time and energy in finding and using the corresponding system.

[0003] Based on this, in order to solve the needs of individual users for processing complex data, the present invention provides a complex data dimensionality reduction method and system based on neural network. Summary of the invention

[0004] In order to solve the problems existing in the above scheme, the present invention provides a complex data dimensionality reduction method and system based on neural network.

[0005] The purpose of the present invention can be achieved through the following technical solutions:

[0006] A complex data dimensionality reduction system based on neural network, including a feature recognition module, a demand analysis module, a dimensionality reduction module and a data analysis module;

[0007] The feature recognition module is used to perform feature recognition on complex data, establish a feature recognition model, obtain complex data that the user needs to analyze, mark it as target data, analyze the target data through the feature recognition model, and generate a multidimensional body.

[0008] Furthermore, the method for establishing the feature recognition model is:

[0009] Obtain the service scope of the platform, identify the various complex data types corresponding to the service scope, identify the various data features of each of the complex data types, select and mark each of the data features, and obtain various data reserve features; establish a feature recognition model based on each of the data reserve features and complex data types.

[0010] Furthermore, the method for selecting and marking each data feature includes:

[0011] Obtain the feature recognition requirements of the platform and establish a feature requirement judgment model. The expression of the feature requirement judgment model is:

[0012]

[0013] Where: (q, p) is the input data, q is the data feature of the corresponding complex data type, and p is the feature recognition requirement; q→p means that the data feature of the corresponding complex data type meets the feature recognition requirement; the output data is the feature requirement judgment value, and the feature requirement judgment value is 1 or 0;

[0014] Analyze the feature recognition requirements and the data features of each complex data type through the feature requirement judgment model to obtain the feature requirement judgment value corresponding to each data feature;

[0015] When the feature requirement judgment value is 1, the data feature is marked as a data reserve feature;

[0016] When the feature requirement judgment value is 0, no corresponding selection mark is performed.

[0017] Furthermore, the method of analyzing the target data by the feature recognition model includes:

[0018] The target data is subjected to feature recognition by the feature recognition model to obtain data recognition features and data unknown features; the data recognition features are data features that can be recognized by the feature recognition model; the data unknown features are data features that cannot be recognized by the feature recognition model;

[0019] Evaluate the correlation value between each of the data identification features; classify each of the data identification features according to each of the correlation values ​​to form each unit classification; integrate each of the data unknown features into a unit classification;

[0020] Evaluate the unit similarity values ​​between each of the unit classifications; and generate a multidimensional volume based on the unit similarity values ​​between each of the unit classifications.

[0021] Furthermore, the range of the associated value is [0, 100].

[0022] Furthermore, the method of classifying each data identification feature according to each association value includes:

[0023] Step SA1: The platform presets a limit value matching table, which is used to count the classification limits corresponding to each data feature;

[0024] Step SA2: matching the corresponding classification limit value for each of the data identification features according to the limit value matching table; identifying the association value between each of the data identification features;

[0025] Step SA3: performing classification evaluation according to the associated values ​​and classification limits, classifying the data identification features that meet the classification requirements into one category, and obtaining an initial classification;

[0026] Step SA4: classify and evaluate the initial classification and the remaining data identification features, and assign the data identification features that meet the classification requirements to the initial classification to obtain a new initial classification;

[0027] Step SA5: looping step SA4 until there is no data identification feature that meets the classification requirements, marking the initial classification as unit classification;

[0028] Step SA6: looping steps SA3 to SA5 until all the data identification features are classified, obtaining the classification of each unit, and forming a classification method to be selected;

[0029] Step SA7: Loop step SA3 to step SA5 F times, where F is a positive integer, to obtain several candidate classification methods; identify the number of unit classifications corresponding to each of the candidate classification methods, and determine the target classification method according to the number of unit classifications; classify according to the target classification method to obtain each unit classification.

[0030] Furthermore, methods for evaluating the unit similarity between unit classifications include:

[0031] Identify the association value between each data identification feature in two unit classifications; mark the obtained association value as GLij, i = 1, 2, ..., n, n is a positive integer; j = 1, 2, ..., m, m is a positive integer;

[0032] According to the formula Calculate unit proximity values ​​between corresponding unit classifications;

[0033] Where: DP is the unit approximate value.

[0034] The demand analysis module is used to perform demand analysis, identify the user's data processing requirements, analyze the multidimensional body according to the data processing requirements, and determine the unit demand classification corresponding to the data processing requirements.

[0035] The dimension reduction module is used to reduce the dimension of the target data, mark the unit classification corresponding to the unknown feature of the data as the unit unknown classification, obtain the demand classification and data processing requirements of each unit, evaluate the unit unknown classification according to the demand classification and data processing requirements of each unit, and obtain the corresponding unknown impact value;

[0036] The multidimensional body is adjusted according to each of the unit demand classifications to obtain a multidimensional demand body; and the target data is reduced in dimension according to the multidimensional demand body to obtain target reduced-dimension data.

[0037] The data analysis module is used to analyze the target data, obtain the target dimension reduction data and data processing requirements, analyze the target dimension reduction data according to the data processing requirements, obtain the analysis results, and display the analysis results to the user.

[0038] Furthermore, when the user is not satisfied with the analysis result and the unknown impact value is greater than 0, the data corresponding to the unknown classification of the unit is marked in the target data and displayed to the user.

[0039] A complex data dimensionality reduction method based on a neural network, the method comprising:

[0040] Establish a feature recognition model to obtain the target data that the user needs to analyze, analyze the target data through the feature recognition model, and generate a multidimensional volume;

[0041] Identify the user's data processing requirements, analyze the multidimensional body according to the data processing requirements, and determine the unit requirement classification corresponding to the data processing requirements;

[0042] The unit classification corresponding to the unknown feature of the data is marked as the unit unknown classification, and the unit unknown classification is evaluated according to the unit requirement classification and data processing requirements to obtain the corresponding unknown impact value;

[0043] The multidimensional body is adjusted according to the demand classification of each unit to obtain a multidimensional demand body; the target data is reduced in dimension according to the multidimensional demand body to obtain target reduced-dimension data.

[0044] Analyze the target dimensionality reduction data according to data processing requirements, obtain analysis results, and display the analysis results to users.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] Through the cooperation between the feature recognition module, the demand analysis module, the dimensionality reduction module and the data analysis module, the dimensionality reduction processing of complex data is realized; the present invention breaks the limitations of the traditional data processing method and no longer divides different types of analysis tasks into different systems. Users no longer need to find suitable tools in different systems according to the type of data (such as text, pictures, etc.) and functional requirements (such as face recognition, quantity recognition, etc.). This improvement not only reduces the complexity of the user's operation, but also reduces the errors and losses in the transmission and conversion of data between different systems. By integrating the extracted features into a multidimensional body, the present invention can intuitively represent the various features of high-dimensional data and their correlation. At the same time, according to the user's dimensionality reduction requirements, the present invention can process the multidimensional body to form low-dimensional data, so that users can understand and analyze data more conveniently. It combines the characteristics of publicness and personalized customization. On the one hand, it provides a general data analysis function that can meet the basic needs of most users; on the other hand, when the user needs to perform precise analysis on a specific data set or a specific analysis target, the present invention can set a personal version in a targeted manner to meet the personalized needs of the user. This combination not only improves the flexibility and scalability of the system, but also reduces the development cost and time cost of the user. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0048] Figure 1 This is a principle block diagram of the present invention. DETAILED DESCRIPTION

[0049] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] like Figure 1 As shown, a complex data dimensionality reduction system based on neural network includes a feature recognition module, a demand analysis module, a dimensionality reduction module and a data analysis module;

[0051] The feature recognition module is used to perform feature recognition on complex data, which can be various high-dimensional data such as text data, image data, text image data, etc.; the platform sets its service scope, which includes various types of complex data that can be feature recognized, identifies each type of complex data corresponding to the service scope, determines various data features possessed by the corresponding complex data type based on historical data, selects and marks each data feature according to the needs of the platform, and marks the selected marked data features as data reserve features, that is, selects which data features are within the recognition scope according to the needs of the platform, and the platform may screen and select from relevant data such as recognition cost and number of potential users; establishes corresponding feature recognition models according to each data reserve feature and complex data type;

[0052] Obtain the complex data that the user needs to analyze, mark it as target data, analyze the target data through the feature recognition model, and generate a multidimensional body. The multidimensional body is composed of recognized data features and unrecognized data features.

[0053] In one embodiment, each data feature is selectively marked according to the needs of the platform, which can be done using existing technologies, such as manual marking.

[0054] In one embodiment, the method for selecting and marking each data feature according to the platform requirements includes:

[0055] The platform sets the corresponding feature recognition requirements and establishes a feature requirement judgment model. The feature requirement judgment model is used to judge whether the data features of the corresponding complex data types meet the feature recognition requirements. The expression of the feature requirement judgment model is: Where: (q, p) is the input data, q is the data feature of the corresponding complex data type, and p is the feature recognition requirement; q→p means that the data feature of the corresponding complex data type meets the feature recognition requirement; the output data is the feature requirement judgment value, and the feature requirement judgment value is 1 or 0;

[0056] Analyze each data feature and feature recognition requirement through the feature requirement judgment model to obtain the feature requirement judgment value corresponding to each data feature;

[0057] When the feature requirement judgment value is 1, the corresponding data feature is marked for selection;

[0058] When the feature requirement judgment value is 0, no corresponding selection mark is performed.

[0059] In one embodiment, the feature recognition model is established based on existing technologies, such as based on a neural network, and is used to realize the recognition of each data reserve feature.

[0060] In one embodiment, the method of analyzing target data by using a feature recognition model includes:

[0061] The target data is identified by the feature recognition model, and each identified data reserve feature is obtained and marked as a data identification feature. When there is an unidentifiable data feature, the corresponding data feature is marked as a data unknown feature;

[0062] Evaluate the correlation between each data identification feature and express it with a correlation value. The value range of the correlation value is [0, 100]. No correlation is 0 and the maximum correlation is 100. Other values ​​are set according to correlation differences and interpolation methods. Classify each data identification feature according to each correlation value to form each unit classification. For example, the background tree in the picture has leaf features, branch features, fruit features, etc., which can form a unit classification. Different background trees form each unit classification. A corresponding correlation evaluation model can be established based on a neural network. A corresponding training set is established manually for training. The training set includes input data and output data. The input data target data corresponds to the corresponding data identification feature, and the output data is the correlation value. The corresponding correlation value is obtained by analyzing the correlation evaluation model after successful training.

[0063] Integrate the unknown features of each data into a unit classification;

[0064] Evaluate the correlation between each unit classification, and mark it as a unit similarity value for distinction; the larger the unit similarity value, the closer the two unit classifications are; generate a multidimensional body based on the unit similarity values ​​between each unit classification.

[0065] That is, first generate a data model corresponding to each unit classification, generally a three-dimensional model, marked as a unit model, and then integrate the unit models corresponding to each unit classification according to the similar values ​​of each unit to form an overall model, marked as a multidimensional body; it can be generated based on existing model technology or other technologies.

[0066] In one embodiment, each data identification feature is classified according to each association value, and the classification can be performed based on existing technologies.

[0067] In one embodiment, a method for classifying each data identification feature according to each association value includes:

[0068] Step SA1: The platform presets classification limits corresponding to different data features, that is, the minimum correlation value of the data feature and other data features classified into one category cannot be lower than the classification limit; after corresponding integration, a limit matching table is established;

[0069] Step SA2: Matching corresponding classification limits for each data identification feature according to the limit value matching table; identifying the association value between each data identification feature;

[0070] Step SA3: Compare each associated value with the corresponding classification limit, which refers to the largest classification limit between the two, and classify the data identification features that are not less than the corresponding classification limit into one category to obtain an initial classification;

[0071] Step SA4: classify and analyze the initial classification and the remaining data identification features to determine whether the classification requirements are met, that is, whether the corresponding associated values ​​in the initial classification meet the corresponding classification limit requirements; assign the data identification features that meet the classification requirements to the initial classification to obtain a new initial classification;

[0072] Step SA5: looping step SA4 until there is no data identification feature that meets the classification requirements, marking the current initial classification as unit classification;

[0073] Step SA6: looping steps SA3 to SA5 until all data identification features are classified, obtaining the classification of each unit, and forming a classification method to be selected;

[0074] Step SA7: Loop steps SA3 to SA5 F times, where F is a preset positive integer, to obtain several candidate classification methods. Because of differences in the selected merging order, different unit classifications may be formed, so multiple loop analysis is required; identify the number of unit classifications corresponding to each candidate classification method, select the candidate classification method with the least number of unit classifications for classification, and obtain each unit classification; if the number is the same, you can screen according to various additional factors such as generation time, distribution concentration of each unit classification, etc.

[0075] In one embodiment, a method of evaluating unit proximity between unit classifications includes:

[0076] Identify the correlation value between each data identification feature in two unit classifications; for the unit classification corresponding to the unknown data feature, the single item approximation value can be assumed to be the smallest, which is 0, and the obtained correlation value is marked as GLij, i = 1, 2, ..., n, where n is a positive integer; j = 1, 2, ..., m, where m is a positive integer;

[0077] According to the formula Calculate unit proximity values ​​between corresponding unit classifications;

[0078] Where: DP is the unit approximate value.

[0079] The demand analysis module is used to analyze the user's data processing needs. The user enters the corresponding data processing needs in the input bar, such as quantity recognition, text recognition, face recognition and other data processing needs, identifies the user's data processing needs, analyzes the multidimensional body according to the identified data processing needs, determines the unit classifications that are useful for the data processing needs, eliminates useless data identification features, and marks them as unit requirement classifications; uses the data required for existing data analysis to make the determination.

[0080] The dimension reduction module is used to reduce the dimension of the target data, identify each unit requirement classification, mark the unit classification corresponding to the unknown feature of the data in the multidimensional body as the unit unknown classification, evaluate the unit unknown classification according to each unit requirement classification and the data processing requirement, and analyze whether it affects the data processing requirement, that is, whether the data processing requirement can be applied to the data corresponding to the unit unknown classification. It can be evaluated from whether each unit requirement classification can completely solve the data processing requirement. If it is estimated that there is a 90% probability of solving it, the impact of the unit unknown class is 10%, which is marked as an unknown impact value;

[0081] The multidimensional body is adjusted according to each unit demand classification, and the non-unit demand classification is eliminated to form a multidimensional demand body; the target data is reduced in dimension according to the multidimensional demand body to obtain the target reduced-dimension data.

[0082] The data analysis module is used to analyze the target dimensionality reduction data according to the data processing requirements and obtain the corresponding analysis results, so as to utilize the subsequent existing technology for analysis and realize the data processing requirements, such as quantity identification results, etc.; and mark the corresponding unknown impact values ​​for the analysis results.

[0083] When the user is not satisfied with the analysis result, and the unknown impact value is greater than 0, the data corresponding to the unknown classification of the unit is marked in the target data and displayed to the user.

[0084] A complex data dimensionality reduction method based on a neural network, the method comprising:

[0085] Establish a feature recognition model to obtain the target data that the user needs to analyze, analyze the target data through the feature recognition model, and generate a multidimensional volume;

[0086] Identify the user's data processing requirements, analyze the multidimensional body according to the data processing requirements, and determine the unit requirement classification corresponding to the data processing requirements;

[0087] The unit classification corresponding to the unknown feature of the data is marked as the unit unknown classification, and the unit unknown classification is evaluated according to the unit requirement classification and data processing requirements to obtain the corresponding unknown impact value;

[0088] The multidimensional body is adjusted according to the demand classification of each unit to obtain a multidimensional demand body; the target data is reduced in dimension according to the multidimensional demand body to obtain target reduced-dimension data.

[0089] Analyze the target dimensionality reduction data according to data processing requirements, obtain analysis results, and display the analysis results to users.

[0090] The above formulas are all calculated by removing dimensions and taking numerical values. The formula is a formula that is closest to the actual situation obtained by collecting a large amount of data and performing software simulation. The preset parameters and preset thresholds in the formula are set by technical personnel in this field according to actual conditions or obtained by simulating a large amount of data.

[0091] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A complex data dimensionality reduction system based on neural network, characterized in that: It includes feature recognition module, demand analysis module, dimension reduction module and data analysis module; The feature recognition module is used to perform feature recognition on complex data, establish a feature recognition model, obtain complex data that the user needs to analyze, mark it as target data, analyze the target data through the feature recognition model, and generate a multidimensional volume; The demand analysis module is used to perform demand analysis, identify the user's data processing requirements, analyze the multidimensional body according to the data processing requirements, and determine the unit demand classification corresponding to the data processing requirements; The dimension reduction module is used to reduce the dimension of the target data, mark the unit classification corresponding to the unknown feature of the data as the unit unknown classification, obtain the demand classification and data processing requirements of each unit, evaluate the unit unknown classification according to the demand classification and data processing requirements of each unit, and obtain the corresponding unknown impact value; Adjusting the multidimensional body according to each unit demand classification to obtain a multidimensional demand body; reducing the dimension of the target data according to the multidimensional demand body to obtain target reduced-dimensional data; The data analysis module is used to analyze the target data, obtain the target dimension reduction data and data processing requirements, analyze the target dimension reduction data according to the data processing requirements, obtain the analysis results, and display the analysis results to the user.

2. A complex data dimensionality reduction system based on neural network according to claim 1, characterized in that: The method for establishing the feature recognition model is: Obtain the service scope of the platform, identify the various complex data types corresponding to the service scope, identify the various data features of each of the complex data types, select and mark each of the data features, and obtain various data reserve features; establish a feature recognition model based on each of the data reserve features and complex data types.

3. A complex data dimensionality reduction system based on neural network according to claim 2, characterized in that: Methods for selecting and marking each data feature include: Obtain the feature recognition requirements of the platform and establish a feature requirement judgment model. The expression of the feature requirement judgment model is: Where: (q, p) is the input data, q is the data feature of the corresponding complex data type, and p is the feature recognition requirement; q→p means that the data feature of the corresponding complex data type meets the feature recognition requirement; the output data is the feature requirement judgment value, and the feature requirement judgment value is 1 or 0; Analyze the feature recognition requirements and the data features of each complex data type through the feature requirement judgment model to obtain the feature requirement judgment value corresponding to each data feature; When the feature requirement judgment value is 1, the data feature is marked as a data reserve feature; When the feature requirement judgment value is 0, no corresponding selection mark is performed.

4. The complex data dimensionality reduction system based on neural network according to claim 1, characterized in that: Methods for analyzing target data through feature recognition models include: The target data is subjected to feature recognition by the feature recognition model to obtain data recognition features and data unknown features; the data recognition features are data features that can be recognized by the feature recognition model; the data unknown features are data features that cannot be recognized by the feature recognition model; Evaluate the correlation value between each of the data identification features; classify each of the data identification features according to each of the correlation values ​​to form each unit classification; integrate each of the data unknown features into a unit classification; Evaluate the unit similarity values ​​between each of the unit classifications; and generate a multidimensional volume based on the unit similarity values ​​between each of the unit classifications.

5. A complex data dimensionality reduction system based on neural network according to claim 4, characterized in that: The associated value range is [0, 100].

6. A complex data dimensionality reduction system based on neural network according to claim 4, characterized in that: Methods for classifying each data identification feature according to each associated value include: Step SA1: The platform presets a limit value matching table, which is used to count the classification limits corresponding to each data feature; Step SA2: matching the corresponding classification limit value for each of the data identification features according to the limit value matching table; identifying the association value between each of the data identification features; Step SA3: performing classification evaluation according to the associated values ​​and classification limits, classifying the data identification features that meet the classification requirements into one category, and obtaining an initial classification; Step SA4: classify and evaluate the initial classification and the remaining data identification features, and assign the data identification features that meet the classification requirements to the initial classification to obtain a new initial classification; Step SA5: looping step SA4 until there is no data identification feature that meets the classification requirements, marking the initial classification as unit classification; Step SA6: looping steps SA3 to SA5 until all the data identification features are classified, obtaining the classification of each unit, and forming a classification method to be selected; Step SA7: Loop step SA3 to step SA5 F times, where F is a positive integer, to obtain several candidate classification methods; identify the number of unit classifications corresponding to each of the candidate classification methods, and determine the target classification method according to the number of unit classifications; classify according to the target classification method to obtain each unit classification.

7. The complex data dimensionality reduction system based on neural network according to claim 4, characterized in that: Methods for evaluating unit similarity between unit classifications include: Identify the association value between each data identification feature in two unit classifications; mark the obtained association value as GLij, i = 1, 2, ..., n, n is a positive integer; j = 1, 2, ..., m, m is a positive integer; According to the formula Calculate unit proximity values ​​between corresponding unit classifications; Where: DP is the unit approximate value.

8. The complex data dimensionality reduction system based on neural network according to claim 1, characterized in that: When the user is not satisfied with the analysis result and the unknown impact value is greater than 0, the data corresponding to the unknown classification of the unit is marked in the target data and displayed to the user.

9. A complex data dimensionality reduction method based on neural network, characterized in that: A complex data dimensionality reduction system based on a neural network as claimed in any one of claims 1 to 8, the method comprising: Establish a feature recognition model to obtain the target data that the user needs to analyze, analyze the target data through the feature recognition model, and generate a multidimensional volume; Identify the user's data processing requirements, analyze the multidimensional body according to the data processing requirements, and determine the unit requirement classification corresponding to the data processing requirements; The unit classification corresponding to the unknown feature of the data is marked as the unit unknown classification, and the unit unknown classification is evaluated according to the unit requirement classification and data processing requirements to obtain the corresponding unknown impact value; According to the demand classification of each unit, the multidimensional body is adjusted to obtain a multidimensional demand body; according to the multidimensional demand body, the target data is reduced in dimension to obtain target reduced-dimension data; Analyze the target dimensionality reduction data according to data processing requirements, obtain analysis results, and display the analysis results to users.

Images