Data analysis method and system based on digital brain

Abstract

The present invention relates to the field of data processing technology, and in particular to a data analysis method and system based on a digital brain. The method includes: obtaining samples containing multiple features in a target field through a multi-source data acquisition approach; obtaining a sample data set with labels, and generating several subspaces of the sample data set; determining the multi-scale manifold distance between samples; obtaining the reliability of each sample; determining the feature importance of each feature; constructing a random forest model based on the feature importance, applying the sample data set to train the random forest model, and using the trained random forest model to analyze the newly acquired test data. By determining the feature importance of each feature, the present invention can fully consider the differences in the contributions of different features to decision-making. When constructing a random forest model, it avoids the problem of inaccurate final output results due to the influence of inefficient features, thereby improving the accuracy of model decision-making.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a data analysis method and system based on a digital brain. Background Art

[0002] In the era of big data, data volumes are growing exponentially. Its diversity, velocity, and sheer volume have profoundly transformed decision-making across various fields, making data a crucial driver of decision-making. Traditional decision-making methods have long relied on decision-makers' past experience, subjective intuition, and extremely limited data samples. In today's complex and volatile business environments and challenging scientific research scenarios, traditional methods, relying on limited data and experience, struggle to fully capture these factors. This leads to significantly reduced decision accuracy, lengthy processes, and low efficiency.

[0003] Chinese patent application publication number CN114549211A discloses a random forest-based data analysis method, apparatus, device, and storage medium. The method comprises: receiving an analysis instruction for pending account data, the analysis instruction including pending account information, the pending account information generated from attribute information of the pending account data in a preset format; parsing the pending account information based on the preset format to obtain attribute information of the pending account data; obtaining scenario information of the pending account data and attribute information of associated data associated with the pending account data based on the attribute information of the pending account data; obtaining a target random forest model corresponding to the scenario information; and inputting the attribute information of the pending account data and the attribute information of the associated data into the target random forest model to obtain the reason for the pending account data.

[0004] However, as a commonly used decision-making model, random forest also has certain defects. When constructing random forest models and making integrated decisions, traditional random forests lack in-depth consideration of the differences in the contributions of different features in the decision-making process. When faced with multi-source heterogeneous data, such as scenarios that integrate multiple types of data such as text, images, and numbers, the model cannot accurately distinguish between important features and inefficient features, causing the decision-making process to be frequently misled by inefficient features, and the final output decision results are inaccurate. Summary of the Invention

[0005] In order to solve the problem that traditional random forests do not fully consider the differences in the contributions of different features to decision-making, and when faced with multi-source heterogeneous data, the decisions of random forest models are easily interfered by inefficient features, resulting in inaccurate final output results, the present invention provides a data analysis method and system based on digital brain.

[0006] In a first aspect, the present invention provides a data analysis method based on a digital brain, which adopts the following technical solutions:

[0007] A data analysis method based on a digital brain includes: obtaining samples containing multiple features in a target domain through a multi-source data collection approach; standardizing each feature of the sample, performing label encoding based on the standardized features of all samples to obtain a labeled sample dataset, performing dimensionality reduction on the sample dataset using t-SNE to generate several subspaces at different scales; for any two samples in the sample dataset, determining a multiscale manifold distance between the two samples based on the difference in values ​​of the two samples in each subspace and the average overlap of all samples in each subspace with preset neighboring samples in other subspaces; determining multiple neighboring samples of each sample based on the size of the multiscale manifold distance, and using the normalized result of the mean of the multiscale manifold distances between each sample and all its neighboring samples as the reliability of each sample; determining the feature importance of each feature based on the reliability, the value of each sample in each feature, and the mean value of all samples in each feature; constructing a random forest model based on the feature importance, training the random forest model using the sample dataset, and using the trained random forest model to analyze newly acquired test data.

[0008] The beneficial effects are: through the standardization, label encoding and t-SNE dimensionality reduction of sample data, the multi-dimensional information of the data can be captured from multiple scales. The subspace generated after dimensionality reduction enables the complex structure of the data and the relationship between each feature to be effectively presented, providing higher quality feature input for subsequent analysis; the multi-scale manifold distance measurement method is introduced, and by calculating the relative position and overlap of samples in multiple subspaces, the model can more accurately identify the similarities and differences between samples, effectively avoid interference caused by feature noise, and improve the robustness of the model in processing heterogeneous data; by combining the multi-scale manifold distance of each sample and the information of its adjacent samples , calculate the reliability of the sample, and based on this, comprehensively consider the performance of each sample in different features and the mean of all samples to evaluate the importance of the features, which can more accurately focus on those features that have a greater impact on decision-making, thereby improving the accuracy of the model; reconstruct the random forest model based on the feature importance calculation, so that the model can pay more attention to key features and suppress the interference of inefficient or irrelevant features, thereby improving the overall performance of the random forest model; by standardizing different features and optimizing feature selection, it can effectively process heterogeneous data with diverse sources and complex structures, and provide an efficient data analysis method for processing large-scale and diverse application scenarios.

[0009] Furthermore, the multi-source data collection approach is a relational database.

[0010] Furthermore, the standardization adopts Z-score standardization.

[0011] Furthermore, the multi-scale manifold distance is obtained as follows: any subspace is recorded as the target subspace, and for any sample, the ratio of the average overlapping number of preset neighbor samples of the sample in the target subspace and the remaining subspaces to the number of preset neighbor samples is recorded as the trust weight of the sample, and the sum of the trust weights of all samples is recorded as the trust weight of the target subspace; the product of the trust weight and the difference between the values ​​of the two samples in the target subspace is recorded as the manifold distance of the two samples in the target subspace; the standard normalized result of the sum of the manifold distances of the two samples in all subspaces is recorded as the multi-scale manifold distance of the two samples.

[0012] The beneficial effects are: by introducing the distance calculation of subspaces and the differences between samples in each subspace, the similarity of samples is comprehensively evaluated at multiple scales, which better captures the sample relationships under different dimensions, more accurately reflects the similarity between samples, and avoids errors under a single scale; by weighted summation of the distances of different subspaces, the weight of the distance calculation can be adaptively adjusted according to the degree of difference between samples in each subspace, so that the model can dynamically adjust its attention to each subspace according to the data characteristics, thereby more accurately reflecting the true relationship between samples.

[0013] Furthermore, the determining of multiple neighboring samples of each sample includes: for any sample, sorting the multi-scale manifold distances between the sample and all other samples according to size, and selecting multiple samples as neighboring samples of the sample in order from small to large.

[0014] The beneficial effect is that: through the multi-scale manifold distance, the structure of the data at different scales is taken into account, which can more accurately reflect the local structure of the data, thereby selecting more representative neighbor samples, avoiding errors at a single scale, and capturing more potential relationships between samples.

[0015] Furthermore, the normalization is performed using a logic function.

[0016] Furthermore, the feature importance is obtained as follows: any feature is recorded as the target feature, and for any sample, the negative correlation mapping of the square of the difference between the value of the sample in the target feature and the mean of all samples in the target feature is recorded as the consistency of the sample in the target feature and all samples in the target feature; the product of the consistency and the reliability of the sample is recorded as the importance of the sample in the target feature; and the mean of the sum of the importance of all samples in the target feature is recorded as the feature importance of the target feature.

[0017] The beneficial effects are: by introducing the reliability of samples, the distortion of feature importance evaluation caused by poor quality or excessive noise of some samples can be effectively avoided, and the influence of samples with low reliability on the results will be reduced, thereby improving the robustness of the model; due to the use of exponential function (negative correlation mapping), the contribution of samples with larger deviations to feature importance will be more prominent, which helps to identify those key samples with large differences from the mean in features, and can identify samples and features that may be important for classification or prediction tasks in feature selection; by evaluating feature importance, it can help to select those features that have large variability between samples and have significant contributions to the task, avoiding overly mundane or low-information features interfering with model performance.

[0018] Furthermore, the construction of the random forest model includes: the feature importance of each feature is calculated by The mapping result of the function is used as the selection probability of each feature when constructing the random forest model. Based on the selection probability, the random forest model is constructed to obtain the random forest model.

[0019] The beneficial effects are: by considering the importance of features, feature selection becomes more intelligent, which helps reduce the impact of redundant or irrelevant features and improves the accuracy of the model; traditional random forests may ignore some very important features by randomly selecting features. By mapping the importance of features to selection probabilities, it can ensure that important features have a higher selection probability when building decision trees, thereby improving model performance; through the probability distribution of feature selection, the model can select features for training more specifically, reduce unnecessary calculations, and thus improve training efficiency.

[0020] Furthermore, the analysis of the newly acquired test data includes: inputting the newly acquired test data into the trained random forest model, obtaining the decision results corresponding to the labels of the sample data set, and completing the data analysis based on the digital brain.

[0021] In a second aspect, the present invention provides a data analysis system based on the digital brain, which adopts the following technical solutions:

[0022] The data analysis system based on the digital brain includes: a processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the above-mentioned data analysis method based on the digital brain is implemented.

[0023] By adopting the above technical solution, the above-mentioned data analysis method based on the digital brain is generated into a computer program and stored in the memory to be loaded and executed by the processor, so that a terminal device is made according to the memory and the processor for easy use.

[0024] The present invention has the following technical effects:

[0025] (1) By determining the feature importance of each feature, the differences in the contributions of different features to decision-making can be fully considered. When constructing a random forest model, the interference of inefficient features on model decision-making can be reduced, avoiding the problem of inaccurate final output results due to the influence of inefficient features in the case of multi-source heterogeneous data, thereby improving the accuracy of model decision-making.

[0026] (2) t-SNE is used to reduce the dimensionality of the sample data set and generate several subspaces at different scales. Processing the data at multiple scales can more comprehensively explore the potential structure and information in the data, making the subsequent analysis of the relationship between samples more accurate and in-depth; determining the multi-scale manifold distance based on the values ​​of the samples in different subspaces and the average overlap number of preset neighboring samples can more accurately measure the similarity and distance relationship between samples, thereby more reasonably determining the neighboring samples of each sample, providing a more reliable basis for the subsequent calculation of sample reliability and feature importance.

[0027] (3) Calculating the reliability of each sample can quantitatively evaluate the quality of the sample, which helps to make more reasonable use of samples in model training, improve the stability and generalization ability of the model, and avoid the degradation of model performance due to sample quality problems; constructing a random forest model based on feature importance and training it, so that the model can better adapt to the characteristics of multi-source heterogeneous data, and can more accurately analyze the newly acquired test data, thereby improving the model's processing ability and adaptability to different types of data.

[0028] (4) The reliability, the value of the sample in the feature, and the mean of the sample in the feature are comprehensively considered to determine the feature importance, which makes full use of the multi-faceted information of the sample and makes the evaluation of the feature importance more scientific and reasonable, thereby constructing a better random forest model and improving the accuracy of subsequent data analysis. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 This is a flow chart of the data analysis method based on the digital brain in an embodiment of the present invention.

[0030] Figure 2 This is a schematic diagram of the multi-scale manifold distance in the data analysis method based on the digital brain in an embodiment of the present invention. DETAILED DESCRIPTION

[0031] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work shall fall within the scope of protection of the present invention.

[0032] The embodiment of the present invention discloses a data analysis method based on a digital brain, referring to Figure 1 , including steps S1 to S6:

[0033] S1: Obtain samples containing multiple features in the target domain through multi-source data collection.

[0034] It should be noted that the present invention is widely applicable to various fields, such as finance, healthcare, transportation, and energy. Therefore, it is necessary to process multi-source heterogeneous data (i.e., data from different sources and types). Since the random forest model is well suited for making decisions based on multi-source heterogeneous data, it is necessary to first obtain sample data for training in order to facilitate subsequent training of the random forest model. Samples are obtained by directly extracting tabular data (e.g., basic user information, device operating parameters, etc.) from the multi-source data collection channel.

[0035] Implementers can set the number of samples based on specific implementation circumstances, for example, 1000.

[0036] Specifically, the multi-source data collection approach is a relational database (eg, MySQL, Oracle, etc.).

[0037] In another embodiment, the multi-source data collection approach is an application programming interface (e.g., a financial platform transaction API, a traffic sensor real-time data interface, etc.)

[0038] S2: Obtain a labeled sample dataset and generate several subspaces of the sample dataset.

[0039] It should be noted that data labels (e.g., risk level, disease type, etc.) and key features (e.g., transaction amount, consultation time, energy consumption peak, etc.) are defined according to the target field (e.g., financial risk control requires transaction records, medical diagnosis requires electronic medical records, etc.).

[0040] Standardize each feature of the sample, encode the labels based on the standardized features of all samples, and obtain a sample data set with labels. Use t-SNE to reduce the dimension of the sample data set and generate several subspaces at different scales. When using t-SNE for dimensionality reduction, execute Independent t-SNE dimensionality reduction (illustrative, ), preset the number of neighbor samples (Exemplary, ), each time within a preset range (exemplary, ) to generate subspaces at different scales.

[0041] Implementers can set the label encoding method based on the specific implementation, for example, one-hot encoding or label encoding.

[0042] Specifically, the normalization adopts Z-score normalization.

[0043] S3: Determine the multi-scale manifold distance between samples.

[0044] It should be noted that since the characteristics of the samples are distributed in different dimensions, the sample positions are complexly distributed in high-dimensional space. For samples with low overall aggregation, directly using the distance between each sample and the centroid of the sample set will result in low sample reliability. Therefore, Figure 2 As shown, we need to first obtain the multi-scale manifold distance between each sample. In the figure, each origin represents a sample, and the black line is the multi-scale manifold distance between two samples.

[0045] For any two samples in the sample data set, the multiscale manifold distance between the two samples is determined based on the difference in the values ​​of the two samples in each subspace and the average overlap number of all samples in each subspace with the preset neighboring samples in the remaining subspaces.

[0046] Specifically, the multi-scale manifold distance satisfies:

[0047] ;

[0048] Where, For the Samples and The multi-scale manifold distance of samples, and Respectively Samples and The sample in The values ​​in the subspace, For the The sample in The average number of overlaps between the preset neighbor samples of the subspace and the rest of the subspaces, is the number of preset neighbor samples; is the number of samples in the sample data set, is the number of subspaces, is the standard normalization function, is the absolute value symbol.

[0049] in, Indicates the The trust weight of a subspace is: when the average number of overlaps between all samples in the subspace and the preset neighboring samples of samples in other subspaces is greater, the local structure under the subspace is more credible, and the impact on the multi-scale manifold distance is greater; Indicates the Samples and The sample in The distance in the subspace is Samples and The closer the distance between the samples is as a whole, the smaller the multi-scale manifold distance between the two samples is.

[0050] S4: Obtain the reliability of each sample.

[0051] It should be noted that in order to avoid the reduction in accuracy in the subsequent calculation of feature importance due to feature differences of bad samples (samples that are significantly different from normal samples due to sampling bias or noise interference), it is necessary to first obtain the reliability of each sample.

[0052] According to the size of the multi-scale manifold distance, multiple neighboring samples of each sample are determined, and the normalized result of the mean of the multi-scale manifold distance between each sample and all its neighboring samples is used as the reliability of each sample.

[0053] Specifically, determining a plurality of neighboring samples of each sample includes:

[0054] For any sample, the multi-scale manifold distances between the sample and all other samples are sorted by size, and multiple samples are selected as the nearest neighbor samples of the sample in ascending order.

[0055] Specifically, the normalization adopts the logistic function ( , ) for normalization.

[0056] Among them, the logistic function is a common S-shaped curve, and its mathematical expression is: ; is the input value (real number), It is a natural exponential function, which is an exponential function with a natural constant as the base (the natural constant is approximately equal to 2.71828). The logistic function effectively solves the mapping problem from manifold distance to reliability through nonlinear normalization. It is particularly suitable for processing local structural differences and global noise interference in high-dimensional data, and provides a stable and interpretable weight basis for subsequent feature importance calculations.

[0057] S5: Determine the feature importance of each feature.

[0058] It should be noted that due to the wide variety of multi-source heterogeneous data, it is easy for individual features of bad samples to differ significantly, resulting in inaccurate feature importance calculation. The reliability of each sample is obtained through the above steps, and then the feature importance of each feature is obtained based on the features of high-reliability samples. Finally, according to the feature importance, as many effective features as possible are fitted to establish a random forest model, so that the integrated decision of the random forest model is more accurate.

[0059] The feature importance of each feature is determined according to the reliability, the value of each sample in each feature, and the mean value of all samples in each feature.

[0060] Specifically, the feature importance satisfies:

[0061] ;

[0062] Where, For the The feature importance of a feature, For the The reliability of the sample, For the The sample in The value of the feature, For all samples in The mean of the features, is the number of samples in the sample data set, is the natural exponential function.

[0063] in, Indicates the The sample in Features and all samples in The degree of consistency in the features is such that when all samples with high reliability are in the first Features and all samples in The higher the overall consistency of the features, the The higher the feature importance of a feature.

[0064] S6: Based on the feature importance, a random forest model is constructed, the sample data set is applied to train the random forest model, and the trained random forest model is used to analyze the newly acquired test data.

[0065] It should be noted that in order to improve the accuracy of model decision-making, it is also necessary to integrate more features with high feature importance when building the random forest model.

[0066] Specifically, the construction of the random forest model includes:

[0067] The feature importance of each feature is calculated by The mapping result of the function is used as the selection probability of each feature when constructing the random forest model. Based on the selection probability, the random forest model is constructed to obtain the random forest model.

[0068] It should be noted that when training the random forest model using a labeled sample data set, the implementer can set various parameters of the random forest model according to the specific implementation situation. For example, the number of preset trees is , the maximum tree depth is , the minimum number of leaf node samples is 5. If the cross entropy loss function is selected for the classification task, if the mean square error loss function is selected for the regression task, the sample data set is concentrated. The sample is used as the training set, and 20% of the samples are used as the test set. The training set data is used to train the random forest model of the improved ensemble decision. The model is iterated until the loss function is less than the preset value of 0.8 or the test set accuracy is higher than the preset value of 96%. The model training is ended to obtain the trained random forest model.

[0069] Specifically, analyzing the newly acquired test data includes:

[0070] The newly acquired test data is input into the trained random forest model (in the same way as sample acquisition) to obtain the decision results corresponding to the labels of the sample data set (categories for classification tasks and values ​​for regression tasks). Based on the model decision results, subsequent tasks in the corresponding field (for example, tiered diagnosis and treatment in medical care, risk interception thresholds in finance, etc.) are carried out to complete data analysis based on the digital brain.

[0071] An embodiment of the present invention also discloses a data analysis system based on a digital brain, including a processor and a memory, wherein the memory stores computer program instructions. When the computer program instructions are executed by the processor, the data analysis method based on the digital brain according to the present invention is implemented.

[0072] The above system also includes other components well known to those skilled in the art, such as a communication bus and a communication interface. The configuration and functions of these components are known in the art and will not be described in detail here.

[0073] The above are all preferred embodiments of the present invention, and are not intended to limit the scope of protection of the present invention. Therefore, any equivalent changes made based on the structure, shape, and principle of the present invention should be included in the scope of protection of the present invention.

Claims

1. The data analysis method based on the digital brain is characterized by: include: Acquire samples containing multiple features in a target field through a multi-source data collection approach, wherein the target field is the financial field or the medical field, wherein the features in the financial field include transaction amounts, and the features in the medical field include consultation time; Each feature of the sample is standardized, and label encoding is performed based on the standardized features of all samples to obtain a labeled sample dataset. The label of the sample dataset in the financial field is risk level, and the label of the sample dataset in the medical field is disease type. The sample dataset is reduced in dimension using t-SNE to generate several subspaces at different scales. For any two samples in the sample data set, the multiscale manifold distance of the two samples is determined based on the difference in the values ​​of the two samples in each subspace and the average number of overlaps between all samples in each subspace and the preset nearest neighbor samples of the remaining subspaces. The multiscale manifold distance is obtained as follows: any subspace is recorded as a target subspace, and for any sample, the ratio of the average number of overlaps between the preset nearest neighbor samples of the sample in the target subspace and the remaining subspaces to the number of preset nearest neighbor samples is recorded as the trust weight of the sample, and the sum of the trust weights of all samples is recorded as the trust weight of the target subspace; the product of the trust weight and the difference in the values ​​of the two samples in the target subspace is recorded as the manifold distance of the two samples in the target subspace; and the standard normalized result of the sum of the manifold distances of the two samples in all subspaces is recorded as the multiscale manifold distance of the two samples; Determine multiple neighboring samples of each sample according to the size of the multi-scale manifold distance, and use the normalized result of the mean of the multi-scale manifold distance between each sample and all its neighboring samples as the reliability of each sample; Determine the feature importance of each feature based on the reliability, the value of each sample in each feature, and the mean value of all samples in each feature; Based on the feature importance, a random forest model is constructed, and the sample data set is applied to train the random forest model. The trained random forest model is used to analyze the newly acquired test data, including: inputting the newly acquired test data into the trained random forest model, obtaining the decision results corresponding to the labels of the sample data set, and based on the decision results, performing risk interception thresholds in the financial field or tiered diagnosis and treatment in the medical field, thereby completing data analysis based on the digital brain.

2. The data analysis method based on the digital brain according to claim 1 is characterized in that: The multi-source data collection approach is a relational database.

3. The data analysis method based on the digital brain according to claim 1 is characterized in that: The normalization adopts Z-score normalization.

4. The data analysis method based on the digital brain according to claim 1 is characterized in that: Determining a plurality of neighboring samples of each sample includes: For any sample, the multi-scale manifold distances between the sample and all other samples are sorted by size, and multiple samples are selected as the nearest neighbor samples of the sample in ascending order.

5. The data analysis method based on the digital brain according to claim 1 is characterized in that: The normalization is performed using a logic function.

6. The data analysis method based on the digital brain according to claim 1 is characterized in that: The feature importance is obtained as follows: Any feature is recorded as the target feature. For any sample, the negative correlation mapping of the square of the difference between the value of the sample in the target feature and the mean value of all samples in the target feature is recorded as the consistency of the sample in the target feature with all samples in the target feature. The product of the degree of coincidence and the reliability of the sample is recorded as the importance of the sample in the target feature; The mean of the sum of the importance of all samples in the target feature is recorded as the feature importance of the target feature.

7. The data analysis method based on the digital brain according to claim 1 is characterized in that: The random forest model is constructed, comprising: The feature importance of each feature is calculated by The mapping result of the function is used as the selection probability of each feature when constructing the random forest model. Based on the selection probability, the random forest model is constructed to obtain the random forest model.

8. The data analysis system based on the digital brain is characterized by: include: A processor and a memory, wherein the memory stores computer program instructions, and when the computer program instructions are executed by the processor, the data analysis method based on the digital brain according to any one of claims 1 to 7 is implemented.

Images