Computer data analysis system and method based on big data

Abstract

The invention provides a computer data analysis system and method based on big data, and relates to the technical field of computer data analysis.The computer data analysis method comprises the steps that firstly, through time axis alignment and standardization processing, it is guaranteed that data can be processed under a unified time frame, and then through sample label distribution and basic feature vector extraction, the data can be processed under the unified time frame; the method comprises the steps of converting original data into standardized features which can be processed by a machine learning model, then capturing static features of the data by calculating a time sequence feature vector, deeply understanding a behavior pattern of the data changing along with time, fusing basic features and time sequence features to construct feature representation with richer information dimensions, and finally, carrying out feature representation on the basis of the basic features and the time sequence features. According to the method, a feature subset with the highest prediction value is screened out by calculating the correlation strength, the calculation complexity and the over-fitting risk brought by high-dimensional data are overcome, finally, a data analysis result is output through a model training and prediction analysis module, and automatic processing from original data to intelligent decision making is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer data analysis, and particularly relates to a computer data analysis system and method based on big data. BACKGROUND

[0002] In the current information age, with the deepening of enterprise digital transformation, various business systems, sensor devices and user interaction platforms continue to generate massive multi-source heterogeneous data. These data are usually derived from different databases, log files, real-time data streams and other multi-source heterogeneous data sources, and their formats, structures and collection frequencies are different, forming a data island phenomenon. Traditional computer data analysis methods often cannot effectively integrate these multi-source heterogeneous data, especially when dealing with data with time attributes, conventional methods usually only focus on static data snapshots, lack of in-depth mining of dynamic change trends of data, resulting in that the analysis results cannot fully reflect the time sequence rules and evolution characteristics behind the data. On the other hand, due to the high feature dimension in the big data environment, there are often a large amount of redundant features and noise data, which not only increases the computational complexity, but also reduces the accuracy and interpretability of the prediction model. Existing feature selection methods mostly rely on manual experience or simple statistical tests, and it is difficult to automatically identify key features with real predictive value from massive features, and the dynamic change characteristics of features in the time dimension are not fully considered.

[0003] Therefore, it is necessary to provide a computer data analysis system and method based on big data to solve the above technical problems. SUMMARY

[0004] To solve the above technical problems, the present application provides a computer data analysis system and method based on big data, which can process multi-source heterogeneous data, effectively mine time sequence features, intelligently select key features and realize end-to-end analysis.

[0005] The present application provides a computer data analysis method based on big data, comprising:

[0006] S1: parsing entities from multi-source heterogeneous data, aligning the multi-source heterogeneous data to a predefined main time axis and performing standardization processing to generate standard data records of each entity at each time point on the main time axis;

[0007] S2: assigning a sample label to each standard data record based on a predefined business rule to generate a sample label set, and extracting a basic feature vector from each standard data record;

[0008] S3: For each feature dimension of each base feature vector, calculate a change trend indicator of each feature dimension by a preset sliding time window in the order of the main time axis, to generate a time series feature vector corresponding to each time point;

[0009] S4: Concatenate the base feature vector and the time series feature vector of each time point on the main time axis to obtain a joint feature vector of each entity at each time point on the main time axis;

[0010] S5: Construct a feature space with all joint feature vectors of all entities, and take the sample label set as a prediction target, to calculate the correlation strength between each feature and the prediction target;

[0011] S6: According to the correlation strength, sort and select a preset number of features from the feature space to form a core feature subset, input the core feature subset and its corresponding sample label into the prediction model for training, and output the data analysis result using the trained prediction model.

[0012] Preferably, in step S1, the pre-defined main time axis is composed of consecutive time points with a preset fixed time interval.

[0013] Preferably, in step S1, the generation of the standard data record comprises the following steps:

[0014] According to the analysis requirement, mark the time interval between two consecutive time points on the main time axis as a time bucket;

[0015] Map the multi-source heterogeneous data to the corresponding time bucket according to the timestamp;

[0016] According to the business meaning, select an aggregation function to calculate the data mapped to the same time bucket, and compress multiple data into one summary record, i.e. a standard data record.

[0017] Preferably, in step S2, the sample label assignment comprises:

[0018] According to whether at least one field value in the standard data record meets a preset judgment condition, assign a discrete sample label representing a classification result to it.

[0019] Preferably, in step S3, the generation of the time series feature vector comprises:

[0020] Arrange the base feature vectors in the order of the main time axis to form a base feature vector sequence;

[0021] For each feature dimension in the base feature vector sequence, independently extract the numerical value at different time points to form a time series of the feature dimension;

[0022] For each time point on the main time axis, based on a preset sliding time window, a change trend indicator is calculated on the time series of its corresponding feature dimension;

[0023] The values of the change trend indicators calculated for all feature dimensions of the time point are combined to generate a time series feature vector corresponding to the time point.

[0024] Preferably, in step S4, the splicing is vector splicing, that is, the base feature vector and the time series feature vector are connected in dimension to obtain a joint feature vector with higher dimension.

[0025] Preferably, in step S5, the calculation step of the correlation strength includes:

[0026] The feature importance evaluation model is trained with the feature space as the input feature matrix and the prediction target as the target vector;

[0027] The contribution score of each column feature in the feature matrix to the target vector is calculated by the feature importance evaluation algorithm built in the feature importance evaluation model;

[0028] The contribution score of each feature is taken as the correlation strength of the feature to the target vector.

[0029] Preferably, in step S6, the screening step of the core feature subset includes:

[0030] The features in the feature space are arranged in descending order according to the correlation strength to generate a feature importance ranking list;

[0031] The top pre-set number of features in the feature importance ranking list are selected to form a core feature subset, wherein the pre-set number is determined by analyzing the inflection point on the correlation strength distribution curve corresponding to the feature importance ranking list by the elbow rule.

[0032] Preferably, in step S6, the prediction model is a logistic regression model.

[0033] The application also provides a computer data analysis system based on big data, which is applied to a computer data analysis method based on big data and includes:

[0034] A data standardization module is configured to parse entities from multi-source heterogeneous data, align the multi-source heterogeneous data to a predefined main time axis, and perform standardization processing to generate standard data records of each entity at each time point on the main time axis;

[0035] A base feature extraction module is configured to assign a sample label to each standard data record based on a predefined business rule to generate a sample label set, and extract a base feature vector from each standard data record;

[0036] a time sequence feature generation module, configured to calculate a change trend index of each feature dimension through a preset sliding time window in the order of the main time axis for each feature dimension of each basic feature vector, and generate a time sequence feature vector corresponding to each time point;

[0037] a feature splicing module, configured to splice the basic feature vector and the time sequence feature vector of each time point on the main time axis to obtain a joint feature vector of each entity at each time point on the main time axis;

[0038] a feature importance evaluation module, configured to construct a feature space with all joint feature vectors of all entities, take the sample label set as a prediction target, and calculate the correlation strength between each feature and the prediction target;

[0039] a model training and prediction analysis module, configured to filter a preset number of features from the feature space to form a core feature subset according to the correlation strength ranking, input the core feature subset and the corresponding sample label into a prediction model for training, and output a data analysis result by using the trained prediction model.

[0040] Compared with the related art, the computer data analysis system and method based on big data provided by the application has the following beneficial effects:

[0041] Firstly, the application parses entities from multi-source heterogeneous data and aligns them to a unified main time axis, generates standard data records through standardization processing, and ensures that data of different sources and formats can be integrated and compared under a unified time framework. Then, sample labels are assigned to each data record and basic feature vectors are extracted based on predefined business rules, so that the original data is converted into standardized features that can be processed by a machine learning model. Next, by performing sliding window calculation on each feature dimension in time sequence, feature indicators with time dynamic change trend are extracted, which not only captures the static features of the data, but also deeply understands the behavior pattern of the data changing over time, thereby significantly improving the ability to describe dynamic business scenarios. Secondly, the basic features and time sequence features are fused to construct a joint feature space with richer information dimensions and stronger representation ability, providing a comprehensive and in-depth feature representation for the model. Then, the correlation strength between each feature and the prediction target is calculated to provide a scientific basis for feature screening. Subsequently, the core feature screening mechanism automatically identifies the feature subset with the most prediction value. This process not only effectively reduces the data dimension, overcomes the computational complexity and overfitting risk brought by high-dimensional data, but also improves the explainability and generalization ability of the model. Finally, the data analysis result is output by the model training and prediction analysis module, and the entire process realizes end-to-end automatic processing from raw data to intelligent decision-making. BRIEF DESCRIPTION OF DRAWINGS

[0042] Figure 1 A flow chart of a computer data analysis method based on big data according to the present application;

[0043] Figure 2 A module structure diagram of a computer data analysis system based on big data according to the present application. DETAILED DESCRIPTION

[0044] The present application will be further described below in conjunction with the accompanying drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the present application, but not to limit the present application. In addition, it should be noted that, for the convenience of description, only the parts related to the present application are shown in the drawings, but not all the structures. In addition, the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0045] In addition, it should be noted that, for the convenience of description, only the parts related to the present application are shown in the drawings, but not all the contents. Before discussing the exemplary embodiments in more detail, it should be mentioned that some exemplary embodiments are described as processes or methods depicted as flow charts. Although the flow chart describes each operation (or step) as a sequential process, many of the operations can be implemented in parallel, concurrently or simultaneously. In addition, the order of the operations can be rearranged. The process can be terminated when its operations are completed, but can also have additional steps not included in the drawings. The process can correspond to a method, function, procedure, subroutine, etc.

[0046] Embodiment one

[0047] A computer data analysis method based on big data, in the specific implementation process, as shown in Figure 1 A flow chart of a computer data analysis method based on big data, including:

[0048] Step S1: parsing entities from multi-source heterogeneous data, aligning multi-source heterogeneous data to a predefined main time axis and performing standardization processing to generate standard data records of each entity at each time point on the main time axis.

[0049] Specifically, in step S1, the predefined main time axis is composed of continuous time points with preset fixed time intervals.

[0050] Specifically, in step S1, the generation of standard data records includes the following steps:

[0051] According to the analysis requirement, the time interval between two continuous time points on the main time axis is marked as a time bucket;

[0052] Map the multi-source heterogeneous data to the corresponding time bucket according to the timestamp;

[0053] For the data mapped to the same time bucket, select an aggregation function according to the business meaning to calculate, and compress multiple data into one summary record, that is, a standard data record.

[0054] In the implementation process, first, parse the entity from the multi-source heterogeneous data. The entity refers to the core object in business analysis, which is a business subject with a unique identifier and can exist independently and carry data. For example, users in an e-commerce system, packages in a logistics system, and devices in an industrial Internet of Things are all entities. Each entity is tracked and associated through a unique identifier. Then, a master timeline needs to be defined. The master timeline is composed of a series of consecutive time points with a preset fixed time interval. This fixed time interval is set according to business analysis requirements, including but not limited to different granularities such as every minute, every hour, and every day. Each time point represents a unified time scale reference. Based on this master timeline, the time interval between two consecutive time points is marked as a time bucket. The time bucket is the basic time unit for data alignment in the present application, and its length is completely consistent with the preset fixed time interval of the master timeline. For example, when the master timeline is spaced by hours, each time bucket represents a complete hour interval. Next, map the heterogeneous data from different sources, including but not limited to database tables, application logs, and sensor data streams, to the corresponding target time bucket according to their original timestamps. This mapping process ensures that all data is standardized under a unified time framework. For multiple data mapped to the same time bucket, select the corresponding aggregation function according to the business meaning to calculate. For example, for numerical fields, use summation or average value calculation, and for state fields, use the latest value operation. This aggregation process compresses multiple data records in the bucket to generate one standardized summary record. The final output standard data record contains a unified time bucket identifier, entity identifier, and various feature values calculated by aggregation, forming a standardized data basis for subsequent analysis processing. Through this time bucket-based data alignment and aggregation mechanism, the problem of inconsistent time granularity of multi-source heterogeneous data leading to analysis difficulties is effectively solved.

[0055] Step S2: Assign a sample label to each standard data record based on the predefined business rules to generate a sample label set, and extract a basic feature vector from each standard data record.

[0056] Specifically, in step S2, the assignment of sample labels includes:

[0057] Based on whether at least one field value in the standard data record meets a preset judgment condition, a discrete sample label representing the classification result is assigned.

[0058] In the implementation process, after obtaining the standard data record of each entity at each time point, it is necessary to first assign a sample label to each record based on predefined business rules, wherein the predefined business rules refer to a series of logical judgment conditions formulated by domain experts in advance according to the analysis target of a specific business scenario. These rules are usually represented as threshold judgment or logical combination judgment on the value of a specific field in the data record. For example, in the user churn prediction scenario, a typical business rule can be to judge whether the user has not produced any active behavior in a future predetermined period of time. If so, it is marked as a positive sample, otherwise as a negative sample. After applying such business rules to each standard data record, a discrete sample label representing the classification result is assigned to it. The sample label is a discrete classification identifier, usually in the form of an integer value, used to represent the actual business state or category attribution of the data record in a supervised learning task. The sample labels of all records together constitute a sample label set. At the same time, the basic feature vector is extracted from each standard data record. The basic feature vector is a numerical vector data structure obtained by feature engineering processing of various fields in the standard data record. These processing methods include but are not limited to direct value taking of continuous numerical fields, one-hot encoding or label encoding conversion of category fields, keyword extraction and word frequency statistics of text fields, and periodic feature extraction of time fields. The final basic feature vector contains feature representations that can represent the static state of the entity at the current time point. These features and the corresponding sample labels together constitute the training sample data required for supervised learning.

[0059] Step S3: For each feature dimension of each basic feature vector, calculate the change trend indicator of each feature dimension through a preset sliding time window in the order of the main time axis, to generate a time series feature vector corresponding to each time point.

[0060] Specifically, in step S3, the generation step of the time series feature vector includes:

[0061] Arrange the basic feature vectors in the time order of the main time axis to form a basic feature vector sequence;

[0062] For each feature dimension in the basic feature vector sequence, independently extract its values at different time points to form a time series of the feature dimension;

[0063] For each time point on the main time axis, based on a preset sliding time window, calculate the change trend indicator on the time series of its corresponding feature dimension;

[0064] Combine the values of the change trend indicators calculated for all feature dimensions at the time point to generate a time series feature vector corresponding to the time point.

[0065] In the implementation process, first, all the basic feature vectors corresponding to each entity are strictly arranged in the time sequence of the main time axis to form a complete basic feature vector sequence, which completely records the static feature state of the entity at each time point; then each feature dimension in the basic feature vector sequence is processed independently, the feature dimension refers to a single numerical field in the feature vector representing a specific attribute, for example, user consumption amount, device temperature value and other independent indicators, the numerical value of each feature dimension at different time points is extracted separately to form independent time series data of the feature dimension, this process ensures that the change pattern of each feature can be analyzed independently; then for each specific time point on the main time axis, calculation is performed based on a preset sliding time window, the sliding time window refers to a dynamic window interval extending a specific time length forward from the current time point as the end point, for example, a time range of seven days is traced back from the current time point as the reference, the change trend indicators of the time series data of each feature dimension are calculated in this window interval, the change trend indicators include but are not limited to moving average, first-order difference slope, standard deviation coefficient of variation and other statistical quantities, the moving average reflects the average level in the window period, the first-order difference slope represents the change direction and intensity, and the standard deviation measures the fluctuation degree of the data; finally, the various change trend indicator values calculated for all feature dimensions at the time point are combined to generate a time series feature vector completely corresponding to the time point, this time series feature vector contains the dynamic change information of all features in the recent window, and is complementary to the basic feature vector at the same time point, together forming the complete feature representation of the entity at the time point.

[0066] Step S4: Splicing the basic feature vector and the time series feature vector at each time point on the main time axis to obtain the joint feature vector of each entity at each time point on the main time axis.

[0067] Specifically, in step S4, the splicing is vector splicing, that is, the basic feature vector and the time series feature vector are connected in dimension to obtain a joint feature vector with higher dimension.

[0068] In the implementation process, the basic feature vector at each time point on the main time axis is spliced with the time series feature vector. The splicing operation is specifically a vector splicing, which is a feature fusion technology commonly used in machine learning. Its essence is to connect two or more feature vectors on the dimension axis to form a new large-dimension feature vector. Specifically, for each entity at each specific time point on the main time axis, the basic feature vector represents the static feature representation of the entity at the time point. These features include numerical values of original fields extracted from standard data records after encoding or standardization. Exemplarily, they include but are not limited to direct measurement values such as user age, product price and device temperature. The time series feature vector represents the dynamic behavior trend of the entity within a period of time before the time point. These trend indicators are calculated by a sliding time window. Exemplarily, they include but are not limited to indicators such as moving average, slope and volatility. The splicing process first ensures that the basic feature vector and the time series feature vector are completely aligned in the sample dimension, that is, each vector corresponds to the same entity and the same time point. Then, all elements of the basic feature vector and all elements of the time series feature vector are connected in order by numerical operation. Exemplarily, if the basic feature vector has M dimensions and the time series feature vector has N dimensions, the joint feature vector after splicing will have M+N dimensions. The connection in this dimension not only increases the number of features, but also integrates static attributes and dynamic behavior information, thereby providing more comprehensive and rich feature representation. The generation of the joint feature vector enables the subsequent machine learning model to simultaneously utilize the current state and historical change pattern of the entity for learning, thereby enhancing the expression ability and prediction accuracy of the model and providing a data basis for the input of subsequent feature selection and model training.

[0069] Step S5: All joint feature vectors of all entities constitute a feature space, and the sample label set is used as a prediction target to calculate the correlation strength between each feature and the prediction target.

[0070] Specifically, in step S5, the calculation step of the correlation strength includes:

[0071] A feature importance evaluation model is trained with the feature space as an input feature matrix and the prediction target as a target vector.

[0072] The contribution score of each column feature in the feature matrix to the target vector is calculated by a feature importance evaluation algorithm built in the feature importance evaluation model.

[0073] The contribution score of each feature is taken as the correlation strength of the feature to the target vector.

[0074] In the implementation process, first, all joint feature vectors are integrated according to entity identifiers and time points in sequence to form a structured feature space, which is a two-dimensional matrix data structure. Each row represents the complete feature representation of a specific entity at a specific time point, that is, a joint feature vector. Each column represents the value distribution of all samples in the same feature dimension. Here, a sample refers to a complete record composed of a specific entity at a specific time point and its corresponding complete feature data. It is the basic unit of a row of data in the feature space. Each sample is uniquely determined by an entity identifier and a time point, and serves as the smallest data object for machine learning model training and prediction. At the same time, the sample label set is used as the prediction target, which is a vector set corresponding to the row records of the feature space, containing the business label value corresponding to each sample. Then, the correlation strength calculation stage is entered. First, a feature importance evaluation model needs to be trained. The feature importance evaluation model is a machine learning model specifically used to quantify the correlation between features and prediction targets. It learns the mapping relationship between features and prediction targets by analyzing the feature space matrix and the target vector. During model training, the feature importance evaluation algorithm built-in the model is used for calculation. This algorithm is a feature contribution quantification mechanism implemented internally in the model. The feature importance evaluation model includes but is not limited to tree models and linear models. For tree models, the degree of impurity reduction brought by features at split nodes is calculated cumulatively. For linear models, the absolute value of the regression coefficient is evaluated. Finally, the contribution score of each column feature in the feature matrix to the target vector is output. The contribution score is a numerical indicator used to quantify the relative importance of each feature in the prediction target process. Its calculation process involves mathematical quantification of the contribution of each feature in model prediction. For example, by calculating the total information gain brought by the feature in all decision tree split nodes, or by analyzing the degree of influence of feature value disturbance on model output. Finally, the contribution score of each feature is used as its correlation strength with the target vector. The correlation strength is a standardized importance indicator, and its numerical size directly reflects the influence of the feature on the prediction target. The larger the value, the stronger the correlation between the feature and the prediction target, and the higher the priority in the subsequent feature selection process. The entire calculation process ensures the objectivity and accuracy of feature importance evaluation, providing a reliable quantitative basis for subsequent feature selection.

[0075] Step S6: According to the correlation strength ranking, a preset number of features are selected from the feature space to form a core feature subset. The core feature subset and its corresponding sample labels are input into the prediction model for training, and the trained prediction model is used to output data analysis results.

[0076] Specifically, in step S6, the selection of the core feature subset includes:

[0077] arrange the features in the feature space in descending order according to the correlation intensity from large to small to generate a feature importance ranking list;

[0078] select top pre-set number of features from the feature importance ranking list to form a core feature subset, wherein the pre-set number is determined by elbow rule analysis of an inflection point on a correlation intensity distribution curve corresponding to the feature importance ranking list.

[0079] Specifically, in step S6, the prediction model is a logistic regression model.

[0080] In the specific implementation process, first, arrange the features in descending order according to the correlation intensity from large to small to generate a feature importance ranking list, which is an ordered feature name sequence, and the arrangement order directly reflects the importance level of different features for the prediction target; then determine the size of the core feature subset, i.e. the value of the pre-set number, which is achieved by elbow rule, a heuristic method for determining the best value by analyzing the curve inflection point. The specific operation is to draw a correlation intensity distribution curve with the feature order in the feature importance ranking list as the horizontal coordinate and the corresponding correlation intensity value as the vertical coordinate. The curve usually presents a decreasing form from the top left to the bottom right, and there is a clear inflection point on it, which is the elbow point, and the corresponding horizontal coordinate position is the best value of the pre-set number. This point means that the increase of the number of features will sharply weaken the improvement of the model performance. After determining the value of the pre-set number, select top pre-set number of features from the feature importance ranking list to form a core feature subset, which contains the most predictive feature combination. Then extract the data corresponding to the core feature subset from the original feature space and input it into the prediction model together with the corresponding sample labels for training, wherein the prediction model adopts a logistic regression model. Logistic regression is a linear model applied to binary classification problems, which maps the linear combination of features to the probability space through a logistic function, and has the characteristics of strong interpretability and high computational efficiency. After the model training is completed, the trained logistic regression model is used to predict and analyze new data, and the final data analysis result is output in the form of a probability value or a classification label, providing direct data support for decision-making. The whole process ensures the accuracy and reliability of the analysis result through scientific feature selection and special prediction model.

[0081] The working principle of the computer data analysis method based on big data provided by the application is as follows:

[0082] First, entities are parsed from multi-source heterogeneous data, and the data is aligned to a predefined master timeline for standardization processing, the master timeline is composed of continuous time points with preset fixed time intervals, after mapping the multi-source data to the corresponding time bucket according to the time stamp by marking the time interval between the continuous time points as a time bucket, the multi-source data is compressed to generate standard data records according to the business meaning selection aggregation function; Next, a discrete sample label representing the classification result is assigned to each record based on the predefined business rules, and a basic feature vector is extracted, wherein the business rules refer to logical rules for determining whether the field values in the record meet the preset judgment conditions; Then, each feature dimension of each basic feature vector is arranged in time sequence to form a sequence, and the time sequence data of each feature dimension is independently extracted, the change trend index of each feature dimension on the corresponding time sequence is calculated through a preset sliding time window, and all index values are combined to generate a time sequence feature vector; Subsequently, the basic feature vector and the time sequence feature vector are spliced in the dimension of the vector to obtain a joint feature vector containing static features and dynamic trends; The feature space is composed of all joint feature vectors, and the sample label set is used as the prediction target, the feature importance evaluation model is trained, and the contribution score of each feature to the prediction target is calculated as the correlation strength through the built-in algorithm; Finally, the feature importance ranking list is generated in descending order of correlation strength, the elbow rule is used to analyze the inflection point of the correlation strength distribution curve to determine the preset number, the top-ranked features are selected to form a core feature subset, and the subset and the corresponding sample label are input into a logistic regression model for training. Finally, the trained model outputs the data analysis result, realizing a complete technical solution from multi-source heterogeneous data to high-quality prediction analysis.

[0083] Embodiment two

[0084] A computer data analysis system based on big data, in the specific implementation process, as shown in Figure 2 It shows a module structure diagram of a computer data analysis system based on big data, which includes:

[0085] The data standardization module 100 is used for parsing entities from multi-source heterogeneous data, aligning the multi-source heterogeneous data to a predefined master timeline and performing standardization processing, and generating standard data records of each entity at each time point on the master timeline;

[0086] The basic feature extraction module 200 is used for assigning a sample label to each standard data record based on the predefined business rules to generate a sample label set, and extracting a basic feature vector from each standard data record;

[0087] The time sequence feature generation module 300 is configured to calculate a change trend index of each feature dimension through a preset sliding time window in the order of the main time axis for each feature dimension of each basic feature vector, and generate a time sequence feature vector corresponding to each time point.

[0088] The feature splicing module 400 is configured to splice the basic feature vector and the time sequence feature vector of each time point on the main time axis to obtain a joint feature vector of each entity at each time point on the main time axis.

[0089] The feature importance evaluation module 500 is configured to construct a feature space with all joint feature vectors of all entities, take the sample label set as a prediction target, and calculate the correlation strength between each feature and the prediction target.

[0090] The model training and prediction analysis module 600 is configured to filter a preset number of features from the feature space to form a core feature subset according to the correlation strength ranking, input the core feature subset and the corresponding sample label into a prediction model for training, and output a data analysis result using the trained prediction model.

[0091] The working principle of the computer data analysis system based on big data provided by the application is as follows:

[0092] Firstly, the data standardization module 100 parses entities from the multi-source heterogeneous data, aligns these data to the predefined main time axis for standardization processing, and generates standard data records of each entity at each time point. Then, the basic feature extraction module 200 assigns a sample label to each standard data record based on the predefined business rules and generates a sample label set, and extracts a basic feature vector from each record. The time sequence feature generation module 300 arranges each feature dimension of each basic feature vector in the order of the main time axis, calculates the change trend index of each feature dimension through a preset sliding time window, and generates a time sequence feature vector corresponding to each time point. The feature splicing module 400 splices the basic feature vector and the time sequence feature vector at the same time point to form a joint feature vector with higher dimensions. The feature importance evaluation module 500 constructs a feature space with all joint feature vectors of all entities, takes the sample label set as a prediction target, and calculates the correlation strength between each feature and the prediction target. Finally, the model training and prediction analysis module 600 filters a preset number of features from the feature space to form a core feature subset according to the correlation strength ranking, inputs the subset and the corresponding sample label into a prediction model for training, and outputs a data analysis result using the trained prediction model, realizing an automated process from data preprocessing, feature engineering to model training and prediction, and ensuring the accuracy and efficiency of data analysis.

[0093] The computer program instructions can also be loaded onto a computer, other programmable data processing apparatus, or other processing devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions specified in the flowchart block or blocks. Figure 1 The flowchart and / or block diagram in the working examples illustrate the architecture, functionality, and operation of possible implementations of apparatuses (systems), methods and computer program products according to the present disclosure. In this regard, each flowchart and / or block diagram can represent a method, apparatus and / or computer program products according to the present disclosure. It will also be understood that each block of the flowchart and / or block diagrams and combinations of blocks in the flowchart and / or block diagrams can be implemented by computer program instructions such as program code. Figure 1 The flowchart and / or block diagram in the working examples illustrate the architecture, functionality, and operation of possible implementations of apparatuses (systems), methods and computer program products according to the present disclosure. In this regard, each flowchart and / or block diagram can represent a method, apparatus and / or computer program products according to the present disclosure. It will also be understood that each block of the flowchart and / or block diagrams and combinations of blocks in the flowchart and / or block diagrams can be implemented by computer program instructions such as program code.

[0094] Those skilled in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the relevant hardware through a program, and the program can be stored in a computer readable storage medium, including Read-Only Memory (ROM), Random Access Memory (RAM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), One-time Programmable Read-Only Memory (OTPROM), Electrically-Erasable Programmable Read-Only Memory (EEPROM), Compact Disc Read-Only Memory (CD-ROM) or other optical disk memories, magnetic disk memories, magnetic tape memories, or any other medium readable by a computer that can be used to carry or store data.

[0095] It should also be noted that the terms "comprising" "containing" or any other variant thereof are intended to cover a non-exclusive inclusion, such that a process, method, article or apparatus that comprises a list of elements does not only include those elements, but can also include other elements not expressly listed or inherent to such process, method, article or apparatus. Without further limitation, an element defined by the phrase "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

Claims

1. A big data based computer data analysis method, characterized by, The computer data analysis method comprises the following steps: S1: from the multi-source heterogeneous data, the entity is parsed, the multi-source heterogeneous data is aligned to the predefined main time axis and is standardized, and the standard data record of each entity at each time point on the main time axis is generated; S2: based on the predefined business rule, a sample label is assigned to each standard data record to generate a sample label set, and a basic feature vector is extracted from each standard data record; S3: for each feature dimension of each basic feature vector, the change trend index of each feature dimension is calculated through a preset sliding time window in the order of the main time axis, and a time sequence feature vector corresponding to each time point is generated; S4: the basic feature vector and the time sequence feature vector at each time point on the main time axis are spliced to obtain the joint feature vector of each entity at each time point on the main time axis; S5: all joint feature vectors of all entities constitute a feature space, and the sample label set is used as a prediction target to calculate the correlation strength between each feature and the prediction target; S6: according to the correlation strength, a preset number of features are selected from the feature space to form a core feature subset, and the core feature subset and the corresponding sample label are input into a prediction model for training, and the data analysis result is output by using the trained prediction model.

2. The computer data analysis method based on big data according to claim 1, characterized in that, In step S1, the predefined main time axis is composed of consecutive time points with a preset fixed time interval.

3. The computer data analysis method based on big data according to claim 2, characterized in that, In step S1, the generation of the standard data record comprises the following steps: According to the analysis requirement, the time interval between two consecutive time points on the main time axis is marked as a time bucket; The multi-source heterogeneous data is mapped to the corresponding time bucket according to the timestamp; For the data mapped to the same time bucket, an aggregation function is selected according to its business meaning to calculate, and a plurality of data is compressed into a summary record, i.e. a standard data record.

4. The computer data analysis method based on big data according to claim 3, characterized in that, In step S2, the assignment of the sample label comprises: Based on whether at least one field value in the standard data record meets a preset judgment condition, a discrete sample label representing a classification result is assigned.

5. The computer data analysis method based on big data according to claim 4, characterized in that, In step S3, the generation step of the time sequence feature vector comprises: Arranging the basic feature vectors in the order of the main time axis to form a basic feature vector sequence; For each feature dimension in the basic feature vector sequence, the numerical value at different time points is independently extracted to form a time sequence of the feature dimension; For each time point on the main time axis, the change trend index is calculated on the time sequence of the corresponding feature dimension based on a preset sliding time window; The values of the change trend indexes of all feature dimensions at the time point are combined to generate a time sequence feature vector corresponding to the time point.

6. The computer data analysis method based on big data according to claim 5, characterized in that, In step S4, the splicing is vector splicing, i.e. the basic feature vector and the time sequence feature vector are connected in dimension to obtain a joint feature vector with higher dimension.

7. The big data based computer data analysis method of claim 6, wherein, In step S5, the calculation step of the correlation strength comprises: Taking the feature space as an input feature matrix and the prediction target as a target vector, a feature importance evaluation model is trained; The feature importance evaluation algorithm built in the feature importance evaluation model is used to calculate the contribution score of each column of features in the feature matrix to the target vector; The contribution score of each feature is used as the association strength between the feature and the target vector.

8. The computer data analysis method based on big data according to claim 7, characterized in that, In step S6, the screening step of the core feature subset includes: Features in the feature space are arranged in descending order according to the association strength to generate a feature importance ranking list; The top pre-set number of features in the feature importance ranking list are selected to form a core feature subset, wherein the pre-set number is determined by elbow rule analysis of the inflection point on the association strength distribution curve corresponding to the feature importance ranking list.

9. The big data based computer data analysis method of claim 8, wherein, In step S6, the prediction model is a logistic regression model.

10. A big data based computer data analysis system, characterized by, The computer data analysis system is applied to the computer data analysis method based on big data according to any one of claims 1 to 9, and the computer data analysis system comprises: A data standardization module is configured to parse entities from multi-source heterogeneous data, align the multi-source heterogeneous data to a predefined main time axis, and perform standardization processing to generate standard data records of each entity at each time point on the main time axis; A basic feature extraction module is configured to assign a sample label to each standard data record based on predefined business rules to generate a sample label set, and extract a basic feature vector from each standard data record; A time series feature generation module is configured to calculate the change trend index of each feature dimension through a preset sliding time window in the order of the main time axis to generate a time series feature vector corresponding to each time point for each basic feature vector; A feature splicing module is configured to splice the basic feature vector and the time series feature vector at each time point on the main time axis to obtain a joint feature vector of each entity at each time point on the main time axis; A feature importance evaluation module is configured to use all joint feature vectors of all entities to form a feature space, and use the sample label set as a prediction target to calculate the association strength between each feature and the prediction target; A model training and prediction analysis module is configured to screen a pre-set number of features from the feature space to form a core feature subset according to the association strength ranking, input the core feature subset and the corresponding sample label into the prediction model for training, and output a data analysis result using the trained prediction model.

Images