User data analysis method based on big data

Abstract

The invention discloses a user data analysis method based on big data, and relates to the field of data analysis. Comprising the following steps: data acquisition: acquiring multi-source data information of user behavior logs, transaction records and social media interaction from big data; data preprocessing: performing data cleaning and data standardization processing on the acquired multivariate data information; multivariate data fusion: fusing the preprocessed multivariate data; user portrait construction: constructing a user portrait by adopting a dynamic user portrait modeling mode; user classification: using an improved k-means method to classify users for the constructed user portraits; and predicting user behaviors. According to the method, firstly, the multi-dimensional data are acquired, then the data are preprocessed, then multi-dimensional data fusion is performed, and then user portrait establishment and user classification are performed, so that user data analysis under multiple dimensions is realized, and prediction of user behaviors can be realized through a gating mechanism and an attention mechanism of a machine learning algorithm.

Description

Technical Field

[0001] The present invention relates to the field of data analysis, and in particular to a user data analysis method based on big data. Background Art

[0002] With the rapid development of big data technology, enterprises have accumulated a large amount of user data. How to effectively use this data and tap the value behind it has become the key to improving the competitiveness of enterprises. Existing user data analysis methods mostly focus on a single dimension or model, which makes it difficult to fully and deeply reflect user behavior and needs.

[0003] After searching, the Chinese patent announcement number CN118210921A discloses a user data analysis method and device based on big data, including: Step 1: Collect UGC data from various social media platforms, news websites and forums through automated means, Step 2: Clean and organize the collected UGC data, Step 3: Use natural language processing method NLP to extract keywords, sentiment tendency features and topic features from UGC data, Step 4: Use machine learning or deep learning methods to train according to keywords, sentiment tendency features and topic features to build a user data analysis model, Step 5: Use the user data analysis model to classify, sentimentally analyze and extract topics for new UGC data to obtain user data tendencies and hot topic information.

[0004] The above patent has the following shortcomings: the dimensions of data acquisition and data analysis are single, making it difficult to fully and deeply reflect user behavior and needs.

[0005] To this end, the present invention proposes a user data analysis method based on big data Summary of the invention

[0006] The purpose of the present invention is to solve the shortcomings of the prior art and to propose a user data analysis method based on big data.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions: A user data analysis method based on big data comprises the following steps: S1: Data collection, obtaining multi-source data information such as user behavior logs, transaction records, and social media interactions from big data; S2: Data preprocessing, data cleaning and data standardization of the acquired multi-dimensional data information; S3: Multivariate data fusion, fusing the preprocessed multivariate data; S4: User portrait construction, using dynamic user portrait modeling to build user portraits; S5: User classification: using the improved k-means method to classify users based on the constructed user portraits; S6: User behavior prediction, based on historical data, uses machine learning algorithms to build a prediction model to predict user behavior.

[0008] Preferably: in the step S2, the missing values ​​in data cleaning are supplemented by weighted mean, median, mode or sliding window linear interpolation method.

[0009] Preferably: in the step S2, the data standardization model is , where X is the data before standardization, is the standardized data, is the median of the data before standardization, is the absolute median difference.

[0010] Preferably: in the step S3, the method for data fusion comprises the following steps: S31: Timing alignment, its model is: , where i and j are data of different dimensions, is the minimum cumulative distance between the first i points of sequence X and the first j points of Y; S32: Data fusion, the fusion model is: , where A is the fused data, a is the data after time series alignment in different dimensions, is the weight coefficient.

[0011] Preferably: the step S4 comprises the following steps: S41: First, classify the collected user information according to its type , and provide a weight for each category ,and ; S42: Match each type through big data User behavior characteristics under , and for each Provide a behavior frequency ; S43: For user behavior characteristics , the user's matching degree is , and use this to build a multi-dimensional portrait of the user.

[0012] Preferably: in the step S5, the classification model is , J is the clustering objective function, k is the number of clusters, is the i-th cluster, x is the sample point in the cluster, is the centroid of the ith cluster.

[0013] Preferably: in the step S5, the centroid of the first cluster is randomly selected, and the centroid selection probability model of subsequent clusters is: , where P(x) is the probability that data point x is selected, D(x) is the distance function, which represents the Euclidean distance from data point x to the nearest selected center point, and X represents the data set, which contains the full set of all data to be processed.

[0014] Preferably: in the step S6, the user behavior prediction adopts a bidirectional LSTM model, which adopts a gating mechanism and an attention mechanism.

[0015] Preferably: the model of the gating mechanism is: , is the forget gate, which controls the cell state at the previous moment The retention ratio ranges from (0 to 1). , is the input gate, which controls the candidate state The update ratio of , is the candidate cell state, which is based on the current input and apologize hidden state Generate a new state and use tanh activation with a value range of (-1, 1); , is the current cell state; , is the output gate, which controls the current cell state The proportion of external exports; , is the current hidden state; is the weight matrix, connecting the input and the parameter matrix of each gate; is the sigmoid function, ; is the hadamard product.

[0016] Preferably: the model of the attention mechanism is: , ,in: is the attention weight at time step t: it indicates the importance of the t-th input to the current output, and its value range is (0, 1); is the alignment score: how well the encoder hidden state matches the decoder state; is a trainable parameter vector: maps the alignment score to a scalar; is the encoder weight matrix: for the encoder hidden state Linear transformation of ; Decoder weight matrix: for the previous state of the decoder Linear transformation of .

[0017] The beneficial effects of the present invention are: 1. The present invention first acquires multidimensional data, then pre-processes the data and then performs multidimensional data fusion, and then establishes user portraits and classifies users, thereby realizing user data analysis in multiple dimensions, and can also predict user behavior through the gating mechanism and attention mechanism of the machine learning algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is a flow chart of a user data analysis method based on big data proposed by the present invention. DETAILED DESCRIPTION

[0019] The technical solution of the present invention is further described in detail below in conjunction with specific implementation methods.

[0020] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", "connected", and "set" should be understood in a broad sense, for example, they can be fixedly connected or set, or detachably connected or set, or integrally connected or set. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0021] Embodiment 1: A user data analysis method based on big data, comprising the following steps:

[0022] S1: Data collection, obtaining multi-source data information such as user behavior logs, transaction records, and social media interactions from big data; S2: Data preprocessing, data cleaning and data standardization of the acquired multi-dimensional data information; S3: Multivariate data fusion, fusing the preprocessed multivariate data; S4: User portrait construction, using dynamic user portrait modeling to build user portraits; S5: User classification: using the improved k-means method to classify users based on the constructed user portraits; S6: User behavior prediction, based on historical data, uses machine learning algorithms to build a prediction model to predict user behavior.

[0023] Embodiment 2: A user data analysis method based on big data, comprising the following steps:

[0024] S1: Data collection, obtaining multi-source data information such as user behavior logs, transaction records, and social media interactions from big data; S2: Data preprocessing, data cleaning and data standardization of the acquired multi-dimensional data information; S3: Multivariate data fusion, fusing the preprocessed multivariate data; S4: User portrait construction, using dynamic user portrait modeling to build user portraits; S5: User classification: using the improved k-means method to classify users based on the constructed user portraits; S6: User behavior prediction, based on historical data, uses machine learning algorithms to build a prediction model to predict user behavior.

[0025] In the step S2, the missing values ​​in data cleaning are supplemented by weighted mean, median, mode or sliding window linear interpolation method.

[0026] In the S2 step, the data standardization model is: , where X is the data before standardization, is the standardized data, is the median of the data before standardization, is the absolute median difference.

[0027] Embodiment 3: A user data analysis method based on big data, comprising the following steps: S1: Data collection, obtaining multi-source data information such as user behavior logs, transaction records, and social media interactions from big data; S2: Data preprocessing, data cleaning and data standardization of the acquired multi-dimensional data information; S3: Multivariate data fusion, fusing the preprocessed multivariate data; S4: User portrait construction, using dynamic user portrait modeling to build user portraits; S5: User classification: using the improved k-means method to classify users based on the constructed user portraits; S6: User behavior prediction, based on historical data, uses machine learning algorithms to build a prediction model to predict user behavior.

[0028] In the step S2, the missing values ​​in data cleaning are supplemented by weighted mean, median, mode or sliding window linear interpolation method.

[0029] In the S2 step, the data standardization model is: , where X is the data before standardization, is the standardized data, is the median of the data before standardization, is the absolute median difference.

[0030] In the step S3, the data fusion method includes the following steps: S31: Timing alignment, its model is: , where i and j are data of different dimensions, is the minimum cumulative distance between the first i points of sequence X and the first j points of Y; S32: Data fusion, the fusion model is: , where A is the fused data, a is the data after time series alignment in different dimensions, is the weight coefficient.

[0031] Embodiment 4: A user data analysis method based on big data, comprising the following steps: S1: Data collection, obtaining multi-source data information such as user behavior logs, transaction records, and social media interactions from big data; S2: Data preprocessing, data cleaning and data standardization of the acquired multi-dimensional data information; S3: Multivariate data fusion, fusing the preprocessed multivariate data; S4: User portrait construction, using dynamic user portrait modeling to build user portraits; S5: User classification: using the improved k-means method to classify users based on the constructed user portraits; S6: User behavior prediction, based on historical data, uses machine learning algorithms to build a prediction model to predict user behavior.

[0032] In the step S2, the missing values ​​in data cleaning are supplemented by weighted mean, median, mode or sliding window linear interpolation method.

[0033] In the S2 step, the data standardization model is: , where X is the data before standardization, is the standardized data, is the median of the data before standardization, is the absolute median difference.

[0034] In the step S3, the data fusion method includes the following steps: S31: Timing alignment, its model is: , where i and j are data of different dimensions, is the minimum cumulative distance between the first i points of sequence X and the first j points of Y; S32: Data fusion, the fusion model is: , where A is the fused data, a is the data after time series alignment in different dimensions, is the weight coefficient.

[0035] The step S4 comprises the following steps: S41: First, classify the collected user information according to its type , and provide a weight for each category ,and ; S42: Match each type through big data User behavior characteristics under , and for each Provide a behavior frequency ; S43: For user behavior characteristics , the user's matching degree is , and use this to build a multi-dimensional portrait of the user.

[0036] Embodiment 5: A user data analysis method based on big data, comprising the following steps: S1: Data collection, obtaining multi-source data information such as user behavior logs, transaction records, and social media interactions from big data; S2: Data preprocessing, data cleaning and data standardization of the acquired multi-dimensional data information; S3: Multivariate data fusion, fusing the preprocessed multivariate data; S4: User portrait construction, using dynamic user portrait modeling to build user portraits; S5: User classification: using the improved k-means method to classify users based on the constructed user portraits; S6: User behavior prediction, based on historical data, uses machine learning algorithms to build a prediction model to predict user behavior.

[0037] In the step S2, the missing values ​​in data cleaning are supplemented by weighted mean, median, mode or sliding window linear interpolation method.

[0038] In the S2 step, the data standardization model is: , where X is the data before standardization, is the standardized data, is the median of the data before standardization, is the absolute median difference.

[0039] In the step S3, the data fusion method includes the following steps: S31: Timing alignment, its model is: , where i and j are data of different dimensions, is the minimum cumulative distance between the first i points of sequence X and the first j points of Y; S32: Data fusion, the fusion model is: , where A is the fused data, a is the data after time series alignment in different dimensions, is the weight coefficient.

[0040] The step S4 comprises the following steps: S41: First, classify the collected user information according to its type , and provide a weight for each category ,and ; S42: Match each type through big data User behavior characteristics under , and for each Provide a behavior frequency ; S43: For user behavior characteristics , the user's matching degree is , and use this to build a multi-dimensional portrait of the user.

[0041] In step S5, the classification model is , J is the clustering objective function, k is the number of clusters, is the i-th cluster, x is the sample point in the cluster, is the centroid of the ith cluster.

[0042] In step S5, the centroid of the first cluster is randomly selected, and the centroid selection probability model of subsequent clusters is: , where P(x) is the probability that data point x is selected, D(x) is the distance function, which represents the Euclidean distance from data point x to the nearest selected center point, and X represents the data set, which contains the full set of all data to be processed.

[0043] Embodiment 6: A user data analysis method based on big data, comprising the following steps: S1: Data collection, obtaining multi-source data information such as user behavior logs, transaction records, and social media interactions from big data; S2: Data preprocessing, data cleaning and data standardization of the acquired multi-dimensional data information; S3: Multivariate data fusion, fusing the preprocessed multivariate data; S4: User portrait construction, using dynamic user portrait modeling to build user portraits; S5: User classification: using the improved k-means method to classify users based on the constructed user portraits; S6: User behavior prediction, based on historical data, uses machine learning algorithms to build a prediction model to predict user behavior.

[0044] In the step S2, the missing values ​​in data cleaning are supplemented by weighted mean, median, mode or sliding window linear interpolation method.

[0045] In the S2 step, the data standardization model is: , where X is the data before standardization, is the standardized data, is the median of the data before standardization, is the absolute median difference.

[0046] In the step S3, the data fusion method includes the following steps: S31: Timing alignment, its model is: , where i and j are data of different dimensions, is the minimum cumulative distance between the first i points of sequence X and the first j points of Y; S32: Data fusion, the fusion model is: , where A is the fused data, a is the data after time series alignment in different dimensions, is the weight coefficient.

[0047] The step S4 comprises the following steps: S41: First, classify the collected user information according to its type , and provide a weight for each category ,and ; S42: Match each type through big data User behavior characteristics under , and for each Provide a behavior frequency ; S43: For user behavior characteristics , the user's matching degree is , and use this to build a multi-dimensional portrait of the user.

[0048] In step S5, the classification model is , J is the clustering objective function, k is the number of clusters, is the i-th cluster, x is the sample point in the cluster, is the centroid of the ith cluster.

[0049] In step S5, the centroid of the first cluster is randomly selected, and the centroid selection probability model of subsequent clusters is: , where P(x) is the probability that data point x is selected, D(x) is the distance function, which represents the Euclidean distance from data point x to the nearest selected center point, and X represents the data set, which contains the full set of all data to be processed.

[0050] In step S6, user behavior prediction adopts a bidirectional LSTM model, which adopts a gating mechanism and an attention mechanism.

[0051] The model of the gating mechanism is: , is the forget gate, which controls the cell state at the previous moment The retention ratio ranges from (0 to 1). , is the input gate, which controls the candidate state The update ratio of , is the candidate cell state, which is based on the current input and apologize hidden state Generate a new state and use tanh activation with a value range of (-1, 1); , is the current cell state; , is the output gate, which controls the current cell state The proportion of external exports; , is the current hidden state; is the weight matrix, connecting the input and the parameter matrix of each gate; is the sigmoid function, ; is the hadamard product.

[0052] The model of the attention mechanism is: , ,in: is the attention weight at time step t: it indicates the importance of the t-th input to the current output, and its value range is (0, 1); is the alignment score: how well the encoder hidden state matches the decoder state; is a trainable parameter vector: maps the alignment score to a scalar; is the encoder weight matrix: for the encoder hidden state Linear transformation of ; Decoder weight matrix: for the previous state of the decoder Linear transformation of .

[0053] The present invention first acquires multidimensional data, then pre-processes the data and then performs multidimensional data fusion, and then establishes user portraits and classifies users, thereby realizing user data analysis in multiple dimensions, and can also predict user behavior through the gating mechanism and attention mechanism of the machine learning algorithm.

[0054] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A user data analysis method based on big data, characterized in that: The following steps are involved: S1: Data collection, obtaining multi-source data information such as user behavior logs, transaction records, and social media interactions from big data; S2: Data preprocessing, data cleaning and data standardization of the acquired multi-dimensional data information; S3: Multivariate data fusion, fusing the preprocessed multivariate data; S4: User portrait construction, using dynamic user portrait modeling to build user portraits; S5: User classification: using the improved k-means method to classify users based on the constructed user portraits; S6: User behavior prediction, based on historical data, uses machine learning algorithms to build a prediction model to predict user behavior.

2. The user data analysis method based on big data according to claim 1, characterized in that: In the step S2, the missing values ​​in data cleaning are supplemented by weighted mean, median, mode or sliding window linear interpolation method.

3. The user data analysis method based on big data according to claim 1 is characterized in that: In the S2 step, the data standardization model is: , where X is the data before standardization, is the standardized data, is the median of the data before standardization, is the absolute median difference.

4. The user data analysis method based on big data according to claim 1 is characterized in that: In the step S3, the data fusion method includes the following steps: S31: Timing alignment, its model is: , where i and j are data of different dimensions, is the minimum cumulative distance between the first i points of sequence X and the first j points of Y; S32: Data fusion, the fusion model is: , where A is the fused data, a is the data after time series alignment in different dimensions, is the weight coefficient.

5. The user data analysis method based on big data according to claim 1 is characterized in that: The step S4 comprises the following steps: S41: First, classify the collected user information according to its type , and provide a weight for each category ,and ; S42: Match each type through big data User behavior characteristics under , and for each Provide a behavior frequency ; S43: For user behavior characteristics , the user's matching degree is , and use this to build a multi-dimensional portrait of the user.

6. The user data analysis method based on big data according to claim 1, characterized in that: In step S5, the classification model is , J is the clustering objective function, k is the number of clusters, is the i-th cluster, x is the sample point in the cluster, is the centroid of the ith cluster.

7. The user data analysis method based on big data according to claim 6 is characterized in that: In step S5, the centroid of the first cluster is randomly selected, and the centroid selection probability model of subsequent clusters is: , where P(x) is the probability that data point x is selected, D(x) is the distance function, which represents the Euclidean distance from data point x to the nearest selected center point, and X represents the data set, which contains the full set of all data to be processed.

8. The user data analysis method based on big data according to claim 1, characterized in that: In step S6, user behavior prediction adopts a bidirectional LSTM model, which adopts a gating mechanism and an attention mechanism.

9. The user data analysis method based on big data according to claim 8, characterized in that: The model of the gating mechanism is: , is the forget gate, which controls the cell state at the previous moment The retention ratio ranges from (0 to 1). , is the input gate, which controls the candidate state The update ratio of , is the input gate, which controls the candidate state The update ratio of , is the candidate cell state, which is based on the current input and apologize hidden state Generate a new state and use tanh activation with a value range of (-1, 1); , is the current cell state; , is the output gate, which controls the current cell state The proportion of external exports; , is the current hidden state; is the weight matrix, connecting the input and the parameter matrix of each gate; is the sigmoid function, ; is the hadamard product.

10. The user data analysis method based on big data according to claim 8, characterized in that: The model of the attention mechanism is: , ,in: is the attention weight at time step t: it indicates the importance of the t-th input to the current output, and its value range is (0, 1); is the alignment score: how well the encoder hidden state matches the decoder state; is a trainable parameter vector: maps the alignment score to a scalar; is the encoder weight matrix: for the encoder hidden state Linear transformation of ; Decoder weight matrix: for the previous state of the decoder Linear transformation of .

Images