Distributed Collaborative Storage System and Method for Marketing Data Based on Machine Learning

Abstract

The present invention relates to the technical field of data storage, and discloses a distributed collaborative storage system and method for marketing data based on machine learning. First, the present invention performs clustering and classification processing on the initial marketing data to obtain a classified marketing data set; secondly, based on the isolation forest algorithm, an unqualified data recognition model is established according to the classified marketing data set, and the unqualified data is identified and deleted to obtain a processed marketing data set; then the processed marketing data set is subjected to distributed collaborative storage, and a distributed collaborative storage model is established with the maximum load as the goal; finally, the multi-strategy fusion pelican optimization algorithm is used to solve the distributed collaborative storage model to obtain the global optimal solution and obtain the distributed collaborative storage strategy. The present invention realizes the purpose of distributed collaborative storage of marketing data by processing and storing marketing data, and the method is accurate and objective.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage, and specifically to a distributed collaborative storage system and method for marketing data based on machine learning. Background Art

[0002] Chinese Patent CN118331791B discloses a data collaborative backup system based on cloud computing and distributed storage. The system specifically includes a data preprocessing module, a distributed storage module, a collaborative management module, and a data recovery module. The interaction process of each module is as follows: Obtain data and transmit it to the data preprocessing module, calculate the weight of the received data to obtain weight information, generate a backup strategy based on the weight information, and then transmit the backup strategy to the distributed storage module; The distributed storage module distributes and stores the backup strategy. The data recovery module is connected to the distributed storage module and is used to receive a recovery instruction sent by the collaborative management module; The collaborative management module is connected to other modules and monitors the module status and issues instructions to the staff. However, this invention does not eliminate data redundancy, resulting in slow storage speed and long processing time.

[0003] Traditional data distributed collaborative storage methods usually store data in a data storage center without further organizing the data, ignoring the availability and integrity of the data; At the same time, due to the lack of technologies such as machine learning, relevant models are not established during the distributed collaborative storage of marketing data, resulting in the inability to quickly and effectively provide decision-making support for the marketing team. Summary of the Invention

[0004] In view of the problems in the related art, the present invention provides a distributed collaborative storage system and method for marketing data based on machine learning to overcome the above-mentioned technical problems existing in the existing related technologies.

[0005] To solve the above technical problems, the present invention is realized through the following technical solutions:

[0006] The present invention is a distributed collaborative storage method for marketing data based on machine learning, including the following steps:

[0007] S1. Obtain marketing-related data to obtain an initial marketing data set, use an improved density clustering algorithm to cluster and organize the initial marketing data set, and then train an XgBoost classification model for classification processing to obtain a classified marketing data set;

[0008] S2. Establish an outlier data recognition model based on the isolation forest algorithm, use the outlier data recognition model to detect outlier data in the classified marketing data set, and delete the outlier data to obtain a processed marketing data set;

[0009] S3. The processed marketing data set uses a multi - fork tree strategy for distributed collaborative storage. Taking the maximum load as the goal, a storage maximum load function is established, and constraint conditions are set to obtain a distributed collaborative storage model;

[0010] S4. Use the pelican optimization algorithm with multi - strategy fusion to solve the distributed collaborative storage model, obtain the global optimal solution, determine the storage nodes and storage devices for the processed marketing data set, generate a distributed collaborative storage strategy, and complete the distributed collaborative storage of marketing data.

[0011] The invention improves the accuracy and effectiveness of clustering and reduces the calculation time by clustering and classifying the initial marketing data. When clustering, the data is divided according to the sparsity degree, and sparse data and invalid data sets with a large number of missing data are deleted. After clustering, it is input into the XGBoost classification model, which can effectively improve the classification accuracy of the model and enhance the generalization ability of the model compared with direct input. Secondly, an unqualified data recognition model is established based on the isolation forest algorithm, and unqualified data is recognized by calculating the anomaly scores of the data on the isolation tree in the unqualified data recognition model. This method has good resistance to missing data, higher detection speed and calculation accuracy compared with traditional anomaly recognition methods, and is suitable for the processing of large - scale data. Then, the processed marketing data is stored in a distributed and collaborative manner. Taking the maximum load as the goal, a distributed collaborative storage model is established. This model can avoid the delay problem in the data storage process, solve the problem of heavy load in the distributed storage process, and facilitate the visualization of the data storage process. Finally, use the pelican optimization algorithm with multi - strategy fusion to solve the distributed collaborative storage model. By simulating the behavior of pelicans during hunting, the global optimal solution is obtained, and a distributed collaborative storage strategy is generated. Compared with traditional algorithms, this algorithm reduces the probability of obtaining local optimal solutions and increases the global exploration ability by introducing the Cauchy mutation strategy, and integrates the sparrow early warning mechanism, greatly improving the convergence speed.

[0012] Preferably, S1 includes the following steps:

[0013] S11. Obtain marketing - related data from different market sales channels. The marketing - related data includes sales volume, market growth rate, market share, etc. The marketing - related data of each market sales channel forms a marketing data subset, and all marketing data subsets generate an initial marketing data set. Use an improved density - based spatial clustering of applications with noise (DBSCAN) algorithm to cluster and organize the initial marketing data set to obtain an organized marketing data set. The specific steps are as follows:

[0014] S111. Project the initial marketing data set into a three-dimensional space. Treat the initial marketing data in the initial marketing data set as initial marketing data points. Select any initial marketing data point and denote it as the first marketing data point. Set a standard data point, calculate the standard data point feature vector and the first marketing data point feature vector, and calculate the distance between the feature vectors. The calculation formula is as follows:

[0015] ;

[0016] where l represents the distance between the standard data point feature vector and the first marketing data point feature vector, represents the coordinates of the standard data point feature vector, represents the coordinates of the first marketing data point feature vector;

[0017] Select the second marketing data point in sequence until all initial marketing data points are traversed to obtain a feature vector distance array. Sort the feature vector distance array in ascending order, and count the number of initial marketing data points corresponding to the standard data point distance to obtain a distance count array , where represents the corresponding number when the distance of the feature vector is ;

[0018] S112. According to the distance count array, set a critical point. Classify the initial marketing data points corresponding to the distance count less than the critical point in the distance count array into the sparse data point set, classify the initial marketing data points corresponding to the distance count greater than the critical point in the distance count array into the dense data point set, and denote the initial marketing data points corresponding to the distance count equal to the critical point as swing data points; perform clustering processing on the sparse data point set and the dense data point set to obtain the first central data point and the second central pixel point respectively. Calculate the similarity between the swing data points and the first central data point and the second central pixel point, compare the similarity sizes, classify the swing data points into the data point set with the larger similarity, then delete the sparse data point set, and perform clustering processing again to obtain several clustering clusters, forming the sorted marketing data set to complete the clustering sorting;

[0019] S12. Re-obtain the marketing-related data of previous years to form a previous-year marketing data set. After performing clustering sorting on the previous-year marketing data set, obtain a marketing sample data set. Extract training samples from the marketing sample data set to obtain m training sample sets; Set that the XGBoost algorithm contains m weak classifiers, and input the m training sample sets correspondingly intom It is trained among m weak classifiers until convergence to obtain m trained weak classifiers; The

[0020] trained weak classifiers are combined to form an XGBoost classification model. The sorted marketing data set is input into the XGBoost classification model. Each trained weak classifier in the XGBoost classification model outputs a score, and all the scores are added together to obtain the final output result. According to the final output result, the sorted marketing data set is classified to obtain a classified marketing data set. The classified marketing data set contains several classified marketing data subsets, and the classified marketing data subsets correspond to marketing-related data.

[0021] Preferably, S2 includes the following steps:

[0022] S21. Select any classified marketing data subset from the classified marketing data set and denote it as the marketing data sample set to be processed , where represents the n th marketing data sample to be processed. Set the maximum data dimension and the minimum data dimension of the marketing data sample set to be processed, and select any data dimension between the maximum data dimension and the minimum data dimension and denote it as the tree splitting point; Generate a hyperplane on the dimension where the tree splitting point is located to obtain the first data sample space, which is the first layer of the isolation tree. Add the marketing data samples to be processed in the marketing data sample set to be processed that are less than the corresponding tree splitting point to the left subtree of the tree splitting point, and add the marketing data samples to be processed in the marketing data sample set to be processed that are greater than or equal to the corresponding tree splitting point to the right subtree of the tree splitting point to complete the first-layer splitting of the isolation tree; Further, the left subtree of the tree splitting point and the right subtree of the tree splitting point generate the second data sample space, and continue to iterate to split the left subtree of the tree splitting point and the right subtree of the tree splitting point until the maximum depth of the isolation tree is reached to obtain the first isolation tree;

[0023] S22. Generate several isolation trees based on other classified marketing data subsets in the classified marketing data set. Stop the iteration of the isolation forest to obtain an unqualified data recognition model. Use the unqualified data recognition model to detect unqualified data in the classified marketing data set. The specific steps are as follows:

[0024] S221. The unqualified data recognition model includes several isolation trees. Traverse all the isolation trees, count the depth of the marketing data samples to be processed in the isolation trees, and calculate the average depth of the marketing data samples to be processed in the isolation trees.

[0025] S222. Set the depth function as , and calculate the outlier of the marketing data samples to be processed in the isolation tree. The calculation formula is as follows:

[0026] ;

[0027] Among them, represents the outlier of the e th marketing data sample to be processed in the isolation tree, represents the average depth of the marketing data samples to be processed in the isolation tree, represents the depth of the e th marketing data sample to be processed in the isolation tree, represents the depth function of the e th marketing data sample to be processed;

[0028] When , , at this time, regard the e th marketing data sample to be processed in the isolation tree as unqualified data, otherwise regard the e th marketing data sample to be processed in the isolation tree as qualified data;

[0029] S23. Traverse all the isolation trees in the unqualified data recognition model, calculate the outlier of the classified marketing data set, identify all unqualified data, and delete the unqualified data to obtain a processed marketing data set.

[0030] The present invention realizes the recognition of unqualified data by using the isolation forest algorithm to establish an unqualified data recognition model and calculating the outlier scores of the data on the isolation trees in the unqualified data recognition model. This method has good resistance to missing data, and has higher detection speed and calculation accuracy compared with traditional outlier recognition methods, and is suitable for the processing of large-scale data.

[0031] Preferably, the S3 includes the following steps:

[0032] S31. Use the multi - fork tree strategy to set that there is an empty data storage tree in the storage space. The storage nodes of the data storage tree are denoted as , where represents the i th storage node. Each storage node contains j storage devices. Input the processed marketing data set into the empty data storage tree in sequence to establish a data storage tree. Record the size of the processed marketing data received by each storage node, denoted as the storage node load. Then, perform distributed collaborative storage on the processed marketing data on the storage nodes, store the processed marketing data in the storage devices under the storage nodes, and calculate the selection coefficient of each processed marketing data stored in the storage device. Taking the maximum load of the data storage tree as the goal, establish a storage maximum load function, and the calculation formula is as follows:

[0033] ;

[0034] Among them, represents the storage maximum load function, represents the size of the g th processed marketing data, represents the selection coefficient of the g th processed marketing data on the h th storage node, represents the selection coefficient of the h th storage node on the k th storage device, represents the number of processed marketing data, represents the number of storage devices, , , ;

[0035] S32. Set model constraint conditions according to the storage maximum load function. Set that the storage capacity of the storage device is less than or equal to the storage device capacity. When , it means that the g th processed marketing data is stored on the h th storage node. When , it means that the g th processed marketing data is not stored on the h th storage node. represents the h th storage node on the th storage device. Combining the model constraint conditions and the storage maximum load function, a distributed collaborative storage model is obtained.

[0036] The invention solves the problem of heavy load in the distributed storage process by performing distributed collaborative storage on the processed marketing data, taking the maximum load as the target, establishing a distributed collaborative storage model, which can avoid the delay problem in the data storage process and facilitate the visualization of the data storage process.

[0037] Preferably, the S4 includes the following steps:

[0038] S41. Take the storage maximum load function as the fitness function, introduce the Cauchy mutation strategy and the sparrow early warning mechanism, improve the pelican optimization algorithm to obtain a multi-strategy fusion pelican optimization algorithm, and use the multi-strategy fusion pelican optimization algorithm to solve the distributed collaborative storage model to obtain the global optimal solution. The specific steps are as follows:

[0039] S411. Regard the data storage tree as the algorithm search space. Assume that there is a pelican population in the algorithm search space, and each pelican individual in the pelican population is regarded as the processed marketing data; the dimension of the pelican population is l , represents the upper bound of the pelican population, represents the lower bound of the pelican population, represents a random number between the interval [0, 1]. The initial position of the p th pelican individual in the pelican population in the q th dimension is ; In the exploration stage of the pelican population, assume that the current iteration number is t , the position of the t th iteration of the p th pelican individual in the pelican population in the q th dimension is denoted as , the prey position is , [[ID=3б]] represents a random number of 1 or 2;

[0040] [[ID=4ο]]Calculate the fitness function value of the prey position, denoted as , calculate the fitness function value of the current iteration pelican individual, denoted as [[ID=4φ]]; When , , when , ; Update to obtain the current best fitness function value, and the current best fitness function value corresponds to the optimal solution of the current distributed collaborative storage model;

[0041] S412. After the pelican population completes the exploration stage and reaches the water surface, the pelican population enters the exploitation stage. Assume that the maximum number of iterations is T , calculate the pThe fitness function value of each pelican individual and the average fitness function value of the pelican population are respectively denoted as and ; when , adopt

[0042] the traditional pelican individual position update formula to obtain the position of the t +1-th iteration of the p -th pelican individual in the q -th dimension ; when , at this time, introduce the Cauchy mutation strategy, set the standard Cauchy distribution as , the position of the best pelican individual in the t -th pelican population is denoted as , then ;

[0043] Integrate the sparrow early warning mechanism. When the pelican population encounters danger, the pelican population transfers to a safe area. Set to represent a normal distribution random number with a mean of 0 and a variance of 1, to represent a constant. The position of the worst pelican individual in the t -th pelican population is denoted as , update the position , and the update formula is as follows:

[0044] ;

[0045] Among them, represents the fitness function value corresponding to the position , represents the fitness function value corresponding to the position ;

[0046] S413. After the pelican population completes the exploitation stage, update the state of the pelican population and the current best fitness function value. At this time, enter the next iteration. When the current iteration number reaches the maximum iteration number, stop the iteration to obtain the global best position, and the global best position corresponds to the global optimal solution of the distributed collaborative storage model;

[0047] S42. Use the global optimal solution to solve the distributed collaborative storage model to obtain the maximum storage load that satisfies the model constraint conditions, obtain the final state of the pelican population at the maximum iteration number, migrate the positions of the pelican individuals in the pelican population at the maximum iteration number to the data storage tree, determine the positions of the processed marketing data in the processed marketing data set in the storage nodes and storage devices, generate a distributed collaborative storage strategy, and complete the distributed collaborative storage of marketing data.

[0048] The invention solves the distributed collaborative storage model by using a multi-strategy fusion pelican optimization algorithm. By simulating the behavior of pelicans during hunting, the global optimal solution is obtained, and a distributed collaborative storage strategy is generated. Compared with traditional algorithms, by introducing the Cauchy mutation strategy, the probability of obtaining a local optimal solution is reduced, and the global exploration ability is increased. By integrating the sparrow early warning mechanism, the convergence speed is greatly improved.

[0049] This embodiment also discloses a system for the distributed collaborative storage method of marketing data based on machine learning, specifically including: a data clustering, sorting and classification module, an unqualified data identification and deletion module, a data distributed collaborative storage module, and a distributed collaborative storage strategy generation module;

[0050] The data clustering, sorting and classification module is used to cluster, sort and classify the initial marketing data set;

[0051] The unqualified data identification and deletion module is used to establish an unqualified data identification model to identify and delete unqualified data;

[0052] The data distributed collaborative storage module is used to establish a distributed collaborative storage model to perform distributed collaborative storage on the data;

[0053] The distributed collaborative storage strategy generation module is used to use an optimization algorithm to solve the distributed collaborative storage model and obtain a distributed collaborative storage strategy.

[0054] The present invention has the following beneficial effects:

[0055] 1. The invention improves the accuracy and effectiveness of clustering and reduces the calculation time by clustering, sorting and classifying the initial marketing data. Among them, by dividing the data according to the sparsity degree, sparse data is deleted, and at the same time, invalid data sets with a large amount of missing data are also deleted.

[0056] 2. After clustering, the invention inputs the data into the XGBoost classification model, which can effectively improve the classification accuracy of the model and enhance the generalization ability of the model compared with direct input.

[0057] 3. The invention establishes an unqualified data identification model by using the isolation forest algorithm. This method has good resistance to missing data and has higher detection speed and calculation accuracy compared with traditional anomaly identification methods, and is suitable for the processing of large-scale data.

[0058] 4. The invention solves the problem of heavy load during the distributed storage process by using the maximum load as the target for distributed collaborative storage, establishes a distributed collaborative storage model, can avoid the delay problem during data storage, and is convenient for describing the data storage process.

[0059] 5. The invention uses a pelican optimization algorithm with multi-strategy fusion to solve the distributed collaborative storage model, obtain the global optimal solution, and generate a distributed collaborative storage strategy. Compared with traditional algorithms, by introducing a Cauchy mutation strategy, the probability of obtaining a local optimal solution is reduced, and the global exploration ability is increased. By integrating the sparrow early warning mechanism, the convergence speed is greatly improved.

[0060] Of course, it is not necessary for any product implementing the present invention to achieve all the above-mentioned advantages simultaneously. Brief Description of the Drawings

[0061] In order to more clearly illustrate the technical solutions of the embodiments of the invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0062] Figure 1 It is a schematic flow diagram of the distributed collaborative storage of marketing data for the distributed collaborative storage system of marketing data based on machine learning provided by the present invention. Detailed Embodiments

[0063] The following will clearly and completely describe the technical solutions in the embodiments of the present invention with reference to the drawings in the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0064] In the description of the present invention, it should be understood that the terms "opening", "upper", "lower", "top", "middle", "inner", etc. indicating the orientation or position relationship are only for the convenience of describing the invention and simplifying the description, rather than indicating or implying that the components or elements referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation of the invention.

[0065] Embodiment 1

[0066] Please refer to Figure 1 , this embodiment discloses a method for distributed collaborative storage of marketing data based on machine learning, which specifically includes the following contents:

[0067] S1. Obtain marketing-related data to obtain an initial marketing data set, use an improved density clustering algorithm to cluster and sort the initial marketing data set, and then train an XgBoost classification model for classification processing to obtain a classified marketing data set;

[0068] The S1 includes the following steps:

[0069] S11. Obtain marketing-related data from different market sales channels. The marketing-related data includes sales volume, market growth rate, market share, etc. The marketing-related data of each market sales channel forms a marketing data subset, and all marketing data subsets generate an initial marketing data set. Use an improved density clustering algorithm to cluster and organize the initial marketing data set to obtain an organized marketing data set. The specific steps are as follows:

[0070] S111. Project the initial marketing data set into a three-dimensional space. Regard the initial marketing data in the initial marketing data set as initial marketing data points. Select any initial marketing data point and denote it as the first marketing data point. Set a standard data point, calculate the standard data point feature vector and the first marketing data point feature vector, and calculate the distance between the feature vectors. The calculation formula is as follows:

[0071] ;

[0072] where l represents the distance between the standard data point feature vector and the first marketing data point feature vector, represents the coordinates of the standard data point feature vector, represents the coordinates of the first marketing data point feature vector;

[0073] Select the second marketing data point in turn until all initial marketing data points are traversed to obtain a feature vector distance array. Sort the feature vector distance array in ascending order, and count the number of initial marketing data points corresponding to the distance from the standard data point to obtain a distance number array , where represents the number of corresponding ones when the distance of the feature vector is ;

[0074] S112. Set a critical point according to the distance number array. Classify the initial marketing data points with the number of distances less than the corresponding critical point in the distance number array into the sparse data point set, classify the initial marketing data points with the number of distances greater than the critical point in the distance number array into the dense data point set, and record the initial marketing data points with the number of distances equal to the critical point as swing data points. Perform clustering processing on the sparse data point set and the dense data point set to obtain the first central data point and the second central pixel point respectively. Calculate the similarity between the swing data points and the first central data point and the second central pixel point, compare the similarity sizes, classify the swing data points into the data point set with the larger similarity, delete the sparse data point set, and then perform clustering processing to obtain several clustering clusters, forming the sorted marketing data set to complete the clustering sorting.

[0075] S12. Re-obtain the marketing-related data of previous years to form the marketing data set of previous years. After performing clustering sorting on the marketing data set of previous years, obtain the marketing sample data set. Extract training samples from the marketing sample data set to obtain m training sample sets. Set that the XGBoost algorithm contains m weak classifiers, and input m training sample sets into m weak classifiers for training until convergence to obtain m trained weak classifiers. Combine m trained weak classifiers to form the XGBoost classification model. Input the sorted marketing data set into the XGBoost classification model. Each trained weak classifier in the XGBoost classification model outputs a score, add up all the scores to obtain the final output result, and classify the sorted marketing data set according to the final output result to obtain the classified marketing data set. The classified marketing data set contains several classified marketing data subsets, and the classified marketing data subsets correspond to marketing-related data.

[0076] S2. Establish an unqualified data recognition model based on the isolation forest algorithm, use the unqualified data recognition model to detect the unqualified data in the classified marketing data set, and delete the unqualified data to obtain the processed marketing data set.

[0077] The S2 includes the following steps:

[0078] S21. Select any classified marketing data subset in the classified marketing data set and denote it as the to-be-processed marketing data sample set , where represent the n th marketing data sample to be processed, set the maximum data dimension and the minimum data dimension of the marketing data sample set to be processed, select any data dimension between the maximum data dimension and the minimum data dimension, and denote it as the tree splitting point; generate a hyperplane on the dimension where the tree splitting point is located to obtain the first data sample space, which is the first layer of the isolation tree. Add the marketing data samples to be processed in the marketing data sample set to be processed that are less than the marketing data sample corresponding to the tree splitting point to the left subtree of the tree splitting point, and add the marketing data samples to be processed in the marketing data sample set to be processed that are greater than or equal to the marketing data sample corresponding to the tree splitting point to the right subtree of the tree splitting point to complete the first layer splitting of the isolation tree; further, the left subtree of the tree splitting point and the right subtree of the tree splitting point generate the second data sample space, and continue to iterate to split the left subtree of the tree splitting point and the right subtree of the tree splitting point until the maximum depth of the isolation tree is reached to obtain the first isolation tree;

[0079] S22. Generate a number of isolation trees according to other classified marketing data subsets in the classified marketing data set, stop the iteration of the isolation forest to obtain an unqualified data recognition model, and use the unqualified data recognition model to detect unqualified data in the classified marketing data set. The specific steps are as follows:

[0080] S221. The unqualified data recognition model contains a number of isolation trees. Traverse all the isolation trees, count the depth of the marketing data samples to be processed in the isolation trees, and calculate the average depth of the marketing data samples to be processed in the isolation trees;

[0081] S222. Set the depth function as , and calculate the outlier of the marketing data samples to be processed in the isolation tree. The calculation formula is as follows:

[0082] ;

[0083] where represents the outlier of the e th marketing data sample to be processed in the isolation tree, represents the average depth of the marketing data samples to be processed in the isolation tree, represents the depth of the e th marketing data sample to be processed in the isolation tree, represents the e th depth function of the marketing data samples to be processed;

[0084] When , , at this time, the eThe sample of marketing data to be processed is regarded as unqualified data, otherwise the e sample of marketing data to be processed is regarded as qualified data;

[0085] S23. Traverse all the isolation trees in the unqualified data recognition model, calculate the outliers of the classified marketing data set, identify all the unqualified data, and delete the unqualified data to obtain the processed marketing data set;

[0086] S3. The processed marketing data set uses the multi-way tree strategy for distributed collaborative storage. Taking the maximum load as the goal, establish a storage maximum load function and set constraints to obtain a distributed collaborative storage model;

[0087] S3 includes the following steps:

[0088] S31. Use the multi-way tree strategy. Assume that there is an empty data storage tree in the storage space. The storage nodes of the data storage tree are denoted as , where represents the i th storage node. Each storage node contains j storage devices. Input the processed marketing data set into the empty data storage tree in sequence to establish a data storage tree. Record the size of the processed marketing data received by each storage node, denoted as the storage node load. Then, perform distributed collaborative storage on the processed marketing data on the storage nodes, store the processed marketing data in the storage devices under the storage nodes, and calculate the selection coefficient of each processed marketing data stored in the storage device. Taking the maximum load of the data storage tree as the goal, establish a storage maximum load function, and the calculation formula is as follows:

[0089] ;

[0090] Among them, represents the storage maximum load function, represents the size of the g th processed marketing data, represents the selection coefficient of the g th processed marketing data on the h th storage node, represents the selection coefficient of the h th storage device on the k th storage node, represents the number of processed marketing data, represents the number of storage devices, , , ;

[0091] S32. Set the model constraint conditions according to the storage maximum load function, and set that the storage capacity of the storage device is less than or equal to the capacity of the storage device. When , it means that the g th processed marketing data is stored on the h th storage node. When , it means that the g th processed marketing data is not stored on the h th storage node. indicates the h th storage device on the th storage node. Combining the model constraint conditions and the storage maximum load function, a distributed collaborative storage model is obtained;

[0092] S4. Use the multi-strategy fusion pelican optimization algorithm to solve the distributed collaborative storage model, obtain the global optimal solution, determine the storage nodes and storage devices of the processed marketing data set, generate a distributed collaborative storage strategy, and complete the distributed collaborative storage of marketing data;

[0093] The S4 includes the following steps:

[0094] S41. Take the storage maximum load function as the fitness function, introduce the Cauchy mutation strategy and the sparrow early warning mechanism, improve the pelican optimization algorithm, obtain the multi-strategy fusion pelican optimization algorithm, and use the multi-strategy fusion pelican optimization algorithm to solve the distributed collaborative storage model to obtain the global optimal solution. The specific steps are as follows:

[0095] S411. Regard the data storage tree as the algorithm search space, and set that there is a pelican population in the algorithm search space. Each pelican individual in the pelican population is regarded as the processed marketing data; the dimension of the pelican population is l , represents the upper bound of the pelican population, represents the lower bound of the pelican population, represents a random number between the interval [0, 1]. The initial position of the p th pelican individual in the pelican population in the q th dimension is ; in the exploration stage of the pelican population, set the current iteration number to t , and the position of the t th iteration of the p th pelican individual in the pelican population in the q th dimension is denoted as , the prey position is , represents a random number of 1 or 2;

[0096] Calculate the fitness function value of the prey position, denoted as ; calculate the fitness function value of the current pelican individual in the current iteration, denoted as ; when , ; when , ; update to obtain the current best fitness function value, and the current best fitness function value corresponds to the optimal solution of the current distributed collaborative storage model;

[0097] S412. After the pelican population completes the reconnaissance stage and reaches the water surface, the pelican population enters the exploitation stage. Set the maximum number of iterations to T , and calculate the fitness function value of the p -th pelican individual in the pelican population during the exploitation stage and the average fitness function value of the pelican population, denoted as and respectively; when , use the traditional pelican individual position update formula to obtain the position of the

[0098] -th pelican individual in the pelican population at the t +1-th iteration in the p -th dimension q ; when , introduce the Cauchy mutation strategy at this time. Set the standard Cauchy distribution to , and denote the position of the best pelican individual in the -th pelican population as t , then ;

[0099] Fuse the sparrow early warning mechanism. When the pelican population encounters danger, the pelican population transfers to a safe area. Set to represent a normal distribution random number with a mean of 0 and a variance of 1, to represent a constant, and denote the position of the worst pelican individual in the t -th pelican population as . Update the position

[0100]

[0101] ; where represents the fitness function value corresponding to the position , and represents the fitness function value corresponding to the position ;

[0102] S413. After the pelican population completes the exploitation stage, update the pelican population status and the current optimal fitness function value. At this time, enter the next iteration. Stop the iteration until the current iteration count reaches the maximum iteration count, and obtain the global best position. The global best position corresponds to the global optimal solution of the distributed collaborative storage model.

[0103] S42. Use the global optimal solution to solve the distributed collaborative storage model, obtain the maximum storage load that meets the model constraints, and obtain the final state of the pelican population at the maximum iteration count. Migrate the positions of the pelican individuals in the pelican population at the maximum iteration count to the data storage tree, determine the positions of the processed marketing data in the processed marketing data set on the storage nodes and storage devices, generate a distributed collaborative storage strategy, and complete the distributed collaborative storage of marketing data.

[0104] Embodiment 2

[0105] This embodiment also discloses a system for the distributed collaborative storage method of marketing data based on machine learning, specifically including: a data clustering, sorting and classification module, a defective data identification and deletion module, a data distributed collaborative storage module, and a distributed collaborative storage strategy generation module.

[0106] The data clustering, sorting and classification module is used to cluster, sort and classify the initial marketing data set.

[0107] The defective data identification and deletion module is used to establish a defective data identification model to identify and delete defective data.

[0108] The data distributed collaborative storage module is used to establish a distributed collaborative storage model to perform distributed collaborative storage of data.

[0109] The distributed collaborative storage strategy generation module is used to use an optimization algorithm to solve the distributed collaborative storage model and obtain a distributed collaborative storage strategy.

[0110] In the description of this specification, the descriptions referring to terms such as "one embodiment", "example", "specific example", etc. mean that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in a suitable manner in any one or more embodiments or examples.

[0111] The preferred embodiments of the invention disclosed above are only used to help illustrate the invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and variations can be made according to the content of this specification. These embodiments are selected and specifically described in this specification to better explain the principle and practical application of the invention, so that those skilled in the art can well understand and utilize the invention.

Claims

1. A distributed collaborative storage method for marketing data based on machine learning, characterized in that, It includes the following steps: S1. Obtain marketing-related data to get an initial marketing data set, use an improved density clustering algorithm to cluster and organize the initial marketing data set, and then train an XgBoost classification model for classification processing to obtain a classified marketing data set; S11. Obtain marketing-related data from different market sales channels to get an initial marketing data set, and use an improved density clustering algorithm to cluster and organize the initial marketing data set to obtain an organized marketing data set; S111. Calculate the feature vectors corresponding to the initial marketing data set, then calculate the distances of the feature vectors, and count the number of times the distances of the feature vectors correspond to obtain a distance count array; S112. Set a critical point, divide the corresponding initial marketing data in the distance count array into a sparse data point set and a dense data point set, delete the sparse data point set, and then perform clustering processing to complete the clustering and organization to obtain several clustering clusters, which form the organized marketing data set; S12. Re-obtain previous-year marketing-related data to form a previous-year marketing data set. After clustering and organizing the previous-year marketing data set, obtain a marketing sample data set. Extract training samples from the marketing sample data set to obtain m training sample sets. Set that the XGBoost algorithm includes m weak classifiers, input the m training sample sets correspondingly into the m weak classifiers for training until convergence to obtain m trained weak classifiers. Combine the m trained weak classifiers to form an XGBoost classification model, input the organized marketing data set into the XGBoost classification model, each trained weak classifier in the XGBoost classification model outputs a score, add up all the scores to obtain a final output result, and classify the organized marketing data set according to the final output result to obtain a classified marketing data set; S2. Establish an unqualified data recognition model based on the isolation forest algorithm, use the unqualified data recognition model to detect unqualified data in the classified marketing data set, and delete the unqualified data to obtain a processed marketing data set; S3. The processed marketing data set uses a multi-way tree strategy for distributed collaborative storage. Taking the maximum load as the goal, establish a storage maximum load function and set constraint conditions to obtain a distributed collaborative storage model; S4. Use an optimization algorithm to solve the distributed collaborative storage model to obtain a global optimal solution, determine the storage nodes and storage devices of the processed marketing data set, generate a distributed collaborative storage strategy, and complete the distributed collaborative storage of marketing data.

2. The distributed collaborative storage method for marketing data based on machine learning according to claim 1, wherein The S2 includes the following steps: S21. Select any subset of the classified marketing data set, set the tree splitting point, and divide the subset of the classified marketing data into the left subtree and the right subtree of the isolation tree. Iteratively divide in turn until the maximum depth of the isolation tree is reached to obtain the first isolation tree. S22. Generate several isolation trees from the classified marketing data set, stop the iteration of the isolation forest to obtain the unqualified data recognition model, and use the unqualified data recognition model to detect the unqualified data in the classified marketing data set. S23. Traverse all the isolation trees in the unqualified data recognition model, identify all the unqualified data, and delete the unqualified data to obtain the processed marketing data set.

3. The distributed collaborative storage method for marketing data based on machine learning according to claim 2, characterized in that, The S22 includes the following steps: S221. Statistically analyze the depth of the marketing data samples to be processed in the unqualified data recognition model, and calculate the average depth of the marketing data samples to be processed in the isolation tree. S222. Set the depth function, calculate the outlier of the marketing data samples to be processed in the isolation tree according to the average depth of the marketing data samples to be processed in the isolation tree, and detect the unqualified data according to the outlier.

4. The distributed collaborative storage method for marketing data based on machine learning according to claim 3, wherein The S3 includes the following steps: S31. Use the multi-way tree strategy, set an empty data storage tree in the storage space, input the processed marketing data set into the empty data storage tree in turn to establish a data storage tree, and establish a storage maximum load function with the maximum load of the data storage tree as the target. S32. Set the model constraint conditions according to the storage maximum load function, and combine the model constraint conditions and the storage maximum load function to obtain the distributed collaborative storage model.

5. The distributed collaborative storage method for marketing data based on machine learning according to claim 4, wherein The S4 includes the following steps: S41. Take the storage maximum load function as the fitness function, introduce the Cauchy mutation strategy and the sparrow warning mechanism to improve the pelican optimization algorithm, obtain the pelican optimization algorithm with multi-strategy fusion, and use the pelican optimization algorithm with multi-strategy fusion to solve the distributed collaborative storage model to obtain the global optimal solution. S42. Use the global optimal solution to solve the distributed collaborative storage model to obtain the storage maximum load that meets the model constraint conditions, obtain the final state of the pelican population at the maximum number of iterations, migrate the positions of the pelican individuals in the pelican population at the maximum number of iterations to the data storage tree, determine the positions of the processed marketing data in the processed marketing data set on the storage nodes and storage devices, generate the distributed collaborative storage strategy, and complete the distributed collaborative storage of the marketing data.

6. The distributed collaborative storage method for marketing data based on machine learning according to claim 5, characterized in that, The S4 includes the following steps: S411. Regard the data storage tree as the algorithm search space. Assume that there is a pelican population in the algorithm search space, and each pelican individual in the pelican population is regarded as the processed marketing data. The dimension of the pelican population is l, g′ represents the upper bound of the pelican population, g″ represents the lower bound of the pelican population, h1 represents a random number in the interval [0, 1], and the initial position of the p-th pelican individual in the q-th dimension of the pelican population During the exploration stage of the pelican population, assume that the current iteration number is t, and the position of the p-th pelican individual in the q-th dimension of the pelican population at the t-th iteration is denoted as The prey position is X′, and h2 represents a random number of 1 or 2; Calculate the fitness function value of the prey position, denoted as Calculate the fitness function value of the current pelican individual in the current iteration, denoted as f1; when At this time When At this time Update to obtain the current best fitness function value, and the current best fitness function value corresponds to the optimal solution of the current distributed collaborative storage model; S412. After the pelican population completes the exploration stage and reaches the water surface, the pelican population enters the exploitation stage. Set the maximum number of iterations as T. Calculate the fitness function value of the p-th pelican individual in the pelican population during the exploitation stage and the average fitness function value of the pelican population, denoted as f2 and When , mining Using the traditional pelican individual position update formula, the position of the p-th pelican individual in the q-th dimension of the pelican population at the (t + 1)-th iteration is obtained When At this time, the Cauchy mutation strategy is introduced. The standard Cauchy distribution is set as β(0, 1), and the position of the best pelican individual in the t-th pelican population is denoted as X best (t), then Integrate the sparrow warning mechanism. When the pelican population encounters danger, the pelican population transfers to a safe area. Let h3 denote a normally distributed random number with a mean of 0 and a variance of 1, and φ denote a constant. The position of the worst pelican individual in the pelican population at the t-th time is denoted as X worst (t). For the position to be updated, the update formula is as follows: Among them, f3 represents the position X best (t) corresponding fitness function value, and f4 represents the position X worst (t) corresponding fitness function value; S413. After the pelican population completes the exploitation stage, update the state of the pelican population and the current best fitness function value. At this time, enter the next iteration until the current iteration number reaches the maximum iteration number, stop the iteration, and obtain the global best position. The global best position corresponds to the global optimal solution of the distributed collaborative storage model.

7. A system for implementing the distributed collaborative storage method of marketing data based on machine learning according to any one of claims 1-6, characterized in that, Specifically include: Data clustering and classification module, unqualified data identification and deletion module, data distributed collaborative storage module, and distributed collaborative storage strategy generation module; The data clustering, sorting and classification module is used to cluster, sort and classify the initial marketing data set; The unqualified data identification and deletion module is used to establish an unqualified data identification model to identify and delete unqualified data; The data distributed collaborative storage module is used to establish a distributed collaborative storage model for distributed collaborative storage of data; The distributed collaborative storage strategy generation module is used to solve the distributed collaborative storage model using an optimization algorithm to obtain a distributed collaborative storage strategy.

Images