Marketing data distributed collaborative storage system and method based on machine learning

Abstract

The invention relates to the technical field of data storage, and discloses a marketing data distributed collaborative storage system and method based on machine learning. The method comprises the following steps: firstly, performing clustering arrangement and classification processing on initial marketing data to obtain a classified marketing data set; secondly, based on an isolated forest algorithm, establishing an unqualified data identification model according to the classified marketing data set, and identifying and deleting unqualified data to obtain a processed marketing data set; carrying out distributed collaborative storage on the processed marketing data set, and establishing a distributed collaborative storage model by taking the maximum load as a target; and finally, solving the distributed collaborative storage model by using a multi-strategy fusion pelican optimization algorithm to obtain a global optimal solution and a distributed collaborative storage strategy. By processing and storing the marketing data, the purpose of distributed collaborative storage of the marketing data is achieved, and the method is accurate and objective.

Description

Technical Field

[0001] The present invention relates to the technical field of data storage, and in particular to a distributed collaborative storage system and method for marketing data based on machine learning. Background Art

[0002] Chinese patent CN118331791B discloses a data collaborative backup system based on cloud computing and distributed storage. The system specifically includes a data preprocessing module, a distributed storage module, a collaborative management module, and a data recovery module. The interaction process of each module is as follows: obtain data and transmit it to the data preprocessing module, perform weight calculation on the received data, obtain weight information, generate a backup strategy with the weight information, and then transmit the backup strategy to the distributed storage module; the distributed storage module stores the backup strategy in a distributed manner, and the data recovery module is connected to the distributed storage module to receive the recovery instruction sent by the collaborative management module; the collaborative management module connects to other modules, monitors the module status, and issues staff instructions. However, this invention does not clear data redundancy, resulting in slow storage speed and long processing time.

[0003] Traditional distributed collaborative data storage methods usually store data in data storage centers without further organizing the data, ignoring the availability and integrity of the data. At the same time, due to the lack of the use of technologies such as machine learning, no relevant models are established in the distributed collaborative storage process of marketing data, resulting in the inability to quickly and effectively provide decision support for the marketing team. Summary of the invention

[0004] In response to the problems in the related technology, the present invention provides a distributed collaborative storage system and method for marketing data based on machine learning to overcome the above-mentioned technical problems existing in the existing related technology.

[0005] In order to solve the above technical problems, the present invention is achieved through the following technical solutions: The present invention is a distributed collaborative storage method for marketing data based on machine learning, comprising the following steps: S1. Obtain marketing-related data to obtain an initial marketing data set, use an improved density clustering algorithm to cluster and sort the initial marketing data set, and then train an XgBoost classification model for classification processing to obtain a classified marketing data set; S2. Establishing an unqualified data identification model based on the isolation forest algorithm, using the unqualified data identification model to detect unqualified data in the classified marketing data set, and deleting the unqualified data to obtain a processed marketing data set; S3, the processed marketing data set is distributed and collaboratively stored using a multi-tree strategy, with the maximum load as the goal, a storage maximum load function is established, and constraints are set to obtain a distributed collaborative storage model; S4. Use the multi-strategy fusion Pelican optimization algorithm to solve the distributed collaborative storage model, obtain the global optimal solution, determine the storage nodes and storage devices of the processed marketing data set, generate a distributed collaborative storage strategy, and complete the distributed collaborative storage of marketing data.

[0006] The invention clusters and classifies the initial marketing data. During clustering, the data is divided according to the degree of sparsity, and the sparse data is deleted while the invalid data set containing many missing data is deleted, thereby improving the accuracy and effectiveness of clustering and reducing the time consumed in calculation. Compared with direct input, the clustered data is input into the XGBoost classification model, which can effectively improve the classification accuracy of the model and enhance the generalization ability of the model. Secondly, an unqualified data recognition model is established based on the isolation forest algorithm, and unqualified data recognition is achieved by calculating the anomaly scores of the data on the isolated trees in the unqualified data recognition model. This method has good resistance to missing data and has higher detection speed and accuracy than traditional anomaly recognition methods. The calculation accuracy is suitable for large-scale data processing; the processed marketing data is then distributed and collaboratively stored, and a distributed collaborative storage model is established with the maximum load as the goal; this model can avoid the delay problem in the data storage process, solve the problem of large load in the distributed storage process, and facilitate the visualization of the data storage process; finally, the multi-strategy fusion pelican optimization algorithm is used to solve the distributed collaborative storage model, and the global optimal solution is obtained by simulating the behavior of pelicans in the hunting process, and a distributed collaborative storage strategy is generated. Compared with the traditional algorithm, this algorithm reduces the probability of obtaining a local optimal solution and increases the global exploration ability by introducing the Cauchy mutation strategy, integrating the sparrow early warning mechanism, and greatly improves the convergence speed.

[0007] Preferably, the S1 comprises the following steps: S11, obtaining marketing-related data from different market sales channels, wherein the marketing-related data includes sales volume, market growth rate, market share, etc. The marketing-related data of each market sales channel constitutes a marketing data subset, and all marketing data subsets generate an initial marketing data set. The initial marketing data set is clustered and sorted using an improved density clustering algorithm to obtain a sorted marketing data set. The specific steps are as follows: S111, projecting the initial marketing data set into a three-dimensional space, regarding the initial marketing data in the initial marketing data set as an initial marketing data point, selecting any initial marketing data point as a first marketing data point, setting a standard data point, calculating a feature vector of the standard data point and a feature vector of the first marketing data point, and calculating a distance between the feature vectors, the calculation formula is as follows: ; in, l represents the distance between the feature vector of the standard data point and the feature vector of the first marketing data point, represents the coordinates of the eigenvector of the standard data point, represents the coordinates of the feature vector of the first marketing data point; Select the second marketing data points in sequence until all the initial marketing data points are traversed to obtain a feature vector distance array, sort the feature vector distance array in ascending order, and count the number of distances from the standard data point corresponding to the initial marketing data point to obtain a distance number array ,in The distance between the feature vectors is The corresponding number; S112, according to the distance number array, set a critical point, classify the initial marketing data points corresponding to the distance number less than the critical point in the distance number array into the sparse data point set, classify the initial marketing data points corresponding to the distance number greater than the critical point in the distance number array into the dense data point set, and record the initial marketing data points corresponding to the distance number equal to the critical point in the distance number array as swing data points; perform clustering processing on the sparse data point set and the dense data point set to obtain a first central data point and a second central pixel point respectively, calculate the similarity between the swing data point and the first central data point and the second central pixel point, and compare the similarities, classify the swing data point into the data point set with greater similarity, delete the sparse data point set, perform clustering processing again, obtain a number of clusters, and form a sorted marketing data set, and complete clustering sorting; S12, reacquire the marketing-related data of previous years to form a marketing data set of previous years, cluster and sort the marketing data set of previous years to obtain a marketing sample data set, extract training samples from the marketing sample data set, and obtain m training sample sets; setting the XGBoost algorithm to include m Weak classifiers, m The training sample set corresponds to the input m We train in weak classifiers until convergence, and get m A trained weak classifier; mThe trained weak classifiers form an XGBoost classification model, the sorted marketing data set is input into the XGBoost classification model, each trained weak classifier in the XGBoost classification model outputs a score, all the scores are added up to obtain a final output result, and the sorted marketing data set is classified according to the final output result to obtain a classified marketing data set, the classified marketing data set includes several classified marketing data subsets, and the classified marketing data subsets correspond to marketing-related data.

[0008] The invention clusters and classifies the initial marketing data, wherein the data is divided according to the degree of sparsity, sparse data is deleted, and invalid data sets containing a lot of missing data are also deleted, thereby improving the accuracy and effectiveness of clustering and reducing calculation time; after clustering, the data is input into the XGBoost classification model, which can effectively improve the classification accuracy of the model and enhance the generalization ability of the model compared to direct input.

[0009] Preferably, S2 comprises the following steps: S21, selecting any classified marketing data subset from the classified marketing data set, and recording it as a marketing data sample set to be processed ,in Indicates n marketing data samples to be processed, the maximum data dimension and the minimum data dimension of the marketing data sample set to be processed are set, and any data dimension between the maximum data dimension and the minimum data dimension is selected and recorded as a tree splitting point; a hyperplane is generated on the dimension where the tree splitting point is located to obtain a first data sample space, and the first data sample space is the first layer of the isolated tree, and the marketing data samples to be processed that are smaller than the corresponding tree splitting point in the marketing data sample set to be processed are added to the left subtree of the tree splitting point, and the marketing data samples to be processed that are greater than or equal to the corresponding tree splitting point in the marketing data sample set to be processed are added to the right subtree of the tree splitting point to complete the first layer segmentation of the isolated tree; further, the left subtree of the tree splitting point and the right subtree of the tree splitting point generate a second data sample space, and the iteration is continued to segment the left subtree of the tree splitting point and the right subtree of the tree splitting point until the maximum depth of the isolated tree is reached to obtain the first isolated tree; S22, generating a plurality of isolated trees according to other classified marketing data subsets in the classified marketing data set, stopping iteration of the isolation forest, obtaining an unqualified data identification model, and using the unqualified data identification model to detect unqualified data in the classified marketing data set, the specific steps are as follows: S221, the unqualified data identification model includes a plurality of isolated trees, traverses all isolated trees, counts the depths of the marketing data samples to be processed in the isolated trees, and calculates the average depths of the marketing data samples to be processed in the isolated trees; S222, set the depth function to , calculate the outlier value of the marketing data sample to be processed in the isolation tree, and the calculation formula is as follows: ; in, Represents the number of isolated trees e outliers in the marketing data samples to be processed, represents the average depth of the marketing data samples to be processed in the isolation tree, Represents the number of isolated trees e The depth of the marketing data samples to be processed, Indicates e A depth function of the marketing data samples to be processed; when hour, , at this time, the first e The marketing data samples to be processed are considered unqualified data, otherwise the first e The samples of marketing data to be processed are considered as qualified data; S23, traverse all isolated trees in the unqualified data identification model, calculate the outliers of the classified marketing data set, identify all unqualified data, delete the unqualified data, and obtain a processed marketing data set.

[0010] The invention establishes an unqualified data identification model by using the isolation forest algorithm, and realizes unqualified data identification by calculating the anomaly scores of the data on the isolated trees in the unqualified data identification model; this method has good resistance to missing data, and compared with traditional anomaly identification methods, it has higher detection speed and calculation accuracy, and is suitable for processing large-scale data.

[0011] Preferably, S3 comprises the following steps: S31, using the multi-tree strategy, assume that there is an empty data storage tree in the storage space, and the storage node of the data storage tree is recorded as ,in Indicates i storage nodes, each storage node contains jA storage device is configured, and the processed marketing data set is input into an empty data storage tree in sequence to establish a data storage tree. The size of the processed marketing data received by each storage node is recorded as the storage node load. The processed marketing data on the storage node is then distributed and collaboratively stored. The processed marketing data is stored in the storage device under the storage node, and the selection coefficient of each processed marketing data stored in the storage device is calculated. The maximum load of the data storage tree is taken as the target, and a storage maximum load function is established. The calculation formula is as follows: ; in, represents the storage maximum load function, Indicates g The size of the processed marketing data, Indicates g The processed marketing data is h The selection coefficient of storage nodes, Indicates h On the storage node k The selection coefficient of each storage device, Indicates the number of processed marketing data. Indicates the number of storage devices. , , ; S32, setting model constraints according to the storage maximum load function, setting the storage capacity of the storage device to be less than or equal to the storage device capacity, when When g The processed marketing data is stored in h On a storage node, When g The processed marketing data is not stored in h On each storage node, Indicates that the h On the storage node storage devices, combined with model constraints and storage maximum load function, a distributed collaborative storage model is obtained.

[0012] This invention solves the problem of heavy load in the distributed storage process by distributing and collaboratively storing the processed marketing data with the maximum load as the goal. It establishes a distributed collaborative storage model, which can avoid the delay problem in the data storage process and facilitate the visualization of the data storage process.

[0013] Preferably, S4 comprises the following steps: S41. The maximum storage load function is used as the fitness function, the Cauchy mutation strategy and the sparrow early warning mechanism are introduced, the Pelican optimization algorithm is improved, and a multi-strategy fusion Pelican optimization algorithm is obtained. The distributed collaborative storage model is solved using the multi-strategy fusion Pelican optimization algorithm to obtain a global optimal solution. The specific steps are as follows: S411. The data storage tree is regarded as an algorithm search space. It is assumed that there is a pelican population in the algorithm search space. Each pelican individual in the pelican population is regarded as processed marketing data. The dimension of the pelican population is l , represents the upper bound of the pelican population, represents the lower bound of the pelican population, represents a random number between the interval [0, 1], the first p The pelican individuals q Initial position of dimension ; The pelican population is in the exploration phase, and the current number of iterations is set to t , No. t In the iteration of the pelican population p The pelican individuals q The position of the dimension is recorded as , the prey position is , A random number representing 1 or 2; Calculate the fitness function value of the prey position, denoted as , calculate the fitness function value of the pelican individual in the current iteration, recorded as ;when hour, ,when hour, ; Update to obtain the current best fitness function value, which corresponds to the optimal solution of the current distributed collaborative storage model; S412, after completing the exploration phase, the pelican population reaches the water surface and enters the mining phase. The maximum number of iterations is set to T , calculate the first p The fitness function value of each individual pelican and the average fitness function value of the pelican population are recorded as and ;when When Using the traditional pelican individual position update formula, we get t +1 iteration of the pelican population p The pelican individuals q Dimensional Position ;when When , the Cauchy mutation strategy is introduced and the standard Cauchy distribution is set as , No.t The best individual pelican position in the secondary pelican population is recorded as ,but ; Integrate the sparrow early warning mechanism. When the pelican population encounters danger, the pelican population moves to a safe area. represents a normally distributed random number with a mean of 0 and a variance of 1. represents a constant, t The position of the worst individual pelican in the secondary pelican population is recorded as , for position Update, the update formula is as follows: ; in, Indicates location The corresponding fitness function value is, Indicates location The corresponding fitness function value; S413, after the pelican population completes the mining stage, the pelican population state and the current best fitness function value are updated, and the next iteration is entered until the current number of iterations reaches the maximum number of iterations, then the iteration is stopped to obtain the global best position, which corresponds to the global optimal solution of the distributed collaborative storage model; S42. Use the global optimal solution to solve the distributed collaborative storage model, obtain the maximum storage load that meets the model constraints, obtain the final state of the pelican population at the maximum number of iterations, migrate the individual positions of pelicans in the pelican population at the maximum number of iterations to the data storage tree, determine the positions of the processed marketing data in the processed marketing data set in the storage nodes and storage devices, generate a distributed collaborative storage strategy, and complete the distributed collaborative storage of marketing data.

[0014] This invention solves the distributed collaborative storage model by using the pelican optimization algorithm with multi-strategy fusion, obtains the global optimal solution by simulating the behavior of pelicans in the hunting process, and generates a distributed collaborative storage strategy. Compared with traditional algorithms, it reduces the probability of obtaining local optimal solutions and increases the global exploration capability by introducing the Cauchy mutation strategy. It also integrates the sparrow early warning mechanism and greatly improves the convergence speed.

[0015] This embodiment also discloses a system for a distributed collaborative storage method of marketing data based on machine learning, which specifically includes: a data clustering and classification module, an unqualified data identification and deletion module, a data distributed collaborative storage module, and a distributed collaborative storage strategy generation module; The data clustering and classification module is used to perform clustering and classification processing on the initial marketing data set; The unqualified data identification and deletion module is used to establish an unqualified data identification model to identify and delete unqualified data; The data distributed collaborative storage module is used to establish a distributed collaborative storage model to perform distributed collaborative storage of data; The distributed collaborative storage strategy generation module is used to solve the distributed collaborative storage model using an optimization algorithm to obtain a distributed collaborative storage strategy.

[0016] The present invention has the following beneficial effects: 1. The invention clusters and classifies the initial marketing data, wherein the data is divided according to the degree of sparsity, sparse data is deleted, and invalid data sets containing a large number of missing data are deleted, thereby improving the accuracy and effectiveness of clustering and reducing the time spent on calculation.

[0017] 2. This invention inputs the data into the XGBoost classification model after clustering, which can effectively improve the classification accuracy of the model and enhance the generalization ability of the model compared to direct input.

[0018] 3. The invention establishes an unqualified data identification model by using the isolation forest algorithm. This method has good resistance to missing data. Compared with traditional anomaly identification methods, it has higher detection speed and calculation accuracy and is suitable for processing large-scale data.

[0019] 4. This invention solves the problem of heavy load in the distributed storage process by performing distributed collaborative storage with the maximum load as the target, and establishes a distributed collaborative storage model, which can avoid the delay problem in the data storage process and facilitate the description of the data storage process.

[0020] 5. This invention solves the distributed collaborative storage model by using the Pelican optimization algorithm with multi-strategy fusion to obtain the global optimal solution and generate a distributed collaborative storage strategy. Compared with the traditional algorithm, by introducing the Cauchy mutation strategy, the probability of obtaining the local optimal solution is reduced, the global exploration capability is increased, and the sparrow early warning mechanism is integrated, which greatly improves the convergence speed.

[0021] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the invention, the drawings required for describing the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the invention. For ordinary technicians in this field, they can also obtain drawings based on these drawings without paying any creative work.

[0023] Figure 1A schematic diagram of the process of distributed collaborative storage of marketing data in the distributed collaborative storage system for marketing data based on machine learning provided by the present invention. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] In the description of the present invention, it is necessary to understand that the terms "opening", "upper", "lower", "top", "middle", "inside" and the like indicating orientation or positional relationship are only for the convenience of describing the invention and simplifying the description, and do not indicate or imply that the components or elements referred to must have a specific orientation, be constructed and operate in a specific orientation, and therefore cannot be understood as limiting the invention.

[0026] Example 1 See also Figure 1 This implementation discloses a distributed collaborative storage method for marketing data based on machine learning, which specifically includes the following contents: S1. Obtain marketing-related data to obtain an initial marketing data set, use an improved density clustering algorithm to cluster and sort the initial marketing data set, and then train an XgBoost classification model for classification processing to obtain a classified marketing data set; The S1 comprises the following steps: S11, obtaining marketing-related data from different market sales channels, wherein the marketing-related data includes sales volume, market growth rate, market share, etc. The marketing-related data of each market sales channel constitutes a marketing data subset, and all marketing data subsets generate an initial marketing data set. The initial marketing data set is clustered and sorted using an improved density clustering algorithm to obtain a sorted marketing data set. The specific steps are as follows: S111, projecting the initial marketing data set into a three-dimensional space, regarding the initial marketing data in the initial marketing data set as an initial marketing data point, selecting any initial marketing data point as a first marketing data point, setting a standard data point, calculating a feature vector of the standard data point and a feature vector of the first marketing data point, and calculating a distance between the feature vectors, the calculation formula is as follows: ; in, lrepresents the distance between the feature vector of the standard data point and the feature vector of the first marketing data point, represents the coordinates of the eigenvector of the standard data point, represents the coordinates of the feature vector of the first marketing data point; Select the second marketing data points in sequence until all the initial marketing data points are traversed to obtain a feature vector distance array, sort the feature vector distance array in ascending order, and count the number of distances from the standard data point corresponding to the initial marketing data point to obtain a distance number array ,in The distance between the feature vectors is The corresponding number; S112, according to the distance number array, set a critical point, classify the initial marketing data points corresponding to the distance number less than the critical point in the distance number array into the sparse data point set, classify the initial marketing data points corresponding to the distance number greater than the critical point in the distance number array into the dense data point set, and record the initial marketing data points corresponding to the distance number equal to the critical point in the distance number array as swing data points; perform clustering processing on the sparse data point set and the dense data point set to obtain a first central data point and a second central pixel point respectively, calculate the similarity between the swing data point and the first central data point and the second central pixel point, and compare the similarities, classify the swing data point into the data point set with greater similarity, delete the sparse data point set, perform clustering processing again, obtain a number of clusters, and form a sorted marketing data set, and complete clustering sorting; S12, reacquire the marketing-related data of previous years to form a marketing data set of previous years, cluster and sort the marketing data set of previous years to obtain a marketing sample data set, extract training samples from the marketing sample data set, and obtain m training sample sets; setting the XGBoost algorithm to include m Weak classifiers, m The training sample set corresponds to the input m We train in weak classifiers until convergence, and get m A trained weak classifier; mtrained weak classifiers to form an XGBoost classification model, input the sorted marketing data set into the XGBoost classification model, each trained weak classifier in the XGBoost classification model outputs a score, all scores are added up to obtain a final output result, and the sorted marketing data set is classified according to the final output result to obtain a classified marketing data set, wherein the classified marketing data set includes a plurality of classified marketing data subsets, and the classified marketing data subsets correspond to marketing-related data; S2. Establishing an unqualified data identification model based on the isolation forest algorithm, using the unqualified data identification model to detect unqualified data in the classified marketing data set, and deleting the unqualified data to obtain a processed marketing data set; The S2 comprises the following steps: S21, selecting any classified marketing data subset from the classified marketing data set, and recording it as a marketing data sample set to be processed ,in Indicates n marketing data samples to be processed, the maximum data dimension and the minimum data dimension of the marketing data sample set to be processed are set, and any data dimension between the maximum data dimension and the minimum data dimension is selected and recorded as a tree splitting point; a hyperplane is generated on the dimension where the tree splitting point is located to obtain a first data sample space, and the first data sample space is the first layer of the isolated tree, and the marketing data samples to be processed that are smaller than the corresponding tree splitting point in the marketing data sample set to be processed are added to the left subtree of the tree splitting point, and the marketing data samples to be processed that are greater than or equal to the corresponding tree splitting point in the marketing data sample set to be processed are added to the right subtree of the tree splitting point to complete the first layer segmentation of the isolated tree; further, the left subtree of the tree splitting point and the right subtree of the tree splitting point generate a second data sample space, and the iteration is continued to segment the left subtree of the tree splitting point and the right subtree of the tree splitting point until the maximum depth of the isolated tree is reached to obtain the first isolated tree; S22, generating a plurality of isolated trees according to other classified marketing data subsets in the classified marketing data set, stopping iteration of the isolation forest, obtaining an unqualified data identification model, and using the unqualified data identification model to detect unqualified data in the classified marketing data set, the specific steps are as follows: S221, the unqualified data identification model includes a plurality of isolated trees, traverses all isolated trees, counts the depths of the marketing data samples to be processed in the isolated trees, and calculates the average depths of the marketing data samples to be processed in the isolated trees; S222, set the depth function to , calculate the outlier value of the marketing data sample to be processed in the isolation tree, and the calculation formula is as follows: ; in, Represents the number of isolated trees e outliers in the marketing data samples to be processed, represents the average depth of the marketing data samples to be processed in the isolation tree, Indicates the number of isolated trees e The depth of the marketing data samples to be processed, Indicates e A depth function of the marketing data samples to be processed; when hour, , at this time, the first e The marketing data samples to be processed are considered unqualified data, otherwise the first e The samples of marketing data to be processed are considered as qualified data; S23, traversing all isolated trees in the unqualified data identification model, calculating the outliers of the classified marketing data set, identifying all unqualified data, deleting the unqualified data, and obtaining a processed marketing data set; S3, the processed marketing data set is distributed and collaboratively stored using a multi-tree strategy, with the maximum load as the goal, a storage maximum load function is established, and constraints are set to obtain a distributed collaborative storage model; The S3 comprises the following steps: S31, using the multi-tree strategy, assume that there is an empty data storage tree in the storage space, and the storage node of the data storage tree is recorded as ,in Indicates i storage nodes, each storage node contains j A storage device is configured, and the processed marketing data set is input into an empty data storage tree in sequence to establish a data storage tree. The size of the processed marketing data received by each storage node is recorded as the storage node load. The processed marketing data on the storage node is then distributed and collaboratively stored. The processed marketing data is stored in the storage device under the storage node, and the selection coefficient of each processed marketing data stored in the storage device is calculated. The maximum load of the data storage tree is taken as the target, and a storage maximum load function is established. The calculation formula is as follows: ; in, represents the storage maximum load function, Indicates gThe size of the processed marketing data, Indicates g The processed marketing data is h The selection coefficient of storage nodes, Indicates h On the storage node k The selection factor of each storage device, Indicates the number of processed marketing data. Indicates the number of storage devices. , , ; S32, setting model constraints according to the storage maximum load function, setting the storage capacity of the storage device to be less than or equal to the storage device capacity, when When g The processed marketing data is stored in h On a storage node, When g The processed marketing data is not stored in h On each storage node, Indicates that the h On the storage node storage devices, combined with model constraints and storage maximum load function, to obtain a distributed collaborative storage model; S4. Use the multi-strategy fusion Pelican optimization algorithm to solve the distributed collaborative storage model, obtain the global optimal solution, determine the storage nodes and storage devices of the processed marketing data set, generate a distributed collaborative storage strategy, and complete the distributed collaborative storage of marketing data; The S4 comprises the following steps: S41. The maximum storage load function is used as the fitness function, the Cauchy mutation strategy and the sparrow early warning mechanism are introduced, the Pelican optimization algorithm is improved, and a multi-strategy fusion Pelican optimization algorithm is obtained. The distributed collaborative storage model is solved using the multi-strategy fusion Pelican optimization algorithm to obtain a global optimal solution. The specific steps are as follows: S411. The data storage tree is regarded as an algorithm search space. It is assumed that there is a pelican population in the algorithm search space. Each pelican individual in the pelican population is regarded as processed marketing data. The dimension of the pelican population is l , represents the upper bound of the pelican population, represents the lower bound of the pelican population, represents a random number between the interval [0, 1], the first p The pelican individuals q Initial position of dimension ; The pelican population is in the exploration phase, and the current number of iterations is set to t , No. t In the iteration of the pelican population p The pelican individuals q The position of the dimension is recorded as , the prey position is , A random number representing 1 or 2; Calculate the fitness function value of the prey position, denoted as , calculate the fitness function value of the pelican individual in the current iteration, recorded as ;when hour, ,when hour, ; Update to obtain the current best fitness function value, which corresponds to the optimal solution of the current distributed collaborative storage model; S412, after completing the exploration phase, the pelican population reaches the water surface and enters the mining phase. The maximum number of iterations is set to T , calculate the first p The fitness function value of each individual pelican and the average fitness function value of the pelican population are recorded as and ;when When Using the traditional pelican individual position update formula, we get t +1 iteration of the pelican population p The pelican individuals q Dimensional Position ;when When , the Cauchy mutation strategy is introduced and the standard Cauchy distribution is set as , No. t The best individual pelican position in the secondary pelican population is recorded as ,but ; Integrate the sparrow early warning mechanism. When the pelican population encounters danger, the pelican population moves to a safe area. represents a normally distributed random number with a mean of 0 and a variance of 1. represents a constant, t The position of the worst individual pelican in the secondary pelican population is recorded as , for position Update, the update formula is as follows: ; in, Indicates location The corresponding fitness function value is, Indicates location The corresponding fitness function value; S413, after the pelican population completes the mining stage, the pelican population state and the current best fitness function value are updated, and the next iteration is entered until the current number of iterations reaches the maximum number of iterations, then the iteration is stopped to obtain the global best position, which corresponds to the global optimal solution of the distributed collaborative storage model; S42. Use the global optimal solution to solve the distributed collaborative storage model, obtain the maximum storage load that meets the model constraints, obtain the final state of the pelican population at the maximum number of iterations, migrate the individual positions of pelicans in the pelican population at the maximum number of iterations to the data storage tree, determine the positions of the processed marketing data in the processed marketing data set in the storage nodes and storage devices, generate a distributed collaborative storage strategy, and complete the distributed collaborative storage of marketing data.

[0027] Example 2 This embodiment also discloses a system for a distributed collaborative storage method of marketing data based on machine learning, which specifically includes: a data clustering and classification module, an unqualified data identification and deletion module, a data distributed collaborative storage module, and a distributed collaborative storage strategy generation module; The data clustering and classification module is used to perform clustering and classification processing on the initial marketing data set; The unqualified data identification and deletion module is used to establish an unqualified data identification model to identify and delete unqualified data; The data distributed collaborative storage module is used to establish a distributed collaborative storage model to perform distributed collaborative storage of data; The distributed collaborative storage strategy generation module is used to solve the distributed collaborative storage model using an optimization algorithm to obtain a distributed collaborative storage strategy.

[0028] In the description of this specification, the description with reference to the terms "one embodiment", "example", "specific example", etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner.

[0029] The preferred embodiments of the invention disclosed above are only used to help explain the invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the invention, so that those skilled in the art can understand and use the invention well.

Claims

1. A distributed collaborative storage method for marketing data based on machine learning, characterized in that: The steps include: S1. Obtain marketing-related data to obtain an initial marketing data set, use an improved density clustering algorithm to cluster and sort the initial marketing data set, and then train an XgBoost classification model for classification processing to obtain a classified marketing data set; S2. Establishing an unqualified data identification model based on the isolation forest algorithm, using the unqualified data identification model to detect unqualified data in the classified marketing data set, and deleting the unqualified data to obtain a processed marketing data set; S3, the processed marketing data set is distributed and collaboratively stored using a multi-tree strategy, with the maximum load as the goal, a storage maximum load function is established, and constraints are set to obtain a distributed collaborative storage model; S4. Use an optimization algorithm to solve the distributed collaborative storage model to obtain a global optimal solution, determine the storage nodes and storage devices of the processed marketing data set, generate a distributed collaborative storage strategy, and complete the distributed collaborative storage of marketing data.

2. The distributed collaborative storage method for marketing data based on machine learning according to claim 1, characterized in that: The S1 comprises the following steps: S11, acquiring marketing-related data from different market sales channels to obtain an initial marketing data set, and using an improved density clustering algorithm to cluster and organize the initial marketing data set to obtain an organized marketing data set; S12, reacquire the marketing-related data of previous years to form a marketing data set of previous years, cluster and sort the marketing data set of previous years to obtain a marketing sample data set, extract training samples from the marketing sample data set, and obtain m training sample sets; setting the XGBoost algorithm to include m Weak classifiers, m The training sample set corresponds to the input m We train in weak classifiers until convergence, and get m A trained weak classifier; Will m The trained weak classifiers form an XGBoost classification model, the sorted marketing data set is input into the XGBoost classification model, each trained weak classifier in the XGBoost classification model outputs a score, all scores are added up to obtain a final output result, and the sorted marketing data set is classified according to the final output result to obtain a classified marketing data set.

3. The distributed collaborative storage method for marketing data based on machine learning according to claim 2, characterized in that: The S11 comprises the following steps: S111, calculating the feature vector corresponding to the initial marketing data set, then calculating the distance of the feature vector, counting the number of feature vector distances, and obtaining a distance number array; S112, setting a critical point, dividing the initial marketing data corresponding to the distance number array into a sparse data point set and a dense data point set, deleting the sparse data point set, and then performing clustering processing to complete clustering sorting to obtain a number of cluster clusters to form a sorted marketing data set.

4. The distributed collaborative storage method for marketing data based on machine learning according to claim 3 is characterized in that: The S2 comprises the following steps: S21, selecting any classified marketing data subset from the classified marketing data set, setting a tree split point, dividing the classified marketing data subset into a left subtree and a right subtree of an isolated tree, and iterating the division in sequence until the maximum depth of the isolated tree is reached, thereby obtaining a first isolated tree; S22, the classified marketing data set generates a plurality of isolated trees, the isolation forest stops iteration, and an unqualified data identification model is obtained, and the unqualified data identification model is used to detect unqualified data in the classified marketing data set; S23. Traverse all isolated trees in the unqualified data identification model, identify all unqualified data, delete the unqualified data, and obtain a processed marketing data set.

5. The distributed collaborative storage method for marketing data based on machine learning according to claim 4 is characterized in that: The S22 comprises the following steps: S221. Count the depths of the marketing data samples to be processed in the unqualified data identification model, and calculate the average depth of the marketing data samples to be processed in the isolated tree; S222. Set a depth function, calculate the outliers of the marketing data samples to be processed in the isolated tree according to the average depth of the marketing data samples to be processed in the isolated tree, and obtain unqualified data according to outlier detection.

6. The distributed collaborative storage method for marketing data based on machine learning according to claim 5, characterized in that: The S3 comprises the following steps: S31, using a multi-tree strategy, assuming that there is an empty data storage tree in the storage space, inputting the processed marketing data set into the empty data storage tree in sequence, establishing a data storage tree, taking the maximum load of the data storage tree as a target, and establishing a storage maximum load function; S32. Setting model constraints according to the storage maximum load function, and combining the model constraints and the storage maximum load function to obtain a distributed collaborative storage model.

7. The distributed collaborative storage method for marketing data based on machine learning according to claim 6, characterized in that: The S4 comprises the following steps: S41, using the maximum storage load function as a fitness function, introducing the Cauchy mutation strategy and the sparrow early warning mechanism, improving the Pelican optimization algorithm, obtaining a multi-strategy fusion Pelican optimization algorithm, and using the multi-strategy fusion Pelican optimization algorithm to solve the distributed collaborative storage model and obtain a global optimal solution; S42. Use the global optimal solution to solve the distributed collaborative storage model, obtain the maximum storage load that meets the model constraints, obtain the final state of the pelican population at the maximum number of iterations, migrate the individual positions of pelicans in the pelican population at the maximum number of iterations to the data storage tree, determine the positions of the processed marketing data in the processed marketing data set in the storage nodes and storage devices, generate a distributed collaborative storage strategy, and complete the distributed collaborative storage of marketing data.

8. The distributed collaborative storage method for marketing data based on machine learning according to claim 7, characterized in that: The S4 comprises the following steps: S411. The data storage tree is regarded as an algorithm search space. It is assumed that there is a pelican population in the algorithm search space. Each pelican individual in the pelican population is regarded as processed marketing data. The dimension of the pelican population is l , represents the upper bound of the pelican population, represents the lower bound of the pelican population, represents a random number between the interval [0, 1], the first p The pelican individuals q Initial position of dimension ; The pelican population is in the exploration phase, and the current number of iterations is set to t , No. t In the iteration of the pelican population p The pelican individuals q The position of the dimension is recorded as , the prey position is , A random number representing 1 or 2; Calculate the fitness function value of the prey position, denoted as , calculate the fitness function value of the pelican individual in the current iteration, recorded as ;when hour, ,when hour, ; Update to obtain the current best fitness function value, which corresponds to the optimal solution of the current distributed collaborative storage model; S412, after completing the exploration phase, the pelican population reaches the water surface and enters the mining phase. The maximum number of iterations is set to T , calculate the first p The fitness function value of each individual pelican and the average fitness function value of the pelican population are recorded as and ;when When Using the traditional pelican individual position update formula, we get t +1 iteration of the pelican population p The pelican individuals q Dimensional Position ;when When , the Cauchy mutation strategy is introduced and the standard Cauchy distribution is set as , No. t The best individual pelican position in the secondary pelican population is recorded as ,but ; Integrate the sparrow early warning mechanism. When the pelican population encounters danger, the pelican population moves to a safe area. represents a normally distributed random number with a mean of 0 and a variance of 1. represents a constant, t The position of the worst individual pelican in the secondary pelican population is recorded as , for position Update, the update formula is as follows: ； in, Indicates location The corresponding fitness function value is, Indicates location The corresponding fitness function value; S413, after the pelican population completes the mining stage, the pelican population status and the current best fitness function value are updated, and the next iteration is entered until the current number of iterations reaches the maximum number of iterations, then the iteration is stopped to obtain the global best position, which corresponds to the global optimal solution of the distributed collaborative storage model.

9. A system for implementing the distributed collaborative storage method for marketing data based on machine learning as claimed in any one of claims 1 to 8, characterized in that: Specifically include: Data clustering and classification module, unqualified data identification and deletion module, data distributed collaborative storage module and distributed collaborative storage strategy generation module; The data clustering and classification module is used to perform clustering and classification processing on the initial marketing data set; The unqualified data identification and deletion module is used to establish an unqualified data identification model to identify and delete unqualified data; The data distributed collaborative storage module is used to establish a distributed collaborative storage model to perform distributed collaborative storage of data; The distributed collaborative storage strategy generation module is used to solve the distributed collaborative storage model using an optimization algorithm to obtain a distributed collaborative storage strategy.

Images