Big data collection and distribution method and big data platform

Abstract

The application relates to the technical field of big data, in particular to a big data collection and release method and a big data platform, which comprises the following steps: S1, collecting data through a data transmission device; S2, performing mining processing on the collected data to obtain classified data; and S3, performing integrated classification and release on the data obtained through the classification processing. The application can mine and classify massive data through a data mining method, can classify according to user requirements, and can obtain an optimal weight matrix through optimization of a weight adjustment factor by using an improved genetic algorithm, so that the data collected by different collection modes can be processed respectively through a classification and integration method based on a classification function, the method has wide applicability, the obtained data has low redundancy and high quality.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of big data, and particularly relates to a big data collection and release method and a big data platform. BACKGROUND

[0002] With the advent of the big data era, people are provided with massive and rich data information, and the information status is greatly equalized. Big data is widely applied in many fields, such as information retrieval, personalized recommendation, criminal investigation, etc. The advantage of big data is very prominent, but the use of big data also faces some practical difficulties, and the fundamental reason is the information redundancy brought by big data. Therefore, a big data collection and release method is urgently needed to process, integrate and release the collected big data, improve the quality of data output, and provide the most valuable information for users. The prior art sets different interfaces for data collection and release, but the data collection includes multiple modes, and the data collected through the configured interface has low applicability. The present application processes the data collected by different collection modes through data mining and classification integration release methods, and has wide applicability, low redundancy and high quality of the obtained data. SUMMARY

[0003] The purpose of the present application is to solve the defects in the background art by providing a big data collection and release method and a big data platform.

[0004] The technical scheme adopted by the present application is as follows:

[0005] A big data collection and release method is provided, comprising the following steps:

[0006] S1: collecting data through a data transmission device;

[0007] S2: mining and processing the collected data to obtain classified data;

[0008] S3: integrating and classifying the data obtained in the classification processing.

[0009] As a preferred technical scheme of the present application: in S1, data is collected through sensor collection, crawler collection, input collection, import collection and interface collection.

[0010] As a preferred technical scheme of the present application: in S2, the collected data is also cleaned.

[0011] As a preferred technical scheme of the present application: in S2, the data obtained after data cleaning is processed based on a mining algorithm.

[0012] As a preferred technical solution of the present application: in S2, the data mining is performed according to the attributes of the collected data and the mining conditions to obtain a judgment matrix, and based on the judgment matrix, the data is trained and classified as the output weight matrix of the RBF neural network.

[0013] As a preferred technical solution of the present application: the mining algorithm in S2 is specifically as follows:

[0014] Set the data attribute set as A:

[0015] A={a 1 ,a 2 ,…,a |A|}

[0016] Wherein, a |A| represents the |A|th data attribute in the data mining process;

[0017] Set the data mining condition set as B:

[0018] B={b 1 ,b 2 ,…,b |B|}

[0019] Wherein, b |B| represents the |B|th mining condition in the data mining process;

[0020] The mining judgment matrix P is obtained as follows:

[0021]

[0022] Wherein, represents the judgment basis in the judgment matrix according to the |A|th attribute and the |B|th condition;

[0023] The weight distribution of the mining judgment matrix P is as follows:

[0024]

[0025] Wherein, ω k represents the kth weight, δ is the weight adjustment factor, |A| k represents the total number of data of attribute k;

[0026] As a preferred technical solution of the present application: in S2, the weight adjustment factor is optimized based on the genetic algorithm:

[0027] Set the floating point number coding, select the roulette method as the fitness function f, and the possible parameters of the roulette are set as E,

[0028]

[0029] The selection operation is carried out according to the survival of the fittest criterion, the individual fitness is calculated, and the selected individual is subjected to the crossover operation, and the crossover method is as follows:

[0030]

[0031]

[0032] wherein, respectively represent the position of the ith individual in the t+1th and tth evolution, respectively represent the position of the jth individual in the t+1th and tth evolution, P c is a crossover probability, and t is the evolution number of the population;

[0033]

[0034] wherein, P c1 , P c2 is a crossover probability adjustment factor, f b is the larger fitness value of the two individuals participating in the crossover, f max is the maximum fitness value in the current population, is the average fitness value in the current population;

[0035] The mutation is replaced by the parent-offspring order comparison:

[0036] G new = max(G f , G c )

[0037] wherein, G new represents the comparison result of the parent-offspring chromosome fitness value, G f , G c respectively represent the parent-offspring chromosome fitness value.

[0038] x′ i = x i +x i ·rand(x b -x i )

[0039] wherein, x′ i represents the updated position of the ith individual, x i represents the mutated position of the ith individual, and x b is the spatial position corresponding to the best fitness individual.

[0040] As a preferred technical solution of the present application: in S2, the optimal weight adjustment factor obtained based on the genetic algorithm optimization is used to construct the weight matrix ω lAnd the weight matrix is input to the output weight matrix of the data of the RBF neural network, and the data is trained.

[0041] As a preferred technical solution of the application, the training is as follows:

[0042] Supposing that the dimensions of the input vector and the output vector of the RBF neural network are g and h respectively, that is, X=[X1, X2,..., Xg], y=[y1, y2,..., yh], the scale of the hidden layer is r, and the activation function of the hidden layer node is g . h The output ym of the RBF network at the mth node is as follows: m

[0043]

[0044] wherein, is a Gaussian radial basis function used in the network, c l is the center vector of the hidden layer node l in the RBF network, and sigma is the width of the RBF neural network.

[0045] The data is classified based on a classification algorithm:

[0046]

[0047] wherein, Y m represents the classification result of the mth output data, M represents the number of output data, and e is a mathematical constant.

[0048] The application provides a big data platform, comprising:

[0049] A data acquisition layer is configured to acquire data through a data transmission device.

[0050] A mining classification layer is configured to mine the acquired data to obtain classified data.

[0051] An integration and publishing layer is configured to integrate and publish the classified data.

[0052] The application provides a big data acquisition and publishing method and a big data platform, which have the following beneficial effects compared with the prior art:

[0053] The application can classify massive data according to user requirements through a data mining method, and can obtain the best weight matrix through the optimization of the weight adjustment factor by using an improved genetic algorithm, and can classify and integrate and publish the data through a classification function, so that the data collected in different ways can be processed, the applicability is high, the redundancy of the obtained data is low, and the quality is high. BRIEF DESCRIPTION OF DRAWINGS ​​

[0054] Figure 1 Method flow chart for preferred embodiments of the present application;

[0055] Figure 2 Platform architecture diagram for preferred embodiments of the present application.

[0056] The meanings of various marks in the figure are as follows: 100, data acquisition layer; 200, mining classification layer; 300, integration and publishing layer. DETAILED DESCRIPTION

[0057] It should be noted that the embodiments in the present embodiment and the features in the embodiments can be combined with each other without conflict, and the technical solutions in the embodiments of the present application will be described clearly and completely in combination with the drawings of the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0058] With reference to Figure 1 The preferred embodiments of the present application provide a big data acquisition and publishing method, comprising the following steps:

[0059] S1: collecting data through a data transmission device;

[0060] S2: mining and processing the collected data to obtain classified data;

[0061] S3: integrating and publishing the data obtained by classification processing.

[0062] In the S1, data is collected through sensor collection, crawler collection, input collection, import collection and interface collection.

[0063] In the S2, the collected data is also cleaned.

[0064] In the S2, the data obtained after data cleaning is processed based on a mining algorithm.

[0065] In the S2, a judgment matrix is obtained according to the properties of the collected data and mining conditions, and based on the judgment matrix, the data is trained and classified as an output weight matrix of an RBF neural network.

[0066] The mining algorithm in the S2 is as follows:

[0067] Let the data property set be A:

[0068] A={a 1 ,a 2 ,…,a|A|}

[0069] wherein a |A| represents the |A|th data attribute in the data mining process;

[0070] The set of data mining conditions is set as B:

[0071] B = {b 1 , b 2 , …, b |B|}

[0072] wherein b |B| represents the |B|th mining condition in the data mining process;

[0073] The mining judgment matrix P is obtained as follows:

[0074]

[0075] wherein, represents the judgment basis in the judgment matrix according to the |A|th attribute and the |B|th condition;

[0076] The weight distribution of the mining judgment matrix P is performed as follows:

[0077]

[0078] wherein ω k represents the kth weight, δ is the weight adjustment factor, |A| k represents the total sum of the data quantity of the attribute k;

[0079] The weight adjustment factor is optimized based on the genetic algorithm:

[0080] The floating point number coding is set, the roulette method is selected as the fitness function f, and the possible parameters of the roulette are set as E,

[0081]

[0082] The selection operation is performed according to the survival of the fittest criterion, the individual fitness is calculated, and the selected individuals are subjected to the crossover operation, and the crossover method is as follows:

[0083]

[0084]

[0085] wherein, respectively represent the position of the ith individual in the t+1th and tth evolution, respectively represent the position of the jth individual in the t+1th and tth evolution, P c is the crossover probability, and t is the evolution number of the population.

[0086]

[0087] where P c1 is the crossover probability adjustment factor, f c2 is the larger fitness value of the two individuals involved in the crossover, f b is the maximum fitness value in the current population, max is the average fitness value in the current population. is the average fitness value in the current population.

[0088] Replace mutation with parent-offspring order comparison:

[0089] G new = max(G f , G c )

[0090] where G new represents the comparison result of the parent and offspring chromosome fitness values, G f and G c represent the parent and offspring chromosome fitness values, respectively.

[0091] x′ i = x i + x i · rand(x b - x i )

[0092] where x′ i represents the updated position of the i-th individual, x i represents the mutated position of the i-th individual, and x b is the spatial position corresponding to the individual with the best fitness value.

[0093] In each generation, the fitness value of each individual is calculated, and the crossover probability P c is determined based on the relationship between this value and the average fitness value f of the population. c Therefore, for each individual, the corresponding crossover probability P c is the final adjustment factor δ.

[0094] In the iterative process of the genetic algorithm, individuals evolve through selection, crossover, and mutation operations, and the calculation of P c is updated according to the fitness of each generation. Finally, when the optimization process ends, the P l corresponding to the individual with the best fitness value is the δ value after optimization.

[0095] In S2, the optimal weight adjustment factor obtained based on the genetic algorithm optimization is used to construct the weight matrix ω lAnd the weight matrix is taken as the input of the data of the RBF neural network to train the data.

[0096] The training is as follows:

[0097] Suppose the dimensions of the input vector and the output vector of the RBF neural network are g and h respectively, that is, X = [X1, X2, …, Xg], y = [y1, y2, …, yh], the scale of the hidden layer is r, and the activation function of the hidden layer node is g . h The output ym of the RBF network at the mth node is: m .

[0098]

[0099] wherein, is a Gaussian radial basis function used in the network, c l is the center vector of the hidden layer node l in the RBF network, and sigma is the width of the RBF neural network.

[0100] The data is classified based on the classification algorithm:

[0101]

[0102] wherein, Y m represents the classification result of the mth output data, M represents the number of output data, and e is a mathematical constant.

[0103] Referring to Figure 2 , a big data platform is provided, comprising:

[0104] A data acquisition layer 100 is configured to acquire data through a data transmission device;

[0105] A mining and classification layer 200 is configured to mine the acquired data to obtain classified data;

[0106] An integration and publishing layer 300 is configured to integrate and publish the classified data.

[0107] In this embodiment, the data acquisition layer 100 can acquire data in the manners of sensor acquisition, crawler acquisition, input acquisition, import acquisition and interface acquisition. Taking the data acquired by the sensor as an example, the data acquired by the sensor is cleaned to remove abnormal data or repeated and unreasonable sensor data. The sensor data obtained after data cleaning is processed by the mining and classification layer 200 based on a mining algorithm. Suppose the data attribute set of the acquired sensor data is A:

[0108] A = {a 1 ,a​2 ,…,a |A|}

[0109] wherein a |A| represents the |A|th data attribute in the data mining process;

[0110] Set the data mining condition set this time as B:

[0111] B = {b 1 ,b 2 ,…,b |B|}

[0112] wherein b |B| represents the |B|th mining condition in the data mining process;

[0113] Get the mining judgment matrix P as follows:

[0114]

[0115] wherein, represents the judgment basis in the judgment matrix according to the |A|th attribute and the |B|th condition;

[0116] Weight distribution is performed on the mining judgment matrix P:

[0117]

[0118] wherein ω k represents the kth weight, δ is the weight adjustment factor, |A| k represents the total data quantity of attribute k;

[0119] Optimization of the weight adjustment factor is performed based on the genetic algorithm:

[0120] Set the floating-point number coding, select the roulette method as the fitness function f, and set the possible parameters of the roulette as E,

[0121]

[0122] Selection operation is performed according to the survival of the fittest criterion, the individual fitness is calculated, and the selected individuals are subjected to crossover operation, and the crossover method is as follows:

[0123]

[0124]

[0125] wherein, respectively represent the positions of the ith individual in the 9th and 8th evolution, respectively represent the positions of the jth individual in the 9th and 8th evolution, P cLet the crossover probability be denoted as ; an adaptive crossover strategy is introduced to ensure that the optimal crossover probability is achieved in each evolution:

[0126]

[0127] Among them, P c1 P c2 f is the crossover probability adjustment factor. b f is the larger fitness value among the two individuals participating in the crossover. max This represents the maximum fitness value in the current population. This represents the average fitness value in the current population. Traditional genetic algorithms perform random mutations on individual codes after crossover and introduce new genes into the population. However, since floating-point encoding is used, mutation can affect the stability of population iteration. Therefore, a method of comparing parent and offspring generations will be used instead of mutation.

[0128] Replace variation with a parent-child generation sequential comparison method:

[0129] G new =max(G f G c )

[0130] Among them, G new This represents the comparison of chromosome fitness values ​​between the parent and offspring, G. f G c These represent the chromosome fitness values ​​of the parent and offspring, respectively.

[0131] x′ i =x i +x i rand(x) b -x i )

[0132] Where, x′ i Let x represent the final update position of the i-th individual. i x represents the position of the mutation in the i-th individual. b This represents the spatial location of the individual with the best fitness. By updating the location, while maintaining population diversity, we ensure that individuals move closer to the optimal individual after this iteration.

[0133] Traditional genetic algorithms suffer from subjective selection, slow convergence speed, and unstable error. The genetic algorithms described above improve upon these shortcomings.

[0134] The weight matrix ω is constructed based on the optimal weight adjustment factor δ0 obtained by the genetic algorithm. land the weight matrix is taken as the input of the RBF neural network to the output weight matrix of the sensor data, where the dimensions of the input vector and the output vector of the RBF neural network are g and h respectively, i.e. g ], y = [y1, y2, …, y h ], the scale of the hidden layer is r, and the activation function of the hidden layer node is The output y m of the RBF network at the 5th node is:

[0135]

[0136] wherein, is a Gaussian radial basis function used in the network, c l is the center vector of the hidden layer node l in the RBF network, and σ is the width of the RBF neural network.

[0137] The sensor data is classified based on the classification algorithm:

[0138]

[0139] wherein Y5 represents the classification result of the 5th output data, M represents the number of output data, and e is a mathematical constant.

[0140] The data is integrated and published by the integration and publishing layer 300 based on the classification result of each data.

[0141] It is apparent to those skilled in the art that the present application is not limited to the details of the foregoing exemplary embodiments, and that the present application can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application. Therefore, the embodiments should be considered in all respects as illustrative and not restrictive, the scope of the present application being defined by the appended claims rather than by the foregoing description, and it is intended that all changes coming within the meaning and equivalency range of the claims are intended to be embraced therein. Any reference signs in the claims should not be construed as limiting the claims to the figures in which the reference signs are used.

[0142] In addition, it should be understood that although the present specification is described in terms of embodiments, each embodiment does not necessarily contain only one independent technical solution, and the description of the specification is only for the sake of clarity, and those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can be appropriately combined to form other embodiments that those skilled in the art can understand.

Claims

1. A big data collection and distribution method, characterized in that: Comprise the following steps: S1: collecting data through a data transmission device; S2: mining the collected data to obtain classified data; S3: integrating and classifying the data obtained by classification processing for publishing; In S2, the data is mined according to the attributes of the collected data and the mining conditions to obtain a judgment matrix, and based on the judgment matrix, the data is trained and classified as the output weight matrix of the RBF neural network; The mining algorithm in S2 is as follows: The set of data attributes is set to : wherein, represents the i-th data attribute in the data mining process; represents the i-th data attribute in the data mining process; The set of data mining condition setting data is : wherein, represents the i-th mining condition in the data mining process; represents the i-th mining condition in the data mining process; obtaining a digested judgment matrix as follows: wherein, represents a judgment basis in the judgment matrix formed according to the i-th attribute and the j-th condition; represents a judgment basis in the judgment matrix formed according to the i-th attribute and the j-th condition;​ To mine a judgment matrix Perform weight distribution: wherein, represents the first weight, is a weight adjustment factor, represents the sum of the data quantities of the attributes In S2, the weight adjustment factor is optimized based on a genetic algorithm: Set the float number encoding, select the roulette method as the fitness function , the possible parameter settings of the roulette are E, The selection operation is performed according to the survival of the fittest criteria, the individual fitness is calculated, and the selected individuals are subjected to crossover operation, and the crossover method is as follows: wherein, , denote the positions of the first individuals in the first and the second evolution, respectively, , denote the positions of the first individuals in the first and the second evolution, respectively, is the crossover probability, is the number of population evolution; wherein, , is a crossover probability adjustment factor, is the larger fitness value of the two individuals involved in the crossover, is the maximum fitness value in the current population, is the average fitness value in the current population; Replace mutation with parent-child sequence comparison: wherein, represents the comparison result of the chromosome fitness value of the parent and the child, , respectively represent the chromosome fitness value of the parent and the child; wherein, represents the position of the final individual, represents the position of the mutated individual, is the spatial position corresponding to the best fitness individual;​​ In the S2, the optimal weight adjustment factor obtained based on the genetic algorithm optimization is used to construct a weight matrix and the weight matrix is used as the input of the RBF neural network to obtain the output weight matrix of the data, and the data is trained. The training is as follows: Let the dimension of the input vector and the output vector of the RBF neural network be , , , , the scale of the hidden layer be , and the activation function of the hidden layer node be . The output of the RBF network at the th node is wherein, is a Gaussian radial basis function used in the network, is a center vector of a hidden layer node in the RBF network, is a width of the RBF neural network; Classify the data based on the classification algorithm: wherein, represents the classification result of the output data, represents the number of output data, is a mathematical constant.

2. The big data collection and distribution method of claim 1, wherein: In S1, data is collected through sensor collection, crawler collection, input collection, import collection, and interface collection.

3. The big data collection and distribution method of claim 2, wherein: In S2, the collected data is also cleaned.

4. The big data collection and distribution method of claim 3, wherein: In S2, the data obtained after data cleaning is mined based on a mining algorithm.

5. A big data platform based on the big data collection and distribution method of any one of claims 1-4. Comprise: Data collection layer (100): for collecting data through a data transmission device; Mining and classification layer (200): for mining the collected data to obtain classified data; Integration and publishing layer (300): for integrating and classifying the data obtained by classification processing for publishing.

Images