Machine learning model training method under open data

Abstract

The invention relates to the technical field of model training under open data, and discloses a machine learning model training method under open data, and the method comprises the steps: carrying out the preprocessing and semantic feature extraction of an open data set, and obtaining the semantic feature vector of each group of open data; clustering the open data based on the semantic feature vectors of the open data; and taking different types of open data sets as training data sets for training to obtain multiple groups of local optimization parameters, and performing global asynchronous parameter optimization. According to the method, clustering processing is carried out based on the semantic similarity between the open data, splitting processing of large-scale open data is achieved, a parallel simultaneous training mode is adopted, a machine learning model is trained, multiple sets of local optimization parameters are obtained, and the algorithm is more efficient. And realizing global asynchronous optimization processing by adopting an asynchronous parameter optimization mode which does not need to wait for iteration completion of all local optimization parameters, so as to obtain optimal machine learning model parameters suitable for multiple scenes.

Description

Technical Field

[0001] The present invention relates to the technical field of model training under open data, and in particular to a machine learning model training method under open data. Background Art

[0002] In today's era of information explosion, data is considered to be one of the most valuable assets. A large number of data sources - including social media, sensors, cloud storage, open data, etc. - continue to emerge, and these data carry rich information and insights. Open data refers to publicly accessible and reusable data sets with a wide range of sources and diverse application scenarios. Government departments, scientific research institutions, enterprises and individuals can obtain more information, insights and innovations through open data. Using open data for machine learning model training can not only improve the accuracy and generalization ability of the model, but also promote information sharing and social development. However, open data is usually huge in scale and uneven in quality. How to effectively use these data to train high-quality models is an important issue. In response to this problem, the present invention proposes a machine learning model training method under open data, which integrates multi-source heterogeneous open data to enhance data quality and improve the effect and generalization ability of the model. Summary of the invention

[0003] In view of this, the present invention provides a machine learning model training method under open data, the purpose of which is to: 1) encode the collected open data in combination with context encoding information, and calculate the attention weight in combination with the frequency of occurrence of different word segmentation results in the open data to realize semantic feature extraction of the open data, and calculate the aggregation degree of the open data based on the number of neighboring data between the open data and the average distance. The higher the aggregation degree, the more neighboring data of the open data and the greater the density, which improves the clustering efficiency of the clustering algorithm, obtains open data sets of different categories, and realizes the splitting processing of large-scale open data; 2) Use the open data sets of different categories as training data sets respectively , a parallel simultaneous training method is adopted to train the machine learning model, and the improvement of the model parameter training effect is used as the iteration reward value during the model parameter training and solution process. The model parameters are iterated based on the gradient information of the model parameters. The iterative process of the model parameters and the retained gradient information constitute multiple groups of local optimization parameters. Combined with the link coefficients between the training data sets corresponding to the local optimization parameters, an adaptive step size for global asynchronous parameter optimization is constructed to achieve asynchronous parameter optimization without waiting for the completion of all local optimization parameter iterations. Parallel model parameter training of multi-category open data and global asynchronous optimization processing are realized to obtain the optimal machine learning model parameters suitable for multiple scenarios.

[0004] To achieve the above object, the present invention provides a machine learning model training method under open data, comprising the following steps:

[0005] S1: Collect open data sets, preprocess and extract semantic features of the open data sets, and obtain semantic feature vectors for each set of open data;

[0006] S2: Cluster the open data based on its semantic feature vectors to obtain different categories of open data sets;

[0007] S3: Using different categories of open data sets as training data sets, training the machine learning model to obtain multiple sets of local optimization parameters, where the parallel search strategy is the main implementation method of the machine learning model;

[0008] S4: Perform global asynchronous parameter optimization on multiple groups of local optimization parameters to obtain optimal machine learning model parameters, wherein the global asynchronous optimization strategy with adaptive step size is the main implementation method of the global asynchronous parameter optimization.

[0009] As a further improvement method of the present invention:

[0010] Optionally, in step S1, collecting an open data set, preprocessing the open data set, and extracting semantic features, include:

[0011] The collection of open data constitutes an open data set, where open data is publicly accessible and reusable data. The form of the open data set is:

[0012] data = {x n |n∈[1,N]}

[0013] in:

[0014] data represents an open data collection;

[0015] x n represents the nth group of open data in the open data set, and N represents the total number of open data in the open data set;

[0016] Each set of open data in the open data set is preprocessed and semantic features are extracted to obtain the semantic feature vector of each set of open data, where the nth set of open data x n The preprocessing and semantic feature extraction process is as follows:

[0017] S11: Construct a word segmentation dictionary, and use the forward maximum matching method based on the word segmentation dictionary to search the nth group of open data x n Perform word segmentation to obtain the nth group of open data x n The word segmentation result sequence:

[0018]

[0019] in:

[0020] Represents the nth group of open data x n The i-th word segmentation result, num n Represents the nth group of open data x n The total number of word segmentation results, i∈[1,num n ];

[0021] S12: Open data x for the nth group n The segmentation result sequence is encoded to obtain the nth group of open data x n The encoded data sequence is used as the nth group of open data x n The preprocessing result of , where the calculation process of the encoded data sequence is:

[0022]

[0023] in:

[0024] X n Represents the nth group of open data x n The coded data sequence, Indicates word segmentation results The encoding processing result;

[0025] exp(·) represents an exponential function with a natural constant as the base;

[0026] W represents the encoding matrix of the word segmentation result, and * represents convolution processing;

[0027] Onehot(·) means one-hot encoding of the word segmentation results;

[0028] Indicates word segmentation results The above coded information, Indicates word segmentation results The following coded information;

[0029] S13: Encode the data sequence X n Perform semantic feature extraction to obtain the nth group of open data x n The semantic feature vector f n .

[0030] Optionally, the step S13 is to encode the data sequence X n Perform semantic feature extraction, including:

[0031] S131: Extract the coded data sequence X x The probability representation parameter of any encoding processing result in The probability representation parameters are:

[0032]

[0033] in:

[0034] Indicates word segmentation results In the nth group of open data x n The frequency of occurrence in Indicates word segmentation results The frequency of occurrence in N sets of open data;

[0035] Indicates the result of encoding processing The probability of represents the parameter;

[0036] S132: Combine the probability representation parameters of the encoding processing results to calculate the encoded data sequence X n The attention weight of any encoding processing result in The attention weight calculation formula is:

[0037]

[0038] in:

[0039] Indicates the encoding processing result The attention weight of

[0040] S133: Generate encoded data sequence X based on attention weights n The corresponding semantic feature vector f n :

[0041]

[0042]

[0043] in:

[0044] It means element-by-element addition;

[0045] f n (i) indicates the encoding processing result semantic information;

[0046] ReLU(·) denotes the ReLU activation function.

[0047] Optionally, clustering the open data based on the semantic feature vectors of the open data in step S2 includes:

[0048] Based on the semantic feature vectors of open data, open data are clustered to obtain K clusters. All open data in the clusters are regarded as open data of the same category to form a set of open data of K categories. The clustering process is as follows:

[0049] S21: Calculate the distance between different open data, where the nth group of open data x n With the qth group of open data x q The distance between them is:

[0050]

[0051] in:

[0052] dis(x n ,x q ) represents the nth group of open data x n With the qth group of open data x q The distance between q represents the qth group of open data x q Semantic feature vector, q∈[1,N];

[0053] ||·||2 represents the L2 norm;

[0054] Represents the semantic feature vector f n and the semantic feature vector f q The similarity between

[0055] S22: Filter and obtain the M groups of open data closest to each group of open data, where the open data x n The nearest M groups of open data represent the open data x n The M groups of open data with the closest distance are counted, and the number of the most adjacent open data of each group of open data is counted, where the open data x n The number of mutually nearest open data is count(x n ), indicating open data x n Among the M groups of open data closest to , there are count(x n ) The nearest neighbor of the group of open data is the M group of open data that contains the open data x n ;

[0056] S23: Calculate the aggregation degree of each group of open data, where the aggregation degree of the nth group of open data is:

[0057]

[0058] in:

[0059] δ n represents the aggregation degree of the nth group of open data;

[0060] S24: Select K groups of open data with the highest aggregation degree as cluster centers, and each cluster center corresponds to a cluster;

[0061] S25: Calculate the distance between the open data corresponding to the non-clustering center and the clustering center, and merge the open data corresponding to the non-clustering center into the cluster cluster where the nearest clustering center is located;

[0062] S26: Calculate the mean of the semantic feature vectors corresponding to all the open data in the cluster, and select the open data corresponding to the semantic feature vector with the highest similarity to the mean of the semantic feature vector in the cluster as the update result of the cluster center in the cluster, and return to step S25; until the K cluster centers do not change, forming K clusters;

[0063] S27: Calculate the link coefficient between each cluster, where the link coefficient between the kth cluster and the mth cluster is:

[0064]

[0065] Link(k,m)=Link(m,k)

[0066] in:

[0067] Link(k,m) represents the link coefficient between the kth cluster and the mth cluster, m∈[1,K],m≠k;

[0068] AVG1(k,m) represents the average distance from any open data in the kth cluster to all open data in the mth cluster, and AVG2(k) represents the average distance from any open data in the kth cluster to other open data in the cluster.

[0069] Optionally, in step S3, different categories of open data sets are used as training data sets to train the machine learning model, including:

[0070] Different categories of open data sets are used as training data sets to form K training data sets, and the link coefficients between clusters are used as the link coefficients between the training data sets associated with the clusters, that is, the link coefficient between the kth training data set and the mth training data set is Link(k,m), where the construction process of the kth training data set is:

[0071] Get the open data set of the kth category:

[0072] data k ={y k (d)|d∈[1,Sum k ]}

[0073] in:

[0074] data k represents the open data set of the kth category;

[0075] y k (d) represents open data set data k The dth group of open data in Sum k Represents an open data set data k The total number of open data in;

[0076] Open Data k (d) The corresponding semantic feature vector is f[y k (d)];

[0077] Label all open data in the open data set of the kth category, where the open data y k (d) The label information marked is L(y k (d));

[0078] Construct the kth training data set:

[0079] Data k ={(y k (d),L(y k (d)))|d∈[1,Sum k ]}

[0080] in:

[0081] Data k represents the kth training data set;

[0082] (y k (d),L(y k (d)) represents the training data set Data k The dth group of training data in ;

[0083] Based on K training data sets, the machine learning model is trained to obtain K groups of local optimization parameters.

[0084] Optionally, the training of the machine learning model based on K training data sets to obtain K groups of local optimization parameters includes:

[0085] Based on K training data sets, the machine learning model is trained, and the training process is as follows:

[0086] S31: Distribute K training data sets to K computing devices;

[0087] S32: The computing device constructs a training objective function and initializes and generates the same model parameters, wherein the training objective function constructed by the k-th computing device is:

[0088]

[0089] in:

[0090] F k (θ k ) represents the training objective function constructed by the kth computing device, θ k represents the model parameters to be trained for the kth computing device;

[0091] Indicates that data will be opened k (d) As input, based on the model parameters θ k The output labels of the constructed model;

[0092] φ represents a label set, a∈φ, a represents any label in the label set φ;

[0093] Represents the matrix parameters corresponding to the label a. The matrix parameters corresponding to all labels in the label set φ constitute the parameters θ of the model to be trained. k ;

[0094] G(y k (d); θ k ; a) means to open data y k (d) As input, based on the model parameters θ k The constructed model outputs the probability of label a;

[0095] Indicates the label with the highest probability of selection;

[0096] Y k (d) represents open data y k (d) the corresponding coded data sequence;

[0097] The model parameters of the kth model generated by initialization are θ k (0), where:

[0098] θ1(0)=θ2(0)=...=θ k (0) = ... = θ K (0)

[0099] S33: Set the current iteration number of the model parameter to t, the initial value of t is 0, and the maximum value is Max, then the model parameter θ k The result of the tth iteration is θ k (t);

[0100] S34: Calculate the iterative reward value of the model parameters, where the model parameters θ k The iterative reward value of (t) is:

[0101]

[0102] in:

[0103] τ k (t) represents the model parameter θ k (t) iterative reward value;

[0104] F k (θ k (t)) represents the model parameter θ k (t) as the training objective function F k (·) input, output value of training objective function;

[0105] S35: Calculate the gradient information of the model parameters, where the model parameters θ k The gradient information of (t) is grad k (t), where the gradient information grad k (t) is calculated as follows: k Train the objective function F for the variable pair k (θ k ) is derived and the model parameter θ k Substitute (t) into the derivative result to obtain the model parameter θ k The gradient information grad of (t) k (t);

[0106] S36: Iterate the model parameters based on the iterative reward value and gradient information, where the model parameter θ k The iterative formula of (t) is:

[0107]

[0108] in:

[0109] T stands for transpose;

[0110] S37: Let t=t+1, and return to step S34 until the maximum number of iterations Max is reached; and a local optimization parameter is formed, wherein the local optimization parameter calculated by the k-th computing device is:

[0111] λ k =[(θ k (0),grad k (0)),(θ k (1),grad k(1)),...,(θ k (Max),grad k (Max))]

[0112] in:

[0113] λ k represents the local optimization parameter calculated by the kth computing device;

[0114] The computing device that completes the training immediately sends the local optimization parameters to the server for global asynchronous parameter optimization.

[0115] Optionally, in step S4, performing global asynchronous parameter optimization on multiple groups of local optimization parameters includes:

[0116] The server performs global asynchronous parameter optimization on the local optimization parameters calculated in step S3, wherein the global asynchronous parameter optimization process is as follows:

[0117] S41: Obtain initialization parameters θ(0), where θ(0)=θ1(0)=θ2(0)=...=θ k (0) = ... = θ K (0);

[0118] S42: Set the number of currently received local optimization parameters to u, the initial value of u is 0, and the global asynchronous parameter optimization result based on the u groups of local optimization parameters is θ(u);

[0119] S43: receiving the u+1th group of local optimization parameters λ(u+1), where λ(u+1) represents the u+1th group of local optimization parameters received by the server;

[0120] S44: Based on the u+1th group of local optimization parameters λ(u+1) received by the server, the global asynchronous parameter optimization result θ(u) based on the u group of local optimization parameters is optimized, wherein the optimization formula is:

[0121]

[0122] in:

[0123] Link u,u+1 represents the link coefficient between the training data set corresponding to the u+1th group of local optimization parameters λ(u+1) received by the server and the uth group of local optimization parameters λ(u);

[0124] λ Max (u+1) represents the model parameters obtained by the Max-th iteration in the local optimization parameter λ(u+1);

[0125] avg(grad,u+1) represents the mean of the gradient information corresponding to the Max+1 model parameter iteration results in the local optimization parameter λ(u+1);

[0126] is the adaptive step size corresponding to the local optimization parameter λ(u+1);

[0127] S45: Let u=u+1, return to step S43, until K groups of local optimization parameters are received, and take θ(K) as the model parameter optimization result under open data, that is, the optimal machine learning model parameters.

[0128] In order to solve the above problem, the present invention provides an electronic device, the electronic device comprising:

[0129] A memory storing at least one instruction;

[0130] Communication interface, enabling electronic equipment to communicate; and

[0131] A processor executes instructions stored in the memory to implement the machine learning model training method under open data described above.

[0132] In order to solve the above problems, the present invention also provides a computer-readable storage medium, in which at least one instruction is stored, and the at least one instruction is executed by a processor in an electronic device to implement the machine learning model training method under open data described above.

[0133] Compared with the prior art, the present invention proposes a machine learning model training method under open data, which has the following advantages:

[0134] First, this scheme proposes a semantic feature vector extraction method, which preprocesses and extracts semantic features from each set of open data in the open data set to obtain the semantic feature vector of each set of open data, where the nth set of open data x n The preprocessing and semantic feature extraction process is as follows: construct a word segmentation dictionary, and use the forward maximum matching method based on the word segmentation dictionary to search the nth group of open data x n Perform word segmentation to obtain the nth group of open data x n The word segmentation result sequence:

[0135]

[0136] in: Represents the nth group of open data x n The i-th word segmentation result, num n Represents the nth group of open data x n The total number of word segmentation results, i∈[1,num n]; for the nth group of open data x n The segmentation result sequence is encoded to obtain the nth group of open data x n The encoded data sequence is used as the nth group of open data x n The preprocessing result of , where the calculation process of the encoded data sequence is:

[0137]

[0138] Where: X n Represents the nth group of open data x n The coded data sequence, Indicates word segmentation results The encoding result of ; exp(·) represents an exponential function with a natural constant as the base; W represents the encoding matrix of the word segmentation result, * represents convolution processing; onehot(·) represents the one-hot encoding representation of the word segmentation result; Indicates word segmentation results The above coded information, Indicates word segmentation results The following coded information; for the coded data sequence X n Perform semantic feature extraction to obtain the nth group of open data x n The semantic feature vector f n . Extract the coded data sequence X n The probability representation parameter of any encoding processing result in The probability representation parameters are:

[0139]

[0140] in: Indicates word segmentation results In the nth group of open data x n The frequency of occurrence in Indicates word segmentation results The frequency of occurrence in N sets of open data; Indicates the encoding processing result The probability representation parameter of the encoding processing result is combined with the probability representation parameter to calculate the encoded data sequence X n The attention weight of any encoding processing result in The attention weight calculation formula is:

[0141]

[0142] in: Indicates the encoding processing result Attention weights; Generate encoded data sequence X based on attention weights nThe corresponding semantic feature vector f n :

[0143]

[0144]

[0145] in: represents element-by-element addition; f n (i) indicates the encoding processing result The semantic information of the word; ReLU (·) represents the ReLU activation function. This scheme encodes the collected open data in combination with the contextual encoding information, and calculates the attention weight in combination with the frequency of occurrence of different word segmentation results in the open data, so as to realize the semantic feature extraction of the open data. Based on the number of neighboring data and the average distance between the open data, the aggregation degree of the open data is calculated. The higher the aggregation degree, the more neighboring data of the open data and the greater the density, which improves the clustering efficiency of the clustering algorithm, obtains open data sets of different categories, and realizes the splitting and processing of large-scale open data.

[0146] At the same time, this solution proposes a parallel training method for multiple training sets. Based on K training data sets, the machine learning model is trained. The training process is: distribute the K training data sets to K computing devices; the computing devices construct the training objective function and initialize to generate the same model parameters. The training objective function constructed by the kth computing device is:

[0147]

[0148] Among them: F k (θ k ) represents the training objective function constructed by the kth computing device, θ k represents the model parameters to be trained for the kth computing device; Indicates that data will be opened k (d) As input, based on the model parameters θ k The output label of the constructed model; φ represents the label set, a∈φ, a represents any label in the label set φ; Represents the matrix parameters corresponding to the label a. The matrix parameters corresponding to all labels in the label set φ constitute the parameters θ of the model to be trained. k ; G(y k

[0149] (d); θ k; a) means to open data y k (d) As input, based on the model parameters θ k The constructed model outputs the probability of label a; Indicates the label with the highest probability of selection; Y k (d) represents open data y k (d) The corresponding encoded data sequence; the model parameters of the k-th model generated by initialization are θ k (0), where:

[0150] θ1(0)=θ2(0)=...=θ k (0) = ... = θ k (0)

[0151] Set the current iteration number of the model parameter to t, the initial value of t is 0, and the maximum value is Max, then the model parameter θ k The result of the tth iteration is θ k (t); Calculate the iterative reward value of the model parameters, where the model parameters θ k The iterative reward value of (t) is:

[0152]

[0153] Where: τ k (t) represents the model parameter θ k (t) iterative reward value; F k (θ k (t)) represents the model parameter θ k (t) as the training objective function F k (·) input, the output value of the training objective function; the gradient information of the model parameters is calculated, where the model parameter θ k The gradient information of (t) is grad k (t), where the gradient information grad k (t) is calculated as follows: k Train the objective function F for the variable pair k (θ k ) is derived and the model parameter θ k Substitute (t) into the derivative result to obtain the model parameter θ k The gradient information grad of (t) k (t); Iterate the model parameters based on the iterative reward value and gradient information, where the model parameters θ k The iterative formula of (t) is:

[0154]

[0155] Where: T represents transposition; let t = t + 1, until the maximum number of iterations Max is reached; and constitute the local optimization parameters, where the local optimization parameters calculated by the k-th computing device are:

[0156] λk =[(θ k (0),grad k (0)),(θ k (1),grad k (1)),...,(θ k (Max),grad k (Max))]

[0157] Where: k Represents the local optimization parameters calculated by the kth computing device; the computing device that has completed the training immediately sends the local optimization parameters to the server for global asynchronous parameter optimization. This solution uses different categories of open data sets as training data sets, and adopts parallel and simultaneous training to train the machine learning model. In the process of model parameter training and solution, the improvement of the model parameter training effect is used as the iteration reward value, and the model parameters are iterated based on the gradient information of the model parameters. The iteration process of the model parameters and the retained gradient information constitute multiple groups of local optimization parameters. Combined with the link coefficients between the training data sets corresponding to the local optimization parameters, an adaptive step size for global asynchronous parameter optimization is constructed to achieve asynchronous parameter optimization without waiting for all local optimization parameter iterations to be completed. Parallel model parameter training of multiple categories of open data and global asynchronous optimization processing are achieved to obtain the optimal machine learning model parameters suitable for multiple scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0158] Figure 1 A flowchart of a method for training a machine learning model under open data provided by one embodiment of the present invention;

[0159] Figure 2 A schematic diagram of the structure of an electronic device for implementing a machine learning model training method under open data provided by one embodiment of the present invention.

[0160] In the figure: 1 electronic device, 10 processor, 11 memory, 12 program, 13 communication interface.

[0161] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0162] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0163] The embodiment of the present application provides a machine learning model training method under open data. The execution subject of the machine learning model training method under open data includes but is not limited to at least one of the electronic devices such as a server and a terminal that can be configured to execute the method provided by the embodiment of the present application. In other words, the machine learning model training method under open data can be executed by software or hardware installed on a terminal device or a server device, and the software can be a blockchain platform. The server includes but is not limited to: a single server, a server cluster, a cloud server or a cloud server cluster, etc.

[0164] Embodiment 1:

[0165] S1: Collect open data sets, preprocess the open data sets and extract semantic features to obtain semantic feature vectors for each set of open data.

[0166] In the step S1, an open data set is collected, and preprocessed and semantic features are extracted from the open data set, including:

[0167] The collection of open data constitutes an open data set, where open data is publicly accessible and reusable data. The form of the open data set is:

[0168] data = {x n |n∈[1,N]}

[0169] in:

[0170] data represents an open data collection;

[0171] x n represents the nth group of open data in the open data set, and N represents the total number of open data in the open data set;

[0172] Each set of open data in the open data set is preprocessed and semantic features are extracted to obtain the semantic feature vector of each set of open data, where the nth set of open data x n The preprocessing and semantic feature extraction process is as follows:

[0173] S11: Construct a word segmentation dictionary, and use the forward maximum matching method based on the word segmentation dictionary to search the nth group of open data x n Perform word segmentation to obtain the nth group of open data x n The word segmentation result sequence:

[0174]

[0175] in:

[0176] Represents the nth group of open data x nThe i-th word segmentation result, num n Represents the nth group of open data x n The total number of word segmentation results, i∈[1,num n ];

[0177] S12: Open data x for the nth group n The segmentation result sequence is encoded to obtain the nth group of open data x n The encoded data sequence is used as the nth group of open data x n The preprocessing result of , where the calculation process of the encoded data sequence is:

[0178]

[0179]

[0180] in:

[0181] X n Represents the nth group of open data x n The coded data sequence, Indicates word segmentation results The encoding processing result;

[0182] exp(·) represents an exponential function with a natural constant as the base;

[0183] W represents the encoding matrix of the word segmentation result, and * represents convolution processing;

[0184] Onehot(·) means one-hot encoding of the word segmentation results;

[0185] S13: Encode the data sequence X n Perform semantic feature extraction to obtain the nth group of open data x n The semantic feature vector f n .

[0186] The step S13 encodes the data sequence X n Perform semantic feature extraction, including:

[0187] S131: Extract the coded data sequence X n The probability representation parameter of any encoding processing result in The probability representation parameters are:

[0188]

[0189] in:

[0190] Indicates word segmentation results In the nth group of open data x nThe frequency of occurrence in Indicates word segmentation results The frequency of occurrence in N sets of open data;

[0191] Indicates the encoding processing result The probability of represents the parameter;

[0192] S132: Combine the probability representation parameters of the encoding processing results to calculate the encoded data sequence X n The attention weight of any encoding processing result in The attention weight calculation formula is:

[0193]

[0194] in:

[0195] Indicates the encoding processing result The attention weight of

[0196] S133: Generate encoded data sequence X based on attention weights n The corresponding semantic feature vector f n :

[0197]

[0198] in:

[0199] It means element-by-element addition;

[0200] f n (i) indicates the encoding processing result semantic information;

[0201] ReLU(·) denotes the ReLU activation function.

[0202] S2: Cluster the open data based on its semantic feature vectors to obtain open data sets of different categories.

[0203] The step S2 clusters the open data based on the semantic feature vectors of the open data, including:

[0204] Based on the semantic feature vectors of open data, open data are clustered to obtain K clusters. All open data in the clusters are regarded as open data of the same category to form a set of open data of K categories. The clustering process is as follows:

[0205] S21: Calculate the distance between different open data, where the nth group of open data x n With the qth group of open data xq The distance between them is:

[0206]

[0207] in:

[0208] dis(x n ,x q ) represents the nth group of open data x n With the qth group of open data x q The distance between q represents the qth group of open data x q Semantic feature vector, q∈[1,N];

[0209] ||·||2 represents the L2 norm;

[0210] Represents the semantic feature vector f n and the semantic feature vector f q The similarity between

[0211] S22: Filter and obtain the M groups of open data closest to each group of open data, where the open data x n The nearest M groups of open data represent the open data x n The M groups of open data with the closest distance are counted, and the number of the most adjacent open data of each group of open data is counted, where the open data x n The number of mutually nearest open data is count(x n ), indicating open data x n Among the M groups of open data closest to , there are count(x n ) The nearest neighbor of the group of open data is the M group of open data that contains the open data x n ;

[0212] S23: Calculate the aggregation degree of each group of open data, where the aggregation degree of the nth group of open data is:

[0213]

[0214] in:

[0215] δ n represents the aggregation degree of the nth group of open data;

[0216] S24: Select K groups of open data with the highest aggregation degree as cluster centers, and each cluster center corresponds to a cluster;

[0217] S25: Calculate the distance between the open data corresponding to the non-clustering center and the clustering center, and merge the open data corresponding to the non-clustering center into the cluster cluster where the nearest clustering center is located;

[0218] S26: Calculate the mean of the semantic feature vectors corresponding to all the open data in the cluster, and select the open data corresponding to the semantic feature vector with the highest similarity to the mean of the semantic feature vector in the cluster as the update result of the cluster center in the cluster, and return to step S25; until the K cluster centers do not change, forming K clusters;

[0219] S27: Calculate the link coefficient between each cluster, where the link coefficient between the kth cluster and the mth cluster is:

[0220]

[0221] Link(k,m)=Link(m,k)

[0222] in:

[0223] Link(k,m) represents the link coefficient between the kth cluster and the mth cluster, m∈[1,K],m≠k;

[0224] AVG1(k,m) represents the average distance from any open data in the kth cluster to all open data in the mth cluster, and AVG2(k) represents the average distance from any open data in the kth cluster to other open data in the cluster.

[0225] S3: Use different categories of open data sets as training data sets to train the machine learning model and obtain multiple sets of local optimization parameters.

[0226] In the step S3, different categories of open data sets are used as training data sets to train the machine learning model, including:

[0227] Different categories of open data sets are used as training data sets to form K training data sets, and the link coefficients between clusters are used as the link coefficients between the training data sets associated with the clusters, that is, the link coefficient between the kth training data set and the mth training data set is Link(k,m), where the construction process of the kth training data set is:

[0228] Get the open data set of the kth category:

[0229] data k ={y k (d)|d∈[1,Sum k ]}

[0230] in:

[0231] data krepresents the open data set of the kth category;

[0232] y k (d) represents open data set data k The dth group of open data in Sum k Represents an open data set data k The total number of open data in;

[0233] Open Data k (d) The corresponding semantic feature vector is f[y k (d)];

[0234] Label all open data in the open data set of the kth category, where the open data y k (d) The label information marked is L(y k (d));

[0235] Construct the kth training data set:

[0236] Data k ={(y k (d),L(y k (d)))|d∈[1,Sum k ]}

[0237] in:

[0238] Data k represents the kth training data set;

[0239] (y k (d),L(y k (d)) represents the training data set Data k The dth group of training data in ;

[0240] Based on K training data sets, the machine learning model is trained to obtain K groups of local optimization parameters.

[0241] The machine learning model is trained based on K training data sets to obtain K groups of local optimization parameters, including:

[0242] Based on K training data sets, the machine learning model is trained, and the training process is as follows:

[0243] S31: Distribute K training data sets to K computing devices;

[0244] S32: The computing device constructs a training objective function and initializes and generates the same model parameters, wherein the training objective function constructed by the k-th computing device is:

[0245]

[0246] in:

[0247] F k (θ k ) represents the training objective function constructed by the kth computing device, θ k represents the model parameters to be trained for the kth computing device;

[0248] Indicates that data will be opened k (d) As input, based on the model parameters θ k The output labels of the constructed model;

[0249] φ represents a label set, a∈φ, a represents any label in the label set φ;

[0250] Represents the matrix parameters corresponding to the label a. The matrix parameters corresponding to all labels in the label set φ constitute the parameters θ of the model to be trained. k ;

[0251] G(y k (d); θ k ; a) means to open data y k (d) As input, based on the model parameters θ k The constructed model outputs the probability of label a;

[0252] Indicates the label with the highest probability of selection;

[0253] Y k (d) represents open data y k (d) the corresponding coded data sequence;

[0254] The model parameters of the kth model generated by initialization are θ k (0), where:

[0255] θ1(0)=θ2(0)=...=θ k (0) = ... = θ K (0)

[0256] S33: Set the current iteration number of the model parameter to t, the initial value of t is 0, and the maximum value is Max, then the model parameter θ k The result of the tth iteration is θ k (t);

[0257] S34: Calculate the iterative reward value of the model parameters, where the model parameters θ k The iterative reward value of (t) is:

[0258]

[0259] in:

[0260] τ k (t) represents the model parameter θ k (t) iterative reward value;

[0261] F k (θ k (t)) represents the model parameter θ k (t) as the training objective function F k (·) input, output value of training objective function;

[0262] S35: Calculate the gradient information of the model parameters, where the model parameters θ k The gradient information of (t) is grad k (t), where the gradient information grad k (t) is calculated as follows: k Train the objective function F for the variable pair k (θ k ) is derived and the model parameter θ k Substitute (t) into the derivative result to obtain the model parameter θ k The gradient information grad of (t) k (t);

[0263] S36: Iterate the model parameters based on the iterative reward value and gradient information, where the model parameter θ k The iterative formula of (t) is:

[0264]

[0265] in:

[0266] T stands for transpose;

[0267] S37: Let t=t+1, and return to step S34 until the maximum number of iterations Max is reached; and a local optimization parameter is formed, wherein the local optimization parameter calculated by the k-th computing device is:

[0268] λ k =[(θ k (0),grad k (0)),(θ k (1),grad k (1)),...,(θ k (Max),grad k (Max))]

[0269] in:

[0270] λ k represents the local optimization parameter calculated by the kth computing device;

[0271] The computing device that completes the training immediately sends the local optimization parameters to the server for global asynchronous parameter optimization.

[0272] S4: Perform global asynchronous parameter optimization on multiple sets of local optimization parameters to obtain the optimal machine learning model parameters.

[0273] In the step S4, multiple groups of local optimization parameters are subjected to global asynchronous parameter optimization, including:

[0274] The server performs global asynchronous parameter optimization on the local optimization parameters calculated in step S3, wherein the global asynchronous parameter optimization process is as follows:

[0275] S41: Obtain initialization parameters θ(0), where θ(0)=θ1(0)=θ2(0)=...=θ k (0) = ... = θ K (0);

[0276] S42: Set the number of currently received local optimization parameters to u, the initial value of u is 0, and the global asynchronous parameter optimization result based on the u groups of local optimization parameters is θ(u);

[0277] S43: receiving the u+1th group of local optimization parameters λ(u+1), where λ(u+1) represents the u+1th group of local optimization parameters received by the server;

[0278] S44: Based on the u+1th group of local optimization parameters λ(u+1) received by the server, the global asynchronous parameter optimization result θ(u) based on the u group of local optimization parameters is optimized, wherein the optimization formula is:

[0279]

[0280] in:

[0281] Link u,u+1 represents the link coefficient between the training data set corresponding to the u+1th group of local optimization parameters λ(u+1) received by the server and the uth group of local optimization parameters λ(u);

[0282] λ Max (u+1) represents the model parameters obtained by the Max-th iteration in the local optimization parameter λ(u+1);

[0283] avg(grad,u+1) represents the mean of the gradient information corresponding to the Max+1 model parameter iteration results in the local optimization parameter λ(u+1);

[0284] is the adaptive step size corresponding to the local optimization parameter λ(u+1);

[0285] S45: Let u=u+1, return to step S43, until K groups of local optimization parameters are received, and take θ(K) as the model parameter optimization result under open data, that is, the optimal machine learning model parameters.

[0286] Embodiment 2:

[0287] like Figure 2 , is a schematic diagram of the structure of an electronic device for implementing a machine learning model training method under open data provided by one embodiment of the present invention.

[0288] The electronic device 1 may include a processor 10 , a memory 11 , a communication interface 13 and a bus, and may also include a computer program stored in the memory 11 and executable on the processor 10 , such as a program 12 .

[0289] The memory 11 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory 11 may be an internal storage unit of the electronic device 1, such as a mobile hard disk of the electronic device 1. In other embodiments, the memory 11 may also be an external storage device of the electronic device 1, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 1. Further, the memory 11 may also include both an internal storage unit of the electronic device 1 and an external storage device. The memory 11 may not only be used to store application software and various types of data installed in the electronic device 1, such as the code of the program 12, etc., but also be used to temporarily store data that has been output or is to be output.

[0290] The processor 10 may be composed of an integrated circuit in some embodiments, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips, etc. The processor 10 is the control core (Control Unit) of the electronic device, and uses various interfaces and lines to connect various components of the entire electronic device, and executes or executes programs or modules stored in the memory 11 (programs 12 for implementing machine learning model training under open data, etc.), and calls data stored in the memory 11 to execute various functions of the electronic device 1 and process data.

[0291] The communication interface 13 may include a wired interface and / or a wireless interface (such as a WI-FI interface, a Bluetooth interface, etc.), which is generally used to establish a communication connection between the electronic device 1 and other electronic devices, and to achieve connection and communication between internal components of the electronic device.

[0292] The bus may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The bus is configured to realize connection and communication between the memory 11 and at least one processor 10, etc.

[0293] Figure 2 Only an electronic device with components is shown, and those skilled in the art will understand that Figure 2 The structure shown does not constitute a limitation on the electronic device 1, and may include fewer or more components than shown in the figure, or combine certain components, or arrange the components differently.

[0294] For example, although not shown, the electronic device 1 may also include a power source (such as a battery) for supplying power to each component. Preferably, the power source may be logically connected to the at least one processor 10 through a power management device, so that the power management device can realize functions such as charging management, discharging management, and power consumption management. The power source may also include any components such as one or more DC or AC power sources, recharging devices, power failure detection circuits, power converters or inverters, power status indicators, etc. The electronic device 1 may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be repeated here.

[0295] Optionally, the electronic device 1 may further include a user interface, which may be a display, an input unit (such as a keyboard), or a standard wired interface or a wireless interface. Optionally, in some embodiments, the display may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, and an OLED (Organic Light-Emitting Diode) touch device. The display may also be appropriately referred to as a display screen or a display unit, which is used to display information processed in the electronic device 1 and to display a visual user interface.

[0296] It should be understood that the embodiment is for illustration only and the scope of the patent application is not limited to this structure.

[0297] The program 12 stored in the memory 11 in the electronic device 1 is a combination of multiple instructions. When running in the processor 10, it can achieve:

[0298] Collect open data sets, preprocess and extract semantic features of the open data sets, and obtain semantic feature vectors of each set of open data;

[0299] Cluster the open data based on its semantic feature vectors to obtain different categories of open data sets;

[0300] Different categories of open data sets are used as training data sets to train the machine learning model and obtain multiple sets of local optimization parameters;

[0301] Perform global asynchronous parameter optimization on multiple sets of local optimization parameters to obtain the optimal machine learning model parameters.

[0302] Specifically, the specific implementation method of the processor 10 for the above instructions can refer to Figure 1 to Figure 2 The description of the relevant steps in the corresponding embodiments will not be repeated here.

[0303] It should be noted that the serial numbers of the above-mentioned embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. And the terms "including", "comprising" or any other variants thereof in this article are intended to cover non-exclusive inclusion, so that a process, device, article or method including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, device, article or method. In the absence of further restrictions, an element defined by the sentence "including a ..." does not exclude the presence of other identical elements in the process, device, article or method including the element.

[0304] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0305] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A machine learning model training method under open data, characterized in that: The method comprises: S1: Collect open data sets, preprocess and extract semantic features of the open data sets, and obtain semantic feature vectors for each set of open data; S2: Cluster the open data based on its semantic feature vectors to obtain different categories of open data sets; S3: Use different categories of open data sets as training data sets to train the machine learning model and obtain multiple sets of local optimization parameters; S4: Perform global asynchronous parameter optimization on multiple sets of local optimization parameters to obtain the optimal machine learning model parameters.

2. The method for training a machine learning model under open data according to claim 1, characterized in that: In the step S1, an open data set is collected, and preprocessed and semantic features are extracted from the open data set, including: The collection of open data constitutes an open data set, where open data is publicly accessible and reusable data. The form of the open data set is: data={x n |n∈[1,N]} in: data represents an open data collection; x n represents the nth group of open data in the open data set, and N represents the total number of open data in the open data set; Each set of open data in the open data set is preprocessed and semantic features are extracted to obtain the semantic feature vector of each set of open data, where the nth set of open data x n The preprocessing and semantic feature extraction process is as follows: S11: Construct a word segmentation dictionary, and use the forward maximum matching method based on the word segmentation dictionary to search the nth group of open data x n Perform word segmentation to obtain the nth group of open data x n The word segmentation result sequence: in: Represents the nth group of open data x n The i-th word segmentation result, num n Represents the nth group of open data x n The total number of word segmentation results, i∈[1,num n ]; S12: Open data x for the nth group n The segmentation result sequence is encoded to obtain the nth group of open data x n The coded data sequence X n , as the nth group of open data x n The preprocessing results; S13: Encode the data sequence X n Perform semantic feature extraction to obtain the nth group of open data x n The semantic feature vector f n .

3. The method for training a machine learning model under open data according to claim 2, characterized in that: The step S13 encodes the data sequence X n Perform semantic feature extraction, including: S131: Extract the coded data sequence X n The probability representation parameter of any encoding processing result in The probability representation parameters are: in: Indicates word segmentation results In the nth group of open data x n The frequency of occurrence in Indicates word segmentation results The frequency of occurrence in N sets of open data; Indicates the encoding processing result The probability of represents the parameter; S132: Combine the probability representation parameters of the encoding processing results to calculate the encoded data sequence X n The attention weight of any encoding processing result in The attention weight calculation formula is: in: Indicates the result of encoding processing The attention weight of S133: Generate encoded data sequence X based on attention weights n The corresponding semantic feature vector f n : in: It means element-by-element addition; f n (i) indicates the encoding processing result semantic information; ReLU(·) denotes the ReLU activation function.

4. The method for training a machine learning model under open data according to claim 1, characterized in that: The step S2 clusters the open data based on the semantic feature vectors of the open data, including: Based on the semantic feature vectors of open data, open data are clustered to obtain K clusters. All open data in the clusters are regarded as open data of the same category to form a set of open data of K categories. The clustering process is as follows: S21: Calculate the distance between different open data, where the nth group of open data x n With the qth group of open data x q The distance between them is: in: dis(x n ,x q ) represents the nth group of open data x n With the qth group of open data x q The distance between q represents the qth group of open data x q The semantic feature vector of , q∈[1,N]; ||·||2 represents the L2 norm; Represents the semantic feature vector f n and the semantic feature vector f q The similarity between S22: Filter and obtain the M groups of open data closest to each group of open data, where the open data x n The nearest M groups of open data represent the open data x n The M groups of open data with the closest distance are counted, and the number of the most adjacent open data of each group of open data is counted, where the open data x n The number of mutually nearest open data is count(x n ); S23: Calculate the aggregation degree of each group of open data, where the aggregation degree of the nth group of open data is: in: δ n represents the aggregation degree of the nth group of open data; S24: Select K groups of open data with the highest aggregation degree as cluster centers, and each cluster center corresponds to a cluster; S25: Calculate the distance between the open data corresponding to the non-clustering center and the clustering center, and merge the open data corresponding to the non-clustering center into the cluster cluster where the nearest clustering center is located; S26: Calculate the mean of the semantic feature vectors corresponding to all the open data in the cluster, and select the open data corresponding to the semantic feature vector with the highest similarity to the mean of the semantic feature vector in the cluster as the update result of the cluster center in the cluster, and return to step S25; until the K cluster centers do not change, forming K clusters; S27: Calculate the link coefficient between each cluster, where the link coefficient between the kth cluster and the mth cluster is: Link(k,m)=Link(m,k) in: Link(k,m) represents the link coefficient between the kth cluster and the mth cluster, m∈[1,K],m≠k; AVG1(k,m) represents the average distance from any open data in the kth cluster to all open data in the mth cluster, and AVG2(k) represents the average distance from any open data in the kth cluster to other open data in the cluster.

5. The method for training a machine learning model under open data according to claim 4, characterized in that: In the step S3, different categories of open data sets are used as training data sets to train the machine learning model, including: Different categories of open data sets are used as training data sets to form K training data sets, and the link coefficients between clusters are used as the link coefficients between the training data sets associated with the clusters, that is, the link coefficient between the kth training data set and the mth training data set is Link(k,m), where the construction process of the kth training data set is: Get the open data set of the kth category: data k ={y k (d)|d∈[1,Sum k ]} in: data k represents the open data set of the kth category; y k (d) represents open data set data k The dth group of open data in Sum k Represents an open data set data k The total number of open data in; Label all open data in the open data set of the kth category, where the open data y k (d) The label information marked is L(y k (d)); Construct the kth training data set: <h2 style=";text-align:left;direction:ltr">Data<h2 style=";text-align:left;direction:ltr"> k <h2 style=";text-align:left;direction:ltr"> ={(y<h2 style=";text-align:left;direction:ltr"> k <h2 style=";text-align:left;direction:ltr"> (d),L(y<h2 style=";text-align:left;direction:ltr"> k <h2 style=";text-align:left;direction:ltr"> (d)))|d∈[1,Sum<h2 style=";text-align:left;direction:ltr"> k <h2 style=";text-align:left;direction:ltr"> ]} in: Data k represents the kth training data set; (y k (d),L(y k (d)) represents the training data set Data k The dth group of training data in ; Based on K training data sets, the machine learning model is trained to obtain K groups of local optimization parameters.

6. The method for training a machine learning model under open data according to claim 5, characterized in that: The machine learning model is trained based on K training data sets to obtain K groups of local optimization parameters, including: Based on K training data sets, the machine learning model is trained, and the training process is as follows: S31: Distribute K training data sets to K computing devices; S32: The computing device constructs a training objective function and initializes and generates the same model parameters, wherein the training objective function constructed by the k-th computing device is: in: F k (θ k ) represents the training objective function constructed by the kth computing device, θ k represents the model parameters to be trained for the kth computing device; Indicates that data will be opened k (d) As input, based on the model parameters θ k The output labels of the constructed model; φ represents a label set, a∈φ, a represents any label in the label set φ; Represents the matrix parameters corresponding to the label a. The matrix parameters corresponding to all labels in the label set φ constitute the parameters θ of the model to be trained. k ; G(y k (d); θ k ; a) means to open data y k (d) As input, based on the model parameters θ k The constructed model outputs the probability of label a; Indicates the label with the highest probability of selection; Y k (d) represents open data y k (d) the corresponding coded data sequence; The model parameters of the kth model generated by initialization are θ k (0), where: θ1(0)=θ2(0)=...=θ k (0)=...=θ K (0) S33: Set the current iteration number of the model parameter to t, the initial value of t is 0, and the maximum value is Max, then the model parameter θ k The result of the tth iteration is θ k (t); S34: Calculate the iterative reward value of the model parameter; S35: Calculate the gradient information of the model parameters, where the model parameters θ k The gradient information of (t) is grad k (t), where the gradient information grad k (t) is calculated as follows: k Train the objective function F for the variable pair k (θ k ) is derived and the model parameter θ k Substitute (t) into the derivative result to obtain the model parameter θ k The gradient information grad of (t) k (t); S36: Iterate the model parameters based on the iterative reward value and gradient information; S37: Let t=t+1, and return to step S34 until the maximum number of iterations Max is reached; and a local optimization parameter is formed, wherein the local optimization parameter calculated by the k-th computing device is: l k =[(θ k (0),grad k (0)),(θ k (1),grad k (1)),...,(θ k (Max),degree k (Max))] in: λ k represents the local optimization parameter calculated by the kth computing device; The computing device that completes the training immediately sends the local optimization parameters to the server for global asynchronous parameter optimization.

7. The method for training a machine learning model under open data according to claim 6, characterized in that: In the step S4, multiple groups of local optimization parameters are subjected to global asynchronous parameter optimization, including: The server performs global asynchronous parameter optimization on the local optimization parameters calculated in step S3, wherein the global asynchronous parameter optimization process is as follows: S41: Obtain initialization parameters θ(0), where θ(0)=θ1(0)=θ2(0)=...=θ k (0) = ... = θ K (0); S42: Set the number of currently received local optimization parameters to u, the initial value of u is 0, and the global asynchronous parameter optimization result based on the u groups of local optimization parameters is θ(u); S43: receiving the u+1th group of local optimization parameters λ(u+1), where λ(u+1) represents the u+1th group of local optimization parameters received by the server; S44: Based on the u+1th group of local optimization parameters λ(u+1) received by the server, the global asynchronous parameter optimization result θ(u) based on the u group of local optimization parameters is optimized, wherein the optimization formula is: in: Link u,u+1 represents the link coefficient between the training data set corresponding to the u+1th group of local optimization parameters λ(u+1) received by the server and the uth group of local optimization parameters λ(u); λ Max (u+1) represents the model parameters obtained by the Max-th iteration in the local optimization parameter λ(u+1); avg(grad,u+1) represents the mean of the gradient information corresponding to the Max+1 model parameter iteration results in the local optimization parameter λ(u+1); is the adaptive step size corresponding to the local optimization parameter λ(u+1); S45: Let u=u+1, return to step S43, until K groups of local optimization parameters are received, and take θ(K) as the model parameter optimization result under open data, that is, the optimal machine learning model parameters.

Images