Topic model and parameter updating method thereof, storage medium and electronic equipment

Abstract

The invention discloses a topic model, a parameter updating method thereof, a storage medium and electronic equipment, and the method comprises the steps: training local data of any small model to obtain a corresponding local model parameter, caching the local model parameter into a shared cache region, and refreshing a corresponding state identifier; the master model identifies the state identifier, reads the local model parameters from the shared cache region according to the state identifier, and calculates and feeds back global model parameters; when any small model does not recognize global model parameters fed back by the main model and other small models do not finish local model parameter caching, local data of the small model continues to be trained to obtain new local model parameters; and updating the corresponding local model parameters in the shared cache region according to the new local model parameters and the previous local model parameters. According to the method, resource waste can be avoided, the model training efficiency can be improved, and communication resources are saved.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence, and in particular relates to a topic model and a parameter updating method thereof, a storage medium and an electronic device. Background Art

[0002] With the rapid development of the Internet and social media, text data is showing an explosive growth trend. How to effectively process these massive text data and extract valuable information from them has become an important research topic.

[0003] Topic model is a text mining algorithm based on probability model, which can automatically identify the topics in documents and classify them. In recent years, topic model algorithms such as Latent Dirichlet Allocation (LDA) that can discover hidden semantic structures in unstructured document corpora and provide interpretable document classification capabilities have attracted the attention of researchers. However, in order to solve the problem of low efficiency of traditional Dirichlet allocation topic model algorithm in processing large-scale data sets, the idea of ​​parallel training of MoE (Mixture-of-Experts) framework is borrowed, the training process of LDA model is integrated into MoE, and the data on each node are trained in parallel, which can effectively improve the training efficiency in large-scale data set scenarios. Although the MoE framework has become one of the most popular technologies for training ultra-large-scale models, there are still many problems. For example, when using parallel computing to model massive data, in order to ensure the consistency and controllability of the model, the idea of ​​global synchronization is adopted. That is, the main model can only be synchronized and aggregated to obtain the global topic model parameters after all local small models have been iterated. Under this synchronous working mode, some small models with faster execution need to wait for other small models to complete the calculation before completing a round of iteration, which not only causes waste of resources, but also low model training efficiency. Summary of the invention

[0004] The present application aims to solve at least one of the technical problems existing in the related art. To this end, the present application proposes a topic model and a parameter updating method thereof, a storage medium and an electronic device to reduce resource waste and improve model training efficiency.

[0005] In a first aspect, the present application provides a method for updating parameters of a topic model, wherein the topic model is a topic model based on a MoE framework, including a plurality of small models and a main model that is communicatively connected to the plurality of small models, and the method includes:

[0006] Any of the small models obtains corresponding local model parameters after training its local data, caches the local model parameters in a shared cache area, and refreshes the corresponding state flag;

[0007] The main model identifies the state identifier, reads the local model parameters from the shared buffer area according to the state identifier, and calculates and feeds back the global model parameters;

[0008] When any of the small models fails to recognize the global model parameters fed back by the main model and other small models have not completed the local model parameter caching, they continue to train their local data to obtain new local model parameters, and update the corresponding local model parameters in the shared cache area based on the new local model parameters and the previous local model parameters.

[0009] In a second aspect, the present application provides a topic model, which is a topic model based on the MoE framework, including multiple small models and a main model that is communicatively connected to the multiple small models; wherein:

[0010] Any of the small models is used to obtain corresponding local model parameters after training its local data, cache the local model parameters in a shared cache area, and refresh the corresponding state flag;

[0011] The main model is used to identify the state identifier, and read the local model parameters from the shared buffer area according to the state identifier and calculate the feedback global model parameters;

[0012] Any of the small models is also used to continue training its local data to obtain new local model parameters when the global model parameters fed back by the main model are not identified and the other small models have not completed the local model parameter caching, and to update the corresponding local model parameters in the shared cache area according to the new local model parameters and the previous local model parameters.

[0013] In a third aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for updating topic model parameters as described in the first aspect above is implemented.

[0014] In a fourth aspect, the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for updating topic model parameters as described in the first aspect above is implemented.

[0015] In a fifth aspect, the present application provides a chip, comprising a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the topic model parameter updating method as described in the first aspect.

[0016] In a sixth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the topic model parameter updating method as described in the first aspect above.

[0017] The above one or more technical solutions in the embodiments of the present application have at least the following technical effects:

[0018] The subject model and parameter updating method, storage medium and electronic device provided in the embodiment of the present application, after any small model trains its local data to obtain the corresponding local model parameters, does not wait for the training results of other small models, but continues the next round of training iteration when the global model parameters fed back by the main model are not identified and the local model parameter cache of other small models is not completed, and updates the corresponding local model parameters according to the new local model parameters obtained in the new iteration and the previous local model parameters. In this way, each small model can maximize the use of its own computing resources for deeper iterations to avoid resource waste, and because some small models have been iterated at a deeper level, the local model parameters of these small models can be closer to the optimal value of the local model parameters, thereby improving the efficiency of model training. At the same time, the main model will identify the status identifier of each small model, read the local model parameters from the shared cache area through the status identifier and calculate the feedback global model parameters. Only one round of communication is needed to obtain the local model parameters corresponding to each small model and update the global model parameters, which not only saves communication resources, but also because the acquisition of each local model parameter is executed in the cache layer, the global model parameter update efficiency is higher.

[0019] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0021] Figure 1 It is a schematic diagram of the architecture of the topic model based on the MoE framework provided in an embodiment of the present application;

[0022] Figure 2 It is a flowchart of a method for updating topic model parameters provided in an embodiment of the present application;

[0023] Figure 3 It is a schematic diagram of the structure of the topic model provided in the embodiment of the present application;

[0024] Figure 4 It is a schematic diagram of the process of implementing clustering of local data provided in an embodiment of the present application;

[0025] Figure 5 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments in the present application belong to the scope of protection of this application.

[0027] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0028] For ease of understanding, the relevant technologies involved in this application are described as follows:

[0029] The traditional Dirichlet allocation topic algorithm model is based on the latent semantic indexing method, which can automatically identify topics in documents and classify them. The deep learning model architecture based on sparse MoE is to split the large model into multiple small models. In each iteration, a part of the small models is activated for calculation according to the sample, so as to save computing resources. At the same time, in order to ensure the reasonable allocation of computing power, the gating mechanism is introduced.

[0030] In order to solve the problem of a significant increase in computing resource costs when using the traditional Dirichlet allocation topic algorithm model in the scenario of using large-scale data sets, the LDA algorithm model is transformed based on the inspiration of the deep learning model architecture of sparse MoE. The modified topic model based on the MoE framework is as follows Figure 1As shown, including multiple small models (Expert1, Expert2, ... ExpertN) and gating (gata), different types of documents can use specific small models for training or prediction. In distributed computing, since the data assigned to each node has low relevance, high randomness and relatively small data sets, each node often cannot accurately estimate the parameters of the topic (such as topic distribution, word frequency, etc.) when making local estimates, which leads to a loss of model accuracy or even failure to converge. If the data correlation of each node is high and there is a local optimal value, the global optimal solution can be quickly converged. The topic model based on the MoE framework is generated based on this idea. Each small model in the topic model can focus on document recognition in its own field of expertise, and can independently and parallelly complete related training and prediction, thereby realizing global training and prediction of the large model. As shown in Figure 1 As shown in the figure, specific data sets are collected from large-scale data sets and input into small models Expert1, Expert2, ... ExpertN respectively. Through gating, the final output of the model is G(x)1*Expert1+G(x) i *Experti+…+G(x) N *ExpertN, that is, the final trained topic model can be expressed by the following formula: Among them, E i (x) represents the output result of the i-th small model, G(x) i It represents the gating function corresponding to the i-th small model. It can be seen that gating can realize the selection of small models and the weight proportion of the final output.

[0031] The following, in conjunction with the accompanying drawings, describes in detail the topic model parameter updating method, topic model, electronic device and readable storage medium provided in the embodiments of the present application through specific embodiments and their application scenarios.

[0032] Among them, the topic model parameter updating method can be applied to a terminal, and can be specifically executed by hardware or software in the terminal.

[0033] The topic model parameter updating method provided in the embodiment of the present application may be executed by an electronic device or a functional module or functional entity in the electronic device that can implement the topic model parameter updating method. The electronic devices mentioned in the embodiment of the present application include but are not limited to mobile phones, tablet computers, computers, cameras, and wearable devices. The topic model parameter updating method provided in the embodiment of the present application is described below using an electronic device as an example of the execution subject.

[0034] like Figure 2As shown, the topic model parameter updating method, wherein the topic model is a topic model based on the MoE framework, includes multiple small models and a main model that is communicatively connected to the multiple small models, and the method includes the following steps:

[0035] S100, any of the small models obtains corresponding local model parameters after training its local data, caches the local model parameters in a shared cache area, and refreshes the corresponding status identifier.

[0036] S200, the main model identifies the state identifier, reads the local model parameters from the shared cache area according to the state identifier, and calculates and feeds back global model parameters.

[0037] S300, when any of the small models fails to recognize the global model parameters fed back by the main model and other small models have not completed the local model parameter caching, it continues to train its local data to obtain new local model parameters, and updates the corresponding local model parameters in the shared cache area according to the new local model parameters and the previous local model parameters.

[0038] After any small model in the embodiment of the present application trains its local data to obtain the corresponding local model parameters, it does not wait for the training results of other small models, but continues the next round of training iteration when the global model parameters fed back by the main model are not identified and the local model parameter cache of other small models is not completed, and updates the corresponding local model parameters according to the new local model parameters obtained in the new iteration and the previous local model parameters. In this way, each small model can maximize the use of its own computing resources for deeper iterations to avoid resource waste, and because some small models have undergone deeper iterations, the local model parameters of these small models can be closer to the optimal values ​​of the local model parameters, thereby improving the efficiency of model training. At the same time, the main model will identify the status identifier of each small model, read the local model parameters from the shared cache area through the status identifier, and calculate the feedback global model parameters. Only one round of communication is required to obtain the local model parameters corresponding to each small model and update the global model parameters, which not only saves communication resources, but also because the acquisition of each local model parameter is performed in the cache layer, the global model parameter update efficiency is higher.

[0039] In some embodiments, in step S300, updating the corresponding local model parameters in the shared cache area according to the new local model parameters and the previous local model parameters includes:

[0040] The new local model parameter and the previous local model parameter are averaged to obtain an average local model parameter, and the average local model parameter is used to update the corresponding local model parameter in the shared buffer area.

[0041] By iterating multiple times for each small model before updating the global model parameters, the local optimal value of the small model can be approximated to the maximum extent, thereby improving the efficiency of obtaining the optimal value of the global model parameters. In this process, the local model parameters generated by each iteration of each small model are only cached in the shared cache area, and the local model parameters are not sent to the main model after each iteration, which greatly reduces the number of communications between the small model and the main model.

[0042] Specifically, the average local model parameters can be achieved by the following formula:

[0043] in, represents the average local model parameter corresponding to the i-th small model, N represents the number of small models, and t represents the number of iterations of the i-th small model.

[0044] In some embodiments, the state identifier includes a ready state and a not ready state. The ready state indicates that the small model has completed caching of its corresponding local model parameters. The not ready state indicates that the small model has not completed caching of its corresponding local model parameters.

[0045] The state flag is used to indicate the update state of the local model parameters of each small model, so that the main model can obtain the local model parameters of each small model according to the state flag and perform subsequent calculations of all model parameters. It is worth noting that the state flag of each small model can be represented by a bit in the shared cache area. When the state flag of the small model is ready, the value of the bit is set to 1, and when the state flag of the small model is not ready, the value of the bit is set to 0.

[0046] It is understandable that when any small model completes caching of local model parameters into the shared cache area, the state of the small model is considered to be ready.

[0047] Specifically, in some embodiments, step S200 includes:

[0048] The main model identifies the state identifiers of all the small models, and when it is determined that all the state identifiers are in the ready state, reads the local model parameters of all the small models from the shared buffer area.

[0049] The global model parameters are obtained by averaging the local model parameters of all the small models, and the global model parameters are fed back to the shared cache area.

[0050] When the main model recognizes that the status identifiers of all small models are in the ready state, it means that all small models have completed at least one round of iteration and cached the corresponding local model parameters to the shared cache area. At this time, the main model can obtain the local model parameters of all small models by accessing the shared cache area, and globally aggregate the local model parameters of all small models to obtain the global model parameters. Finally, all the model parameters are returned to the shared cache area for subsequent iterations of each small model.

[0051] During the actual execution process, the main model can periodically identify the status flags of the small models. When it is determined that there are still status flags that have not been updated to the ready state, no subsequent operations will be performed. Only when it is determined that all status flags are in the ready state, the local model parameters of all small models will be read from the shared cache area. In this way, the main model can obtain the local model parameters corresponding to all small models through only one round of communication, reducing unnecessary communication overhead of the main model.

[0052] In some embodiments, the main model averages the local model parameters of all small models to obtain the global model parameters. Specifically, the global model parameters can be realized by the following formula:

[0053] Among them, λ i represents the local model parameters corresponding to the i-th small model, λ G are all model parameters.

[0054] The global model parameters are obtained by averaging the local model parameters of each small model, taking into account the impact of each small model on the final output, thereby improving the accuracy of model training.

[0055] In some embodiments, the main model may also perform weighted calculation on the local model parameters of each small model to obtain the overall model parameters.

[0056] In some embodiments, after feeding back the global model parameters to the shared cache area, the method further includes: updating the status flags of all the small models to the unready state, and using the status flags of the prepared main model in the next iteration.

[0057] In some embodiments, in step S100 and step S300, the small model trains its local data to obtain corresponding local model parameters including:

[0058] Select local partial data from the local data, and train the local partial data to obtain the corresponding local model parameters.

[0059] Each small model selects part of the local data for training. On the one hand, it can improve the efficiency of iteration. On the other hand, different data is used for training in each iteration to improve the realism of each small model.

[0060] It is worth noting that when each small model is iteratively trained before all model parameters are updated, the next round of iteration still uses the old small model for iteration, and does not use the average local model parameters or the local model parameters obtained last time to update the small model, so as to ensure that the small model is in a stable state before all model parameters are updated.

[0061] In some embodiments, after the master model feeds back the global model parameters to the shared buffer, the method includes:

[0062] The small model reads the global model parameters from the shared cache area, and uses the global model parameters to train the local partial data in the local data to obtain the incrementally updated local model parameters.

[0063] In this embodiment, after the global model parameters are updated, each small model will grab the global model parameters from the shared cache area, and use all the model parameters to update its own small model to obtain a new small model. In a new round of global model parameter iteration process, the new small model will re-use local data for training to obtain incrementally updated local model parameters.

[0064] In some embodiments, when the number of iterations of the global model parameters reaches a set value or the global model parameters reach a convergence threshold, the main model and each small model terminate the iteration to obtain a trained topic model.

[0065] In some embodiments, the shared cache area records the average local model parameters, the total model parameters and the state identifier in the form of a state bitmap.

[0066] In actual implementation, the status identifier of the i-th small model can be represented by the symbol S ti Indicates that the local model parameters of the i-th small model can be used to meet λ i or i Indicates that when the small model outputs only one local model parameter in a round of global model parameter iteration, the local model parameter is λ i All model parameters can be used to meet λ G Indicates that the final display format in the shared cache area is as follows Figure 3As shown. It can be understood that, since the global model parameters are the same for all small models, all small models in the shared cache area can use only one global parameter bit area to cache the value of the global model parameter. For each small model, a local parameter bit area and a state identification bit area are set to cache the state identification and local model parameters corresponding to each small model. Among them, the state identification can be represented by only one bit. When the value of the bit is 0, it indicates that the corresponding small model is not ready. When the value of the bit is 1, it indicates that it is ready for a small model.

[0067] In some embodiments, the local data is clustered data for each small model by combining MKM and LSH. Combining MKM and LSH to achieve data aggregation can reduce the amount of calculation.

[0068] In some embodiments, when the local data is clustered by combining MKM and LSH, Figure 4 As shown, the following steps are included:

[0069] S400 , mapping a data point Xb in the collected data subset to a corresponding hash bucket using a hash function group, to obtain an initial hash bucket of the data point Xb.

[0070] Each of the hash buckets corresponds to a cluster center point, and the data point Xb is any data point in the data subset.

[0071] S500, determining the number of the initial hash buckets, and calculating the distance between the data point Xb and the cluster center point corresponding to each of the initial hash buckets.

[0072] S600: Determine the hash bucket to which the data point Xb finally belongs according to the number of the initial hash buckets and the distance.

[0073] In the clustering process of the topic model training data, the data point Xb in the data subset is first mapped to the corresponding initial hash bucket through the hash function group. Since the data point Xb is mapped by the hash function group, the data point Xb may be mapped to multiple different hash buckets. Then the initial hash bucket is determined and the distance between the data point Xb and the cluster center point corresponding to each initial hash bucket is calculated. Finally, the hash bucket to which the data point Xb finally belongs is determined according to the number and distance of the initial hash buckets. In this process, the data point Xb only needs to calculate the distance to the cluster center point corresponding to the initial hash bucket, and there is no need to calculate the distance between the data point Xb and all cluster center points, which greatly reduces the amount of calculation when clustering data.

[0074] It should be noted that, since each of the hash buckets corresponds to a cluster center point, determining the hash bucket to which the data point Xb finally belongs is equivalent to clustering the data point Xb to a certain cluster center point. It is understandable that the specific data points finally determined in each of the hash buckets are the local data corresponding to each small model.

[0075] It can be understood that although the hash function group is introduced in the present application, the calculation of the hash function is simpler than the calculation of the Euclidean distance, so the amount of calculation can be reduced. For example, if the amount of data in the data subset is |B|, the number of cluster center points is L, and the calculation complexity of the Euclidean distance is D, then the complexity of the calculation in the MKM technology is O(L*|B|*D). The solution in the embodiment of the present application is to combine the MKM technology with the LSH (Locality Sensitive Hashing) technology. The calculation complexity of the combined solution is O(l*|B|*D+|B|*h), l can be understood as the number of cluster center points involved in the calculation, which is much smaller than L, and even in some cases there is no need to calculate the distance (for example: the hash function group maps the data point to an initial hash bucket), h can be understood as the calculation complexity of the introduced hash function, and the amount of calculation of the hash function is smaller than the amount of calculation of the distance, so the amount of calculation during clustering can be reduced as a whole.

[0076] It is worth noting that the reason why a hash function group is used when hash mapping the data point Xb in this embodiment is that if only a single hash function is used, the data point will be directly assigned to the corresponding hash bucket according to the hash value of the hash function. In this case, the similarity of the data may not be fully captured because different data points may be assigned to the same bucket due to hash collisions, or similar data points may be assigned to different buckets due to slight differences in hash values. Multiple hash functions, i.e., a combination of hash functions, are used to calculate hash values, and data points are assigned to different buckets according to these hash values. Specifically, the data point will be processed by all selected hash functions to obtain multiple hash values, and then these hash values ​​will jointly determine the bucket affiliation of the data point in each hash table, thereby improving the accuracy of data clustering.

[0077] In some embodiments, step S600 includes:

[0078] If the number of the initial hash buckets is two or more, the cluster center points of the initial hash buckets are used as candidate cluster center points.

[0079] The distance between the data point Xb and each of the candidate cluster center points is calculated, and the initial hash bucket where the candidate cluster center point closest to the data point Xb is located is used as the hash bucket to which the data point Xb finally belongs.

[0080] If the number of the initial hash bucket is one, the initial hash bucket is determined to be the hash bucket to which the data point Xb finally belongs.

[0081] If the data point Xb is mapped to only one initial hash bucket after being mapped by the hash function group, it means that the data point Xb has a high similarity with the cluster center point corresponding to the initial hash bucket, and the data point Xb can be directly attributed to the initial hash bucket, that is, the affiliation of the data point Xb is determined only by the hash function group, and no subsequent distance calculation is required, which further reduces the amount of calculation. If there are two or more initial hash buckets after the data point Xb is mapped by the hash function group, it is necessary to further determine the final hash bucket affiliation of the data point Xb. At this time, it is only necessary to calculate the distance between the data point Xb and the candidate cluster center points corresponding to each initial hash bucket, and the initial hash bucket where the candidate cluster center point with the closest distance is located is used as the hash bucket to which the data point Xb finally belongs. This process only calculates the distance between the data point Xb and a few cluster center points, which greatly reduces the amount of calculation compared to the MKM technology.

[0082] In some embodiments, if the number of the initial hash buckets is two or more, the hash bucket to which the data point Xb finally belongs can also be determined by the number of mappings of the data point Xb by the hash function group to each initial hash bucket. Preferably, the initial hash bucket with a mapping number greater than a preset threshold is used as the hash bucket to which the data point Xb finally belongs, or the initial hash bucket with the most mapping times among the initial hash buckets is used as the hash bucket to which the data point Xb finally belongs. This embodiment does not involve the calculation of distance, further reducing the computational complexity during data clustering.

[0083] In some embodiments, the hash bucket to which the data point Xb finally belongs is calculated according to the following formula:

[0084] R min =argmin||X i -C j || 2 , where C j represents the jth candidate cluster center point, R min Indicates the hash bucket to which the data point Xb finally belongs, 1≤j≤L, and the argmin function represents the initial hash bucket corresponding to the minimum distance between the data point Xb and the candidate cluster center point, i∈{1,2,...,|B|}, |B| is the amount of data in the data subset. This formula can quickly match the data point Xb to the hash bucket to which it finally belongs.

[0085] In some embodiments, step S400 includes:

[0086] The data point Xb is input into the hash function H l, the hash value of the data point Xb is calculated, where l is an integer greater than or equal to 1 and less than or equal to L, and L is the number of cluster center points, which is determined by the number of small models in the topic model.

[0087] Compare the hash value with the cluster point hash value of each hash bucket to obtain the closest cluster point hash value, wherein the cluster point hash value is obtained by the cluster center point of each hash bucket through the hash function H l Calculated.

[0088] The hash bucket corresponding to the hash value of the closest cluster point is used as the initial hash bucket.

[0089] By comparing the hash value of the data point Xb with the hash value of the cluster point of each hash bucket, the data point Xb is quickly mapped to the corresponding initial hash bucket.

[0090] It should be noted that when the hash value of the data point Xb is compared with the hash value of the cluster point of each hash bucket, it is performed under the same hash function. The data point Xb may be mapped to different initial hash buckets through multiple hash functions in the hash function group. For example: the data point Xb obtains a first hash value through the hash function H1, and the cluster center point corresponding to each hash bucket obtains the first cluster point hash value corresponding to each hash bucket through the hash function H1, and the first hash value is compared with the hash values ​​of each first cluster point to obtain the closest first cluster point hash value, so that the data point Xb is mapped to the first initial hash bucket corresponding to the closest first cluster point hash value under the mapping of the hash function H1; the data point Xb obtains a second hash value through the hash function H2, and the cluster center point corresponding to each hash bucket obtains the second cluster point hash value corresponding to each hash bucket through the hash function H2, and the second hash value is compared with the hash values ​​of each second cluster point to obtain the closest second cluster point hash value, so that the data point Xb is mapped to the second initial hash bucket corresponding to the closest second cluster point hash value under the mapping of the hash function H2, and so on, the initial hash bucket of the data point Xb is obtained by the first initial hash bucket, the second initial hash bucket, ... the lth initial hash bucket. In this embodiment, the data point Xb and the cluster point hash value of each hash bucket are used to achieve fast mapping of the data point Xb to the initial hash bucket.

[0091] It can be understood that L is the number of cluster centers, that is, the number of hash buckets. The number of cluster centers is determined by the number of small models in the topic model, that is, L is determined by the number of small models in the topic model, and the value of L is less than or equal to the number N of small models in the topic model.

[0092] In some embodiments, after the topic model is trained, sample data points in the large-scale dataset are input into the corresponding small model based on the following formula:

[0093] G(x)=TopN(1-Min(r l )),l∈[1,L], G(x) is the gating function corresponding to each small model, which is the distance between the sample data point and the cluster center point l. The TopN function can filter out the cluster center point with the closest distance to the sample data point from the L cluster center points.

[0094] In order to achieve fast and accurate classification prediction of large-scale data sets, it is necessary to assign sample data points in large-scale data sets to matching small models, analyze them by different small models, and then use the main model in the topic model to summarize the data of each small model to obtain the final topic classification. The above formula can quickly match sample data points to the corresponding small model and achieve fast prediction of topic classification.

[0095] In some embodiments, the final output result of the topic model can be expressed by the following formula:

[0096] where φ k,w represents the probability of word w appearing in topic k calculated by the lth small model, θ d,k Represents the probability of topic k appearing in document d calculated by the lth small model.

[0097] In some embodiments, before step S400, the method further includes:

[0098] Collect a preset amount of text data from a large-scale dataset multiple times.

[0099] Specific parts of the data in each of the text data are extracted to obtain each data point of the data subset collected each time.

[0100] In order to improve the training efficiency of the model, a preset amount of text data is collected from a large-scale data set for clustering each time. In order to ensure the rapid clustering of text data while ensuring the generalization ability of the model, only specific parts of the text data (such as the title or abstract of the document, etc., because the title or abstract of the document can usually distinguish which field the document belongs to, without the need for overly detailed classification) can be used in subsequent clustering calculations.

[0101] The subject of the topic model parameter updating method provided in the embodiment of the present application may be a topic model parameter updating device. In the embodiment of the present application, the topic model parameter updating device performing the topic model parameter updating method is taken as an example to illustrate the topic model parameter updating device provided in the embodiment of the present application.

[0102] The present application also provides a topic model parameter updating device, which includes:

[0103] The cache refresh module is used to obtain the corresponding local model parameters after training its local data, cache the local model parameters in a shared cache area, and refresh the corresponding state identifier. The main model has been prepared to identify the state identifier, and read the local model parameters from the shared cache area according to the state identifier and calculate the feedback global model parameters.

[0104] An update module is used to continue training its local data to obtain new local model parameters when the global model parameters fed back by the main model are not identified and the other small models have not completed the local model parameter caching, and to update the corresponding local model parameters in the shared cache area according to the new local model parameters and the previous local model parameters.

[0105] According to the topic model parameter updating device provided in the embodiment of the present application, after any small model trains its local data to obtain the corresponding local model parameters, it does not wait for the training results of other small models, but continues the next round of training iteration when the global model parameters fed back by the main model are not recognized and the other small models have not completed the local model parameter caching, and updates the corresponding local model parameters according to the new local model parameters obtained in the new iteration and the previous local model parameters. In this way, each small model can maximize the use of its own computing resources for deeper iterations to avoid waste of resources, and since some small models have undergone deeper iterations, the local model parameters of these small models can be closer to the optimal values ​​of the local model parameters, thereby improving the efficiency of model training.

[0106] In some embodiments, the updating module is further used to obtain an average local model parameter by averaging the new local model parameter and the previous local model parameter, and to update the corresponding local model parameter in the shared cache area with the average local model parameter.

[0107] In some embodiments, the state identifier includes a ready state and a not ready state; the ready state indicates that the small model has completed the caching of its corresponding local model parameters; the not ready state indicates that the small model has not completed the caching of its corresponding local model parameters.

[0108] In some embodiments, the cache refresh module or the update module is further used to select part of the local data in the local data, and train the part of the local data to obtain the corresponding local model parameters.

[0109] In some embodiments, the device also includes an incremental update module, which is used for, after the main model feeds back the global model parameters to the shared cache area, any of the small models reads the global model parameters from the shared cache area, and uses the global model parameters to train the local part of the local data to obtain the incrementally updated local model parameters.

[0110] The present application also provides a topic model parameter updating device, which includes:

[0111] An identification and calculation module is used to identify the state identifier, and read the local model parameters from the shared cache area according to the state identifier and calculate the feedback global model parameters, wherein the state identifier is the state identifier corresponding to any small model when the small model obtains the corresponding local model parameters after training its local data and caches them in the shared cache area, and the global model parameters are used for any small model to read the global model parameters from the shared cache area, use the global model parameters to train the local data, and obtain the incrementally updated local model parameters.

[0112] According to the subject model parameter updating device provided in the embodiment of the present application, the status identifier of each small model is identified by the main model, and the local model parameters are read from the shared cache area through the status identifier and the feedback global model parameters are calculated. Only one round of communication is required to obtain the local model parameters corresponding to each small model and update the global model parameters. There is no need to poll or read the local model parameters of each small model multiple times to calculate the global model parameters. This not only saves communication resources, but also because the acquisition of each local model parameter is executed in the cache layer, the global model parameter update efficiency is higher.

[0113] In some embodiments, the state identifier includes a ready state and a not ready state; the ready state indicates that the small model has completed the caching of its corresponding local model parameters; the not ready state indicates that the small model has not completed the caching of its corresponding local model parameters.

[0114] In some embodiments, the identification and calculation module is also used to identify the status identifiers of all the small models, and when it is determined that all the status identifiers are in the ready state, read the local model parameters of all the small models from the shared cache area; average the local model parameters of all the small models to obtain the global model parameters, and feed the global model parameters back to the shared cache area.

[0115] In some embodiments, the identification and calculation module is further used to update the status identifiers of all the small models to the not ready state.

[0116] The topic model parameter updating device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or it can be other devices other than the terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc., and can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0117] The topic model parameter updating device in the embodiment of the present application may be a device having an operating system. The operating system may be a Microsoft (Windows) operating system, an Android (Android) operating system, an IOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0118] The topic model parameter updating device provided in the embodiment of the present application can achieve Figures 1 to 3 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0119] In some embodiments, the present application also provides a topic model, such as Figure 3 As shown, the topic model is a topic model based on the MoE framework, including multiple small models and a main model that is communicatively connected to the multiple small models.

[0120] in:

[0121] Any of the small models is used to obtain corresponding local model parameters after training its local data, cache the local model parameters in a shared cache area, and refresh the corresponding status identifier.

[0122] The main model is used to identify the state identifier, and read the local model parameters from the shared buffer area according to the state identifier and calculate and feedback the global model parameters.

[0123] Any of the small models is also used to continue training its local data to obtain new local model parameters when the global model parameters fed back by the main model are not identified and the other small models have not completed the local model parameter caching, and to update the corresponding local model parameters in the shared cache area according to the new local model parameters and the previous local model parameters.

[0124] The subject model provided in the embodiment of the present application does not wait for the training results of other small models after any small model trains its local data to obtain the corresponding local model parameters. Instead, when the global model parameters fed back by the main model are not identified and the local model parameter cache of other small models is not completed, the next round of training iteration is continued, and the corresponding local model parameters are updated according to the new local model parameters obtained in the new iteration and the previous local model parameters. In this way, each small model can maximize the use of its own computing resources for deeper iterations to avoid resource waste. Moreover, since some small models have undergone deeper iterations, the local model parameters of these small models can be closer to the optimal values ​​of the local model parameters, thereby improving the efficiency of model training. At the same time, the main model will identify the status identifier of each small model, read the local model parameters from the shared cache area through the status identifier, and calculate the feedback global model parameters. Only one round of communication is required to obtain the local model parameters corresponding to each small model and update the global model parameters, which not only saves communication resources, but also because the acquisition of each local model parameter is performed in the cache layer, the global model parameter update efficiency is higher.

[0125] In some embodiments, any small model is also used to average the new local model parameters and the previous local model parameters to obtain average local model parameters, and use the average local model parameters to update the corresponding local model parameters in the shared cache area.

[0126] In some embodiments, the state identifier includes a ready state and a not ready state; the ready state indicates that the small model has completed the caching of its corresponding local model parameters; the not ready state indicates that the small model has not completed the caching of its corresponding local model parameters.

[0127] In some embodiments, the main model is further used to identify the state identifiers of all the small models, and when it is determined that all the state identifiers are in the ready state, read the local model parameters of all the small models from the shared cache area. The local model parameters of all the small models are averaged to obtain the global model parameters, and the global model parameters are fed back to the shared cache area. The state identifiers of all the small models are updated to the not ready state.

[0128] In some embodiments, any small model is also used to select part of the local data in the local data, and train the part of the local data to obtain the corresponding local model parameters.

[0129] In some embodiments, any small model is also used to read the global model parameters from the shared cache area after the main model feeds back the global model parameters to the shared cache area, and use the global model parameters to train the local part of the local data to obtain the incrementally updated local model parameters.

[0130] In some embodiments, the shared cache area records the average local model parameters, the total model parameters and the state identifier in the form of a state bitmap.

[0131] In some embodiments, the main model may further include: a clustering module for clustering the local data for each small model by combining MKM and LSH.

[0132] In some embodiments, the clustering module specifically includes: a mapping module, used to map the data point Xb in the collected data subset to the corresponding hash bucket using a hash function group to obtain an initial hash bucket of the data point Xb, wherein each of the hash buckets corresponds to a cluster center point, and the data point Xb is any data point in the data subset;

[0133] A calculation module, used to determine the number of the initial hash buckets, and calculate the distance between the data point Xb and the cluster center point of each of the initial hash buckets;

[0134] A determination module is used to determine the hash bucket to which the data point Xb finally belongs according to the number of the initial hash buckets and the distance.

[0135] The data point Xb in the data subset is first mapped to the corresponding initial hash bucket through the hash function group. Since the data point Xb is mapped by the hash function group, the data point Xb may be mapped to multiple different hash buckets. Then the initial hash bucket is determined and the distance between the data point Xb and the cluster center point corresponding to each initial hash bucket is calculated. Finally, the hash bucket to which the data point Xb finally belongs is determined according to the number and distance of the initial hash buckets. In this process, the data point Xb only needs to calculate the distance to the cluster center point corresponding to the initial hash bucket, and there is no need to calculate the distance between the data point Xb and all cluster center points, which greatly reduces the amount of calculation when clustering data.

[0136] In some embodiments, the determination module is further configured to, if the number of the initial hash buckets is two or more, use the cluster center points of each of the initial hash buckets as candidate cluster center points, calculate the distance between the data point Xb and each of the candidate cluster center points, and use the initial hash bucket where the candidate cluster center point closest to the data point Xb is located as the hash bucket to which the data point Xb ultimately belongs. If the number of the initial hash buckets is one, determine the initial hash bucket as the hash bucket to which the data point Xb ultimately belongs.

[0137] In some embodiments, the hash bucket to which the data point Xb finally belongs is calculated according to the following formula:

[0138] R min =argmin||X i -C j || 2 , where C j represents the jth candidate cluster center point, R min Indicates the hash bucket to which the data point Xb finally belongs, 1≤j≤L, i∈{1,2,...,|B|}, |B| is the amount of data in the data subset, and the argmin function represents the initial hash bucket corresponding to the minimum distance between the data point Xb and the candidate cluster center.

[0139] In some embodiments, the mapping module is further configured to input the data point Xb into a hash function H l , calculate the hash value of the data point Xb, where l is an integer greater than or equal to 1 and less than or equal to L, and L is the number of cluster center points, which is determined by the number of small models in the topic model; compare the hash value with the cluster point hash value of each hash bucket to obtain the closest cluster point hash value, where the cluster point hash value is obtained by the cluster center points of each hash bucket through the hash function H l Calculate and use the hash bucket corresponding to the hash value of the closest cluster point as the initial hash bucket.

[0140] In some embodiments, the topic model may further include a collection module and an extraction module, wherein the collection module is used to collect a preset amount of text data from a large-scale data set multiple times. The extraction module is used to extract a specific portion of data from each of the text data to obtain each data point of the data subset collected each time.

[0141] In some embodiments, the sample data points in the large-scale data set are input into the corresponding small model based on the following formula: G(x)=TopN(1-Min(r l )),l∈[1,L], G(x) is the gating function corresponding to each small model, r lis the distance between the sample data point and the cluster center point l. The TopN function can filter out the cluster center point with the closest distance to the sample data point from L cluster centers.

[0142] In some embodiments, Figure 5 As shown, an embodiment of the present application also provides an electronic device 800, including a processor 801, a memory 802, and a computer program stored in the memory 802 and executable on the processor 801. When the program is executed by the processor 801, each process of the above-mentioned topic model parameter updating method embodiment is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0143] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0144] An embodiment of the present application also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the various processes of the above-mentioned topic model parameter updating method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0145] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0146] An embodiment of the present application also provides a computer program product, including a computer program, which implements the above-mentioned topic model parameter updating method when executed by a processor.

[0147] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0148] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned topic model parameter updating method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0149] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0150] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0151] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0152] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

[0153] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "illustrative embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials, or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials, or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0154] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

Claims

1. A method for updating topic model parameters, characterized in that: The topic model is a topic model based on the MoE framework, including multiple small models and a main model that is communicatively connected to the multiple small models. The method includes: Any of the small models obtains corresponding local model parameters after training its local data, caches the local model parameters in a shared cache area, and refreshes the corresponding state flag; The main model identifies the state identifier, reads the local model parameters from the shared buffer area according to the state identifier, and calculates and feeds back the global model parameters; When any of the small models fails to recognize the global model parameters fed back by the main model and other small models have not completed the local model parameter caching, they continue to train their local data to obtain new local model parameters, and update the corresponding local model parameters in the shared cache area based on the new local model parameters and the previous local model parameters.

2. The method according to claim 1, characterized in that The updating of the corresponding local model parameters in the shared buffer area according to the new local model parameters and the previous local model parameters includes: The new local model parameter and the previous local model parameter are averaged to obtain an average local model parameter, and the average local model parameter is used to update the corresponding local model parameter in the shared buffer area.

3. The method according to claim 1, characterized in that The state identifier includes a ready state and a not ready state; The ready state indicates that the small model has completed caching of its corresponding local model parameters; The not-ready state indicates that the small model has not completed the caching of its corresponding local model parameters.

4. The method according to claim 3, characterized in that The main model identifies the state identifier, reads the local model parameters from the shared buffer area according to the state identifier, and calculates and feeds back the global model parameters, including: The main model identifies the state identifiers of all the small models, and when it is determined that all the state identifiers are in the ready state, reads the local model parameters of all the small models from the shared buffer area; The global model parameters are obtained by averaging the local model parameters of all the small models, and the global model parameters are fed back to the shared cache area.

5. The method according to claim 4, characterized in that After feeding back the global model parameters to the shared cache area, the method includes: updating the status flags of all the small models to the not ready state.

6. The method according to claim 4, characterized in that The small model is trained on its local data, including: Selecting part of the local data, and training the part of the local data to obtain the corresponding local model parameters; After the master model feeds back the global model parameters to the shared buffer area, the method further includes: Any of the small models reads the global model parameters from the shared cache area, and uses the global model parameters to train the local partial data in the local data to obtain the incrementally updated local model parameters.

7. The method according to any one of claims 1 to 6, characterized in that: The shared buffer area records the average local model parameters, the total model parameters and the state identifier in the form of a state bitmap.

8. The method according to any one of claims 1 to 6, characterized in that: The local data is clustering data for each small model achieved by combining MKM and LSH.

9. The method according to claim 8, characterized in that When the local data is clustered by combining MKM and LSH, the following steps are included: Mapping a data point Xb in the collected data subset to a corresponding hash bucket using a hash function group to obtain an initial hash bucket of the data point Xb, wherein each of the hash buckets corresponds to a cluster center point, and the data point Xb is any data point in the data subset; Determine the number of the initial hash buckets, and calculate the distance between the data point Xb and the cluster center point corresponding to each of the initial hash buckets; The hash bucket to which the data point Xb finally belongs is determined according to the number of the initial hash buckets and the distance.

10. The method according to claim 9, characterized in that The determining, according to the number of the initial hash buckets and the distance, the hash bucket to which the data point Xb finally belongs includes: If the number of the initial hash buckets is two or more, the cluster center points of the initial hash buckets are used as candidate cluster center points; Calculate the distance between the data point Xb and each of the candidate cluster center points, and use the initial hash bucket where the candidate cluster center point closest to the data point Xb is located as the hash bucket to which the data point Xb finally belongs; If the number of the initial hash bucket is one, the initial hash bucket is determined to be the hash bucket to which the data point Xb finally belongs.

11. The method according to claim 9, characterized in that The method of mapping the data point Xb in the collected data subset to the corresponding hash bucket by using the hash function group to obtain the initial hash bucket of the data point Xb includes: The data point Xb is input into the hash function H l , calculating the hash value of the data point Xb, where l is an integer greater than or equal to 1 and less than or equal to L, and L is the number of cluster center points, which is determined by the number of small models in the topic model; Compare the hash value with the cluster point hash value of each hash bucket to obtain the closest cluster point hash value, wherein the cluster point hash value is obtained by the cluster center point of each hash bucket through the hash function H l Calculated; The hash bucket corresponding to the hash value of the closest cluster point is used as the initial hash bucket.

12. A topic model, characterized in that The topic model is a topic model based on the MoE framework, including multiple small models and a main model that is communicatively connected with the multiple small models; wherein: Any of the small models is used to obtain corresponding local model parameters after training its local data, cache the local model parameters in a shared cache area, and refresh the corresponding state flag; The main model is used to identify the state identifier, and read the local model parameters from the shared buffer area according to the state identifier and calculate the feedback global model parameters; Any of the small models is also used to continue training its local data to obtain new local model parameters when the global model parameters fed back by the main model are not identified and the other small models have not completed the local model parameter caching, and to update the corresponding local model parameters in the shared cache area according to the new local model parameters and the previous local model parameters.

Images