Methods and systems for multi-view and multi-source transfer in neural topic modeling

Abstract

The invention relates to a computer-implemented method of neural topic modeling (NTM), a corresponding computer program, a computer-readable medium and a data processing system. In the method of NTM global view transfer (GVT) with or without multi-source transfer (MST) or multi-view transfer (MTV), i.e. jointly applied GVT and local view transfer (LVT) is utilized. For GVT, a pre-trained topic knowledge base (KB) of latent topic features is prepared and knowledge is transferred to the target by GVT via learning meaningful latent topic features under the guidance of relevant latent topic features of the topic KB. This is achieved by extending the loss function and minimizing the extended loss function. Furthermore, for MVT, additionally a pre-trained word embedding KB of word embeddings is prepared and knowledge is transferred to the target by LVT via learning meaningful word embeddings under the guidance of relevant word embeddings of the word embedding KB. This is achieved by extending for computing pre-activated terms.

Description

Technical Field

[0001] This invention relates to a computer implementation method for neural topic modeling (NTM), as well as a corresponding computer program, a corresponding computer-readable medium, and a corresponding data processing system. In particular, the NTM method utilizes global view transfer (GVT) or multi-view transfer (MTV) with or without multi-source transfer (MST), and in multi-view transfer (MTV), GVT and local view transfer (LVT) are jointly applied. Background Technology

[0002] Probabilistic topic models, such as LDA (Blei et al., 2003, Latent dirichlet allocation. Journal of Machine Learning Research, 3: 993-1022), replicated softmax (RSM) (Salakhutdinov and Hinton, 2009, Replicated softmax: an undirected topic model. In Advances in Neural Information Processing Systems 22: 23rd Annual Conference on Neural Information Processing Systems, pages 1607-1614. Curran Associates, Inc.), and Document Neural Autoregressive Distribution Estimator (DocNADE) (Larochelle and Lauly, 2012, A neural autoregressive topic model. In Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems, pages 2717-2725), are frequently used to extract topics from text collections and learn latent document representations to perform natural language processing tasks such as information retrieval (IR). Although they have been shown to be powerful in modeling large text corpora, topic modeling (TM) remains challenging, especially in sparse data settings (e.g., on corpora of short texts or a small number of documents).

[0003] Word embeddings (Pennington et al., 2014, Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543. Association for Computational Linguistics) have local context (views) in the sense that they are learned based on local collocation patterns in a text corpus, where the representation of each word depends on a local context window (Mikolov et al., 2013, Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems, pages 3111-3119) or a function of one or more sentences (Peterset et al., 2018, Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter) (Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227-2237). Therefore, word occurrence is modeled at a fine-grained level. Word embeddings can be used in (neural) topic modeling to address the aforementioned data sparsity problem.

[0004] On the other hand, topics (Blei et al., 2003) have a global word context (view): Topic modeling TM infers the distribution of topics across documents in the corpus and assigns a topic to each word occurrence, where the assignment depends equally on all other words appearing in the same document. Therefore, it learns from word occurrences across documents and encodes coarse-grained descriptions. Unlike word embeddings, topics can capture the thematic structure (topic semantics) in the underlying corpus.

[0005] Although word embeddings and topics are complementary in terms of how they represent things, they differ in how they learn from the observed word occurrences in text corpora.

[0006] To alleviate the problem of data sparsity, recent work (Das et al., (2015), Gaussian Ida for topic models with word embeddings. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 795-804. Association for Computational Linguistics; Nguyen et al., 2015, Improving topic models with latent feature word representations. TACL, 3: 299-313; and Gupta et al., 2019, Document informed neural auto regressive topic models with distributional prior. In Proceedings of the Thirty-Third AAAIConference on Artificial Intelligence) has shown that TMs can be improved by introducing external knowledge, where they only utilize pre-trained word embeddings (i.e., local views). However, word embeddings ignore topic-contextualized structures (i.e., document-level semantics) and cannot handle ambiguity.

[0007] Furthermore, when the domain is transferred and not handled correctly, knowledge transfer via word embeddings is prone to negative transfer in the target domain (Cao et al., 2010, Adaptive transfer learning. In Proceedings of the Twenty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2010, Atlanta, Georgia, USA, July 11-15, 2010. AAAIPress). For example, consider the target domain... T Short text documents in v [Apple gained its market share in the US]. Here, the word "Apple" refers to the company, and therefore to the document... v and its theme Z In both cases, the word vectors for apples (about fruit) are irrelevant sources of knowledge transfer. Summary of the Invention

[0008] The object of this invention is to overcome or at least mitigate these problems by providing a computer implementation method of neural topic modeling (NTM) according to independent claim 1, as well as corresponding computer programs, corresponding computer-readable media, and corresponding data processing systems according to the other independent claims. Further improvements to the invention are the subject of the dependent claims.

[0009] According to a first aspect of the invention, a method for using a given word v i ( i = l... D (document) v In the case of, targeting T The probabilistic or neural autoregressive topic model uses a computer implementation of neural topic modeling (NTM) in an autoregressive neural network (NN) with global view transfer (GVT), comprising the following steps: preparing a pre-trained topic knowledge base (KB); transferring knowledge to the target object via GVT. T ; and minimizing the expanded loss function In the step of preparing pre-trained topics (KB), latent topic features are prepared. The pre-trained topics (KBs), among which k Indicating the source of hidden topic features S k ( k ≥1) quantity, H Indicates the dimension of the hidden topic, and KIndicates vocabulary size. In knowledge transfer to the target audience via GVT T In the steps, GVT is used via the relevant hidden topic features in topic KB. Z k Under guidance, learn meaningful latent topic features to transfer knowledge to the target. T Knowledge transfer to the target through GVT T The steps include expanding the loss function. The sub-steps in expanding the loss function. In the steps, weighted relevant latent topic features are utilized. Z k Regularization terms to expand the target T Documents v The probability or loss function of a neural autoregressive topic model To form the expanded loss function The loss function It is each word in the autoregressive neural network. v i joint probability The negative log-likelihood for each word v i that probability Based on the preceding words v <i The probability of minimizing the expanded loss function. In the step, minimize the expanded loss function To determine the minimum overall loss.

[0010] According to a second aspect of the invention, a computer program includes instructions that, when executed by a computer, cause the computer to perform the steps of the method according to a first aspect of the invention.

[0011] According to a third aspect of the invention, a computer program according to a second aspect of the invention is stored on a computer-readable medium.

[0012] According to a fourth aspect of the invention, a data processing system includes means for performing the steps of the method according to a first aspect of the invention.

[0013] The probabilistic or neuroautoregressive topic model (the model below) is arranged and configured to: determine the input text or input document vThe topic (such as a short article, a piece of writing, etc.) can be used to model a neural network (NN), such as a deep neural network (DNN), a recurrent neural network (RNN), a feedforward neural network (FFNN), a convolutional neural network (CNN), a long short-term memory network (LSTM), a deep belief network (DBN), and a large memory storage and retrieval neural network (LAMSTAR).

[0014] You can determine the input document v The neural network (NN) can be trained on content and / or topics. Any training method can be used to train the NN. Specifically, the GLOWN algorithm (Pennington et al., 2014, GLOWN: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543. Association for Computational Linguistics) can be used to train the NN.

[0015] document v Including words v 1... v D The number of words, D, is greater than 1. The model determines each word one by one. v i The joint probability, or more precisely, the autoregressive condition. Each joint probability Documentation available from FFNN v The corresponding preceding word in the sequence The model is based on the probability of [the event / condition]. This model can utilize a non-linear activation function. g (·) (such as the sigmoid function, hyperbolic tangent (tanh) function, etc.) and at least one weight matrix, preferably two weight matrices, especially the encoding matrix. and decoding matrix To calculate each probability .

[0016] probability Combined into a joint distribution and the loss function —It is a joint distribution p ( v The negative log-likelihood of ) — providing a basis for .

[0017] Knowledge transfer is based on knowledge from at least one source S k ( k ≥1) pre-trained latent topic features The theme KB. Hidden theme features Z k Includes a group of words belonging to the same topic, such as, for example, { Profits, growth, stock, Apple Fruit, decline, consumer, purchase, billion, shares}→ trade Therefore, the topic KB includes global information about the topic. For GVT, a regularization term is added to the loss function. Thus, the expanded loss function is obtained. Therefore, information from the global view of the topic is transferred to this model. Regularization terms are based on topic features. Z k And may include: weights γ k Its control over thematic characteristics Z k The degree of imitation; alignment matrix Its target T and the kth source S k Latent topic alignment in; and encoding matrix W Therefore, learning (especially) W The generative process of meaningful (latent) topic features in [the context] is... The relevant features in the text provide guidance.

[0018] Finally, the expanded loss function is minimized (e.g., by gradient descent) in the following way. Or more precisely, the overall loss: that is, making W (Hidden) theme features Z k Simultaneously from at least one source S k Inheriting relevant theme features and targeting T Generate meaningful representations.

[0019] Given that word and topic representations encode complementary information, previous work has not considered knowledge transfer via (pre-trained latent) topics from large corpora (i.e., GVT). In the case of GVT, the underlying corpus (target) is captured. T The thematic structure (topic semantics) in the input document. This leads to the input document v A more reliable determination of the subject.

[0020] According to the improvements of the present invention, the probabilistic or neural autoregressive topic model is based on the DocNADE architecture.

[0021] DocNADE (Larochelle and Lauly, 2012, A neural autoregressive topic model. In Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems, pages 2717-2725) is an unsupervised neural network-based probabilistic or neural autoregressive topic model inspired by the benefits of NADE (Larochelle and Murray, 2011, The neural autoregressive distribution estimator. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, AISTATS, volume 15 of JMLR Proceedings, pages 29-37. JMLR.org) and RSM (Salakhutdinov and Hinton, 2009, Replicated softmax: an undirected topic model. In Advances in Neural Information Processing Systems 22: 23rd Annual Conference on Neural Information Processing Systems, pages 2717-2725). 1607-1614. Curran Associates, Inc.) architecture. Due to the difficulty in obtaining negative log-likelihood. The approximate gradient of DocNADE makes RSM difficult, while NADE does not require such an approximation. On the other hand, RSM is a generative model for word counting, while NADE is limited to binary data. Specifically, DocNADE takes input documents... v words in v 1... v D joint probability distribution p ( v Factorization is the process of factoring a probability or conditional distribution. The product of these probabilities is used, and each probability is modeled via an FFNN to efficiently compute the document representation.

[0022] For size D Input document v = ( v 1... v D ), each word v i Take size K The value of the vocabulary {1... K DocNADE learns topics through language modeling (Bengio et al., 2003, A neural probabilistic language model. Journal of Machine Learning Research, 3: 1137-1155), and decomposes the joint distribution. p ( v ), such that each probability or autoregressive condition Input document used by FFNN v The corresponding preceding word in the sequence v <i To model:

[0023]

[0024] in It is a probability function:

[0025]

[0026] in , v <i It is made by all v q The subvectors formed make q < i ,Right now , g (·) is a non-linear activation function, and and It is the deviation parameter vector ( c Pre-activation is possible a (See below).

[0027] Using DocNADE, the extended loss function It is given by the following formula:

[0028]

[0029] in It is an alignment matrix. γ k yes Z k The weight, and control within T middle W Thematic characteristics Z k The degree of imitation, and j Indicator Topic Matrix Z k The topic (i.e., row) index in the table.

[0030] According to an improvement of the present invention, multi-view transfer (MVT) is used by additionally employing local view transfer (LVT), wherein the computer implementation method further includes the following main steps: preparing pre-trained word embeddings KB; and transferring knowledge to the target via LVT. T In the step of preparing the pre-trained word embedding KB, the word embeddings are prepared. The pre-trained word embeddings KB, where E Indicates the dimension of word embedding. This is used in transferring knowledge to the target audience via LVT. T In the steps, LVT is used via the relevant word embeddings in the word embedding KB. E k Learn meaningful word embeddings under guidance to transfer knowledge to the target audience. T LVT transfers knowledge to the target audience. T The steps include expanding the computation of pre-activation. a The sub-steps of the item. In the extension used to compute pre-activation. a In the steps of the item, weighted related latent word embeddings are used. E k To expand the target T The probability of pre-activation of a neural autoregressive topic model a To form an expanded pre-activation a ext The pre-activation a Control each word v i probability The preceding words v <i Activation of the autoregressive neural network.

[0031] First, word and topic representations are learned from multiple source domains. Then, knowledge is transferred within neural topic modeling via an MVT that includes (first) LVT and (then) GVT, by jointly using word embeddings and complementary topic representations. Furthermore, guidance is provided through at least one source domain. S k ( kThe (unsupervised) generative process of learning latent topics in the target domain by using latent topic features and words of ≥1) is used to enable the target T The hidden themes on it become meaningful.

[0032] In the case of LVT, by using at least one source S k ( k ≥1) pre-trained word embeddings Word embedding KB to perform target T Knowledge transfer. Word embeddings can be a list of the nearest neighbors of a word, such as... apple →{ Apples, pears, fruits, berries, multiple pears, strawberries Pre-activation of autoregressive neural network models a Controlling the nodes of the autoregressive neural network for each preceding word v <i Whether it is activated and the strength of activation. Utilizing related word embeddings. E k To expand pre-activation a Thus, the extended pre-activation is obtained. a ext The related words are embedded E k By weight Weighted.

[0033] Extended preactivation in DocNADE a ext It is given by the following formula:

[0034]

[0035] And the probability function in DocNADE It is given by the following formula:

[0036]

[0037] in c = a , λ k yes E k The weight, λ k Based on the target and at least one source S k To control the overlap between the domains T The amount of knowledge transferred from China.

[0038] Therefore, an unsupervised neural topic modeling framework is provided that jointly leverages (external) complementary knowledge, i.e., knowledge from at least one source. S kHidden word and topic features are used to mitigate data sparsity. A computer-based implementation using MVT (Multi-View-Template) can be used to better analyze documents. v Modeling is performed, and given meaningful word and topic representations, noisy topics Z can be modified for coherence.

[0039] According to the improvements of the present invention, multi-source migration (MST) is used, wherein the latent topic features of topic KB are... And optionally or additionally word embeddings of KB Originating from more than one source S k ( k >1).

[0040] Hidden theme features Z k This includes a group of words belonging to the same topic. Typically, there are several topic-word associations across different fields (e.g., across different topics). Z 1- Z Of the 4, Z 1(S 1 ):{ Profit, growth, stocks, Apple, decline, consumption Consumers, purchases, billions, shares}→ trade ; Z 2(S 2 ):{ Smartphone, iPad, Apple, app, iPhone, device, telephone, platform Board equipment}→ Product Line ; Z 3(S 3 ):{ Microsoft, Mac, Linux, IBM, iOS, Apple, XP, Windows}→ Operating System / Public manage ; Z 4(S 4 ):{ Apple, conversation, computer, stock, disease, driver, electronic device, profit, iOS}→ In a given noisy subject (e.g., Z 4) and meaningful themes (e.g., Z 1- Z In case 3), multiple related (source) domains must be identified, and their word and topic representations must be transferred to facilitate meaningful learning in a sparse corpus. To better handle polysemy and alleviate the data sparsity problem, GVT with latent topic features (topic contextualization) is utilized, and optionally, features from multiple sources or source domains are also utilized. S k ( k LVT of word embeddings in MST of ≥1).

[0041] Target T With source S kTopic alignment between them needs to be completed. For example, in the DocNADE architecture, in the extended loss function... middle, j Indicator Hidden Topic Matrix Z k The topic (i.e., row) index in the database. For example, the first source. S 1 The first theme With the goal T of W Align the first row vector (i.e., the topic). However, other topics (e.g., and It needs to be aligned with the target topic. When targeting multiple sources S k When performing LVT and GVT in MVT, the advantages of both MVT and MST are combined in knowledge transfer by using these two complementary representations.

[0042] In the following, an exemplary computer program according to the second aspect of the invention is given as an exemplary algorithm in pseudocode, which includes instructions corresponding to steps of a computer-implemented method according to the first aspect of the invention, which will be executed by a data processing apparatus (e.g., a computer) according to the fourth aspect of the invention:

[0043] Input: A target training document v , k =| S |Source / Source Domain S k

[0044] Input: hidden topic Theme KB

[0045] Input: Word embedding matrix Word embedding KB

[0046] parameter:

[0047] Hyperparameters:

[0048] initialization: and

[0049] for i From 1 to D do

[0050] ,in

[0051]

[0052]

[0053] Pre-activation is calculated at step i:

[0054] if LVT then

[0055] From the source domain S k get v i Word embedding

[0056]

[0057]

[0058] if GVT then

[0059] . Attached Figure Description

[0060] The invention and its technical field will then be explained in further detail with reference to exemplary embodiments shown in the accompanying drawings. These exemplary embodiments are provided only to aid in a better understanding of the invention and should not in any way be construed as limiting the scope of the invention. In particular, it is possible to extract aspects of the subject matter described in the drawings and combine them with other components and findings of this specification or the drawings, unless explicitly described differently. The same reference numerals refer to the same objects, allowing for supplementary use of the explanations in the other figures.

[0061] Figure 1 A schematic flowchart illustrating an embodiment of a computer implementation method according to a first aspect of the present invention using GVT is shown.

[0062] Figure 2 It shows the use of Figure 1 An illustrative overview of an embodiment of a computer implementation method of GVT according to a first aspect of the present invention.

[0063] Figure 3 A schematic flowchart illustrating an embodiment of a computer implementation method according to a first aspect of the present invention using MVT is shown.

[0064] Figure 4 It shows the use of Figure 3 An illustrative overview of an embodiment of a computer implementation method of MVT according to a first aspect of the present invention.

[0065] Figure 5 A schematic overview of an embodiment of a computer implementation method according to the first aspect of the invention is shown, using GVT or MVT and MST.

[0066] Figure 6 A schematic diagram of a computer-readable medium according to a third aspect of the present invention is shown.

[0067] Figure 7 A schematic diagram of a data processing system according to a fourth aspect of the present invention is shown. Detailed Implementation

[0068] exist Figure 1 In the diagram, the first aspect of the invention is schematically depicted in a given word. v i Documents v In the case of, targeting T A flowchart illustrating an exemplary embodiment of a computer implementation method for a probabilistic or neural autoregressive topic model using a neural topic modeling (NTM) in an autoregressive neural network (NN) with global view transfer (GVT). The steps of this computer implementation method are implemented in a computer program according to a second aspect of the invention. The probabilistic or neural autoregressive topic model is a DocNADE architecture (hereinafter referred to as the DocNADE model). [Document] v Include D One word, D ≥1.

[0069] The computer implementation method includes the following steps: preparing (3) a pre-trained subject knowledge base (KB); transferring (4) the knowledge to the target through GVT. T ; and minimizing (5) the expanded loss function Knowledge transfer (4) to the target through GVT T The steps include expanding the (4a) loss function. Sub-steps.

[0070] In the step of preparing (3) pre-trained topic KB, prepare from at least one source S k ( k ≥1) pre-trained latent topic features And it is provided as a topic KB to the DocNADE model.

[0071] Knowledge transfer (4) to the target through GVT T In this step, the prepared topic KB is used to provide the DocNADE model with information from a global view of the topic. This transfer of information from the global view of the topic to the DocNADE model is performed in the extended (4a) loss function. This is accomplished in a sub-step that expands the loss function of the DocNADE model using a regularization term. This is done through the loss function. It is a document v words v 1... v D joint probability distribution p ( v The negative log-likelihood of ). Joint probability distribution. p ( v Based on each word v 1... v D probability or autoregressive condition Autoregressive conditions Including the preceding words v <i The probability. The DocNADE model uses a non-linear activation function. g (·) (such as the sigmoid function, hyperbolic tangent (tanh) function, etc.) and two weight matrices—the encoding matrix. (Encoding matrix and decoding matrix of the DocNADE model) (Decoding matrix of the DocNADE model), to calculate each probability .

[0072]

[0073] in

[0074]

[0075] in It is a probability function:

[0076]

[0077] in , v <i It is made by all v q The subvectors formed make q < i ,Right now , g (·) is a non-linear activation function, and and It is a vector of deviation parameters, in particular, c It is pre-activated a (See below).

[0078] loss function It is extended using a regularization term, which is based on topic features. Z k And includes: weights γk Its control over thematic characteristics Z k The degree of imitation; alignment matrix Its target T and the kth source S k Latent topic alignment in DocNADE; and the encoding matrix of the DocNADE model. W .

[0079]

[0080] Minimize the extended loss function (5) In the step, minimize the expanded loss function Here, it can be minimized using methods such as gradient descent.

[0081] exist Figure 2 The middle section schematically depicts Figure 1 GVT is an embodiment of the computer implementation method.

[0082] Word-by-word analysis by the DocNADE model v 1... v D (Visible cell) input document v Proceed step by step. (Previous words) v <i of The DocNADE model uses the bias parameter c (Hidden bias) is determined. Based on Decoding matrix U Sum and deviation parameters b Words are calculated using the DocNADE model. v 1... v D The probability of each of them, or more precisely, the autoregressive condition. .

[0083] like Figure 2 The illustration is shown for each word. v i ( i = l... D Different topics (here, for example, topic #1, topic #2, and topic #3) have different probabilities. All words... v 1... v D The probabilities are combined to determine the input document. v The most likely theme.

[0084] exist Figure 3The diagram schematically depicts a flowchart of an exemplary embodiment of a computer-implemented method according to the first aspect of the present invention using Multi-View Transition (MVT). This embodiment corresponds to the use of GVT. Figure 1 The embodiments are described, and extended by partial view migration (LVT). The steps of this computer implementation method are implemented in a computer program according to a second aspect of the invention.

[0085] The computer implementation method includes Figure 1 The method comprises the following steps, and further includes the following main steps: preparing (1) pre-trained word embeddings KB; and transferring (2) knowledge to the target via LVT. T Knowledge transfer (2) to the target through LVT T The steps include extended (2a) pre-activation a Sub-steps.

[0086] In the step of preparing (1) the pre-trained word embedding KB, the word embeddings from at least one source are prepared. S k ( k ≥1) pre-trained word embeddings And it is provided to the DocNADE model as a word embedding KB.

[0087] Knowledge transfer (2) to the target through LVT T In this step, the prepared word embeddings KB are used to provide the DocNADE model with information from a local view of the words. This transfer of information from the local view of the word embeddings to the DocNADE model is performed during the pre-activation of extension (2a). a This is completed in a sub-step. It utilizes related word embeddings. E k To expand pre-activation a Thus, the extended pre-activation is obtained. a ext The related words are embedded E k By weight λ k Weighted.

[0088] Extended preactivation in the DocNADE model a ext It is given by the following formula:

[0089]

[0090] And the probability function in the DocNADE model Then it is given by the following formula:

[0091]

[0092] in c = a , λ k yes E k The weight, λ k Based on the target and at least one source S k To control the overlap between the domains T The amount of knowledge transferred from China.

[0093] exist Figure 4 The diagram illustrates the process of first using LTV and then using GVT. Figure 3 The MVT is an embodiment of the computer implementation method. Figure 4 Corresponding to LTV extension Figure 2 .

[0094] For input documents v Each word v i Select relevant words to embed E k And it is set as a deviation parameter through expansion. c The corresponding pre-activation a This embeds relevant words. E k Introduced to use specific λ k Weighted probability function middle.

[0095] exist Figure 5 The diagram schematically depicts the scene in Figure 1 or Figure 3 Multi-source migration (MST) is used in embodiments of the computer implementation method.

[0096] Using source corpus DC k Multiple sources in form S k Includes hidden topic features Z k And optionally include word embeddings E k (Not depicted). Target T With source S k Topic alignment needs to be done in MST. Hidden topic features. Z k Each row in the table is a topic embedding that explains the source corpus. DC kThe basic principal structure. Here, TM refers to the DocNADE model. The extended loss function of the DocNADE model... middle, j Indicator Hidden Topic Matrix Z k The topic (i.e., row) index in the database. For example, the first source. S 1 The first theme With the goal T of W Align the first row vector (i.e., the topic). However, other topics (e.g., and It needs to be aligned with the target theme.

[0097] exist Figure 6 The embodiment of the computer-readable medium 20 according to the third aspect of the present invention is illustrated schematically.

[0098] Here, exemplarily, a computer-readable storage disk 20, such as an optical disc (CD), digital video disc (DVD), high-definition DVD (HD DVD), or Blu-ray disc (BD), has stored thereon the invention according to the second aspect and as Figures 1 to 5 The computer program is schematically illustrated. However, the computer-readable medium may also be a data storage device, such as a magnetic storage device / memory (e.g., magnetic core memory, magnetic tape, magnetic card, magnetic stripe, magnetic bubble storage device, drum storage device, hard disk, floppy disk, or removable storage device), an optical storage device / memory (e.g., holographic memory, light strip, Tesa tape, laser disk, phasewriter (phasewriter dual, PD) or ultra-density optical (UDO)), a magneto-optical storage device / memory (e.g., mini-disk or magneto-optical disk (MO-Disc)), a volatile semiconductor / solid-state memory (e.g., random access memory (RAM), dynamic RAM (DRAM), or static RAM (SRAM)), a non-volatile semiconductor / solid-state memory (e.g., read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically erasable EPROM (EEPROM), flash EEPROM (e.g., USB stick), ferroelectric RAM (FRAM), magnetoresistive RAM (MRAM), or phase-change RAM).

[0099] exist Figure 7 The image schematically depicts an embodiment of a data processing system 30 according to a fourth aspect of the present invention.

[0100] The data processing system 30 can be a personal computer (PC), a laptop computer, a tablet device, a server, a distributed system (e.g., a cloud system), etc. The data processing system 30 includes a central processing unit (CPU) 31, a memory with random access memory (RAM) 32 and non-volatile memory (MEM, e.g., hard disk) 33, a human-machine interface device (HID, e.g., keyboard, mouse, touch screen, etc.) 34, and an output device (MON, e.g., monitor, printer, speaker, etc.) 35.

[0101] CPU 31, RAM 32, HID 34, and MON 35 are communicatively connected via a data bus. RAM 32 and MEM 33 are communicatively connected via another data bus. According to the second aspect of the invention... Figures 1 to 3 The computer program schematically depicted can be loaded into RAM 32 from MEM 33 or another computer-readable medium 20. According to this computer program, the CPU executes the program according to the first aspect of the invention. Figures 1 to 5 Steps 1 to 5, or more precisely steps 3 to 5, of the computer implementation method are schematically depicted in the diagram. This execution can be initiated and controlled by a user via HID 34. The status and / or result of the executed computer program can be indicated to the user via MON 35. The result of the executed computer program can be permanently stored on a non-volatile MEM 33 or another computer-readable medium.

[0102] Specifically, the CPU 31 and RAM 33 used to execute the computer program may include, for example, several CPUs 31 and several RAMs 33 in a computing cluster or cloud system. The HID 34 and MON 35 used to control the execution of the computer program may be included by different data processing systems, such as terminals communicatively connected to the data processing system 30 (e.g., a cloud system).

[0103] Although specific embodiments have been illustrated and described herein, those skilled in the art will appreciate that various alternative and / or equivalent implementations exist. It should be understood that one or more exemplary embodiments are merely examples and are not intended to limit the scope, applicability, or configuration in any way. Rather, the foregoing overview and detailed description will provide those skilled in the art with a convenient roadmap for implementing at least one exemplary embodiment, and it is to be understood that various changes may be made to the function and arrangement of the elements described in the exemplary embodiments without departing from the scope set forth in the appended claims and their legal equivalents. Generally, this application is intended to cover any adaptations or variations of the specific embodiments discussed herein.

[0104] In the foregoing detailed description, various features have been grouped together in one or more examples for the purpose of simplifying this disclosure. It is to be understood that the above description is intended to be illustrative and not restrictive. It is intended to cover all alternatives, modifications, and equivalents that may be included within the scope of this invention. Many other examples will become apparent to those skilled in the art upon review of the foregoing specification. The specific nomenclature used in the foregoing specification is intended to provide a comprehensive understanding of the invention. However, it will be apparent to those skilled in the art, given the description provided herein, that specific details are not required to practice the invention. Therefore, the foregoing description of specific embodiments of the invention is presented for purposes of illustration and description. They are not intended to be exhaustive or to limit the invention to the precise forms disclosed; obviously, many modifications and variations are possible in light of the foregoing teachings. The embodiments were chosen and described in order to best explain the principles of the invention and its practical application, thereby enabling others skilled in the art to best utilize the invention and various embodiments having various modifications as suited to the intended particular use. Throughout this specification, the terms “comprising” and “wherein” are used as simple English equivalents of the corresponding terms “including” and “in which”, respectively. Furthermore, the terms “first,” “second,” and “third,” etc., are used merely as labels and are not intended to impose numerical requirements regarding the importance of their objects or to establish any order of importance of their objects. In the context of this description and claims, the conjunction “or” should be understood as inclusive (“and / or”) rather than exclusive (“either…or”).

[0105] Reference Symbol List

[0106] 1. Pre-trained word embeddings (KB) for preparing word embeddings

[0107] 2. Transferring knowledge to the target through LVT

[0108] 2a Extended Terminology for Computing Preactivation

[0109] 3. Prepare pre-trained topic KBs for latent topic features

[0110] 4. Transfer knowledge to the target through GVT

[0111] 4a Expanded Loss Function

[0112] 5. Minimize the expanded loss function

[0113] 20 Computer-readable media

[0114] 30 Data Processing System

[0115] 31. Central Processing Unit (CPU)

[0116] 32 Random Access Memory (RAM)

[0117] 33. Non-volatile memory (MEM)

[0118] 34 Human-Machine Interface (HID) Devices

[0119] 35 Output device (MON).

Claims

1. A method for finding a given word v i i = l... D Documents v In the case of, targeting T A computer implementation of a probabilistic or neural autoregressive topic model using a global view transfer (GVT) autoregressive neural network (NN) for neural topic modeling (NTM) includes the following steps: - Prepare hidden theme features The pre-trained topic knowledge base KB(3), in which k Indicating the source of hidden topic features S k k Quantity ≥ 1 H Indicates the dimension of the hidden topic, and K Indicator word size, where latent topic features Z k This includes a group of words belonging to the same topic; - Via GVT via relevant hidden topic features in topic KB Z k Under guidance, learn meaningful latent topic features to transfer knowledge to the target. T (4) includes the following sub-steps: - Utilizing relevant latent topic features, including weighted relevance features Z k Regularization terms to expand the target T Documents v The probability or loss function of a neural autoregressive topic model To form the expanded loss function (4a), the loss function It is each word in the autoregressive neural network. v i joint probability The negative log-likelihood for each word v i that probability Based on the preceding words v <i The regularization term is included in the alignment matrix multiplied by the encoding matrix. Features of hidden themes Z k The difference between them; as well as - Minimize the expanded loss function To determine the minimum total loss (5).

2. The computer implementation method according to claim 1, wherein the probabilistic or neural autoregressive topic model is the DocNADE architecture.

3. The computer implementation method according to claim 1 or 2, further comprising the following main steps by additionally using Local View Transition LVT to utilize Multi-View Transition MVT: - Prepare word embeddings The pre-trained word embeddings KB(1), where E Indicates the dimension of word embedding; - Via LVT via related word embeddings in word embedding KB E k Learn meaningful word embeddings under guidance to transfer knowledge to the target audience. T (2) includes the following sub-steps: - Utilizing weighted related latent word embeddings E k To extend the computational target T The probability of pre-activation of a neural autoregressive topic model a The items are used to form an expanded pre-activation. a ext (2a), this pre-activation a Control each word v i probability The preceding words Activation of the autoregressive neural network.

4. The computer implementation method according to claim 1, using multi-source migration MST, wherein the latent topic features of topic KB and / or word embeddings KB word embeddings Originating from more than one source S k ( k > 1).

5. A computer-readable medium (20) having stored thereon computer program instructions that, when executed by a computer, cause the computer to perform the steps of the method according to any one of claims 1 to 4.

6. A data processing system (30) comprising means for performing the steps of the method according to any one of claims 1 to 4.

Images