Category embedding methods, devices, equipment, and readable media in machine recognition scenarios

Abstract

This disclosure provides a category embedding method, apparatus, device, and readable medium for machine recognition scenarios. The category embedding method for machine recognition scenarios includes: learning from a corpus to determine supervisory data in response to a received classification task, the supervisory data having conceptual representations; performing maximum likelihood estimation on the sample distribution of the supervisory data; determining a lower bound for the maximum likelihood estimation based on a dynamic routing algorithm in a capsule network, and determining a loss function for the classification model based on the lower bound of the maximum likelihood estimation; inputting the supervisory data into the classification model to obtain sample representations; constructing a category embedding set based on the conceptual representations, and fusing the sample representations with the category embedding set to achieve category embedding for the classification task. Through the embodiments of this disclosure, the generalization and reliability of embedded categories can be improved, thereby contributing to improving the reliability and accuracy of machine recognition.

Description

Technical Field

[0001] This disclosure relates to the field of machine learning technology, and more specifically, to a method, apparatus, device, and readable medium for category embedding in machine recognition scenarios. Background Technology

[0002] Currently, label embedding is a technique that uses vector representation to represent category information. It can be used to enhance the performance of intent recognition tasks and has shown certain effectiveness in many tasks. The key points are the embedded content and the embedding method.

[0003] In related technologies, type-related information, including hierarchical information, detailed descriptions, and attribute phrases, is used to represent categories, or categories are randomly initialized as vectors, and the final representation is obtained through learning from supervised data. In intent recognition tasks that leverage category embedding, a crucial step is obtaining the category embedding vectors.

[0004] However, existing solutions for category embedding either rely on manual discovery or lack generalization. For example, constructing hierarchical information for a category or describing its meaning using text requires human intervention. Furthermore, if only randomly initialized vectors are used and learning relies solely on supervised data, then the performance will depend on the characteristics of the supervised data; the size of the data will affect the quality of the representation, and the domain will affect the knowledge contained in the representation, thus resulting in a loss of generalization.

[0005] It should be noted that the information disclosed in the background section is only used to enhance the understanding of the background of this disclosure, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0006] The purpose of this disclosure is to provide a machine recognition device, electronic device, and readable medium for at least a certain extent to overcome the problem of poor generalization of category embedding due to the limitations and defects of related technologies.

[0007] According to a first aspect of the present disclosure, a category embedding method for a machine recognition scenario is provided, comprising: in response to a received classification task, learning from corpus to determine supervision data, the supervision data having conceptual representations; performing maximum likelihood estimation on the sample distribution of the supervision data; determining a lower bound of the maximum likelihood estimation based on a dynamic routing algorithm in a capsule network, and determining a loss function of a classification model based on the lower bound of the maximum likelihood estimation; inputting the supervision data into the classification model to obtain sample representations; constructing a category embedding set based on the conceptual representations, and fusing the sample representations with the category embedding set to achieve category embedding for the classification task.

[0008] In one exemplary embodiment of this disclosure, in response to a received classification task, learning from the corpus to determine supervised data includes: performing unsupervised task learning and / or supervised task learning on corpora from multiple specified domains to obtain a sample dataset, wherein the specified domains include at least one of e-commerce, finance, news, law, computer, electronics, chemistry, and biology; determining positive and negative samples based on the sample dataset; and using the contrastive learning algorithm to perform contrastive learning on the positive and negative samples to determine the supervised data.

[0009] In one exemplary embodiment of this disclosure, determining positive and negative samples based on the sample dataset includes: if the sample dataset was determined through the unsupervised task learning, then determining the positive samples in the sample dataset through two dropout algorithms; and determining the other samples in the sample dataset besides the positive samples as the negative samples.

[0010] In one exemplary embodiment of this disclosure, determining positive and negative samples based on the sample dataset and performing maximum likelihood estimation on the sample distribution of the supervised data includes: if the sample dataset was determined through the supervised learning task, then determining multiple labels under the supervised learning task; and determining multiple sample data under the same label as the positive and negative samples to each other.

[0011] In one exemplary embodiment of this disclosure, the maximum likelihood estimation of the sample distribution of the supervised data includes: calculating the likelihood of the logarithm of the supervised data; introducing a conceptual variable as a latent variable; and maximizing the result of the likelihood estimation calculation.

[0012] In one exemplary embodiment of this disclosure, introducing conceptual variables as latent variables and maximizing the result of the likelihood estimation calculation includes: introducing conceptual variables as latent variables and inferring the sample distribution of the latent variables; determining the cluster centers of the latent variable distribution based on the inference result; determining the cluster centers as high-level capsules in the capsule network using a dynamic routing algorithm, and determining the other cluster samples in the latent variable distribution besides the cluster centers as low-level capsules in the capsule network; and maximizing the result of the likelihood estimation calculation based on the iteration result of the capsule network.

[0013] In one exemplary embodiment of this disclosure, constructing a category embedding set based on the conceptual representation and fusing the sample representation with the category embedding set to achieve category embedding for the classification task includes: constructing the category embedding set based on the conceptual representation; determining the sample representation set as a low-level capsule of the capsule network and the category embedding set as a high-level capsule of the capsule network using the dynamic routing algorithm; and fusing the sample representation and the category embedding set to achieve category embedding for the classification task.

[0014] According to a second aspect of the present disclosure, a category embedding apparatus for a machine recognition scenario is provided, comprising: a learning module configured to learn from a corpus to determine supervision data in response to a received classification task, the supervision data having conceptual representations; the learning module further configured to perform maximum likelihood estimation on the sample distribution of the supervision data; a determination module configured to determine a lower bound of the maximum likelihood estimation based on a dynamic routing algorithm in a capsule network, and determine a loss function of a classification model based on the lower bound of the maximum likelihood estimation; the learning module further configured to input the supervision data into the classification model to obtain sample representations; and an embedding module configured to construct a category embedding set based on the conceptual representations, and fuse the sample representations with the category embedding set to achieve category embedding for the classification task.

[0015] According to a third aspect of this disclosure, an electronic device is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to perform the method as described in any one of the claims based on instructions stored in the memory.

[0016] According to a fourth aspect of this disclosure, a computer-readable storage medium is provided having a program stored thereon that, when executed by a processor, implements the category embedding method in a machine recognition scenario as described in any one of the claims.

[0017] In this embodiment of the disclosure, supervised data is determined by learning from the corpus, and a set of category embeddings is constructed based on the concept representation. The sample representations are then fused with the set of category embeddings to achieve category embedding for the classification task. This improves the representation quality and generalization of category embeddings, thereby enhancing the generalization of machine recognition.

[0018] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0019] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure. It is obvious that the drawings described below are merely some embodiments of this disclosure, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0020] Figure 1 A schematic diagram of an exemplary system architecture for a category embedding scheme applicable to machine recognition scenarios according to embodiments of the present disclosure is shown;

[0021] Figure 2 This is a flowchart of a category embedding method in a machine recognition scenario according to an exemplary embodiment of this disclosure;

[0022] Figure 3 This is a flowchart of another category embedding method in a machine recognition scenario according to an exemplary embodiment of this disclosure;

[0023] Figure 4 This is a flowchart of another category embedding method in a machine recognition scenario according to an exemplary embodiment of this disclosure;

[0024] Figure 5 This is a flowchart of another category embedding method in a machine recognition scenario according to an exemplary embodiment of this disclosure;

[0025] Figure 6 This is a flowchart of another category embedding method in a machine recognition scenario according to an exemplary embodiment of this disclosure;

[0026] Figure 7 This is a flowchart of another category embedding method in a machine recognition scenario according to an exemplary embodiment of this disclosure;

[0027] Figure 8 This is a flowchart of another category embedding method in a machine recognition scenario according to an exemplary embodiment of this disclosure;

[0028] Figure 9 This is a flowchart of a category embedding scheme in a machine recognition scenario according to an exemplary embodiment of this disclosure;

[0029] Figure 10 This is a schematic block diagram of a category embedding scheme in a machine recognition scenario according to an exemplary embodiment of this disclosure;

[0030] Figure 11 This is a block diagram of a category embedding device in a machine recognition scenario according to an exemplary embodiment of this disclosure;

[0031] Figure 12 This is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. Detailed Implementation

[0032] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of the example embodiments to those skilled in the art. The described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the following description, numerous specific details are provided to give a full understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced with one or more of the specific details omitted, or other methods, components, apparatus, steps, etc., can be employed. In other instances, well-known technical solutions are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0033] Furthermore, the accompanying drawings are merely illustrative of this disclosure, and the same reference numerals in the drawings denote the same or similar parts, thus repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0034] Figure 1 A schematic diagram of an exemplary system architecture for a category embedding scheme applicable to machine recognition scenarios according to embodiments of the present disclosure is shown.

[0035] like Figure 1 As shown, system architecture 100 may include one or more of terminal devices 101, 102, and 103, a network 104, and a server 105. Network 104 serves as the medium for providing communication links between terminal devices 101, 102, and 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0036] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, there can be any number of terminal devices, networks, and servers. For example, server 105 could be a server cluster composed of multiple servers.

[0037] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Terminal devices 101, 102, and 103 can be various electronic devices with displays, including but not limited to smartphones, tablets, laptops, and desktop computers, etc.

[0038] In some embodiments, the machine recognition scenario provided in this disclosure is generally executed by server 105, and correspondingly, the category embedding device in the machine recognition scenario is generally disposed in terminal device 103 (or terminal device 101 or 102). In other embodiments, some terminals may have functions similar to those of the server device to execute this method.

[0039] The exemplary embodiments of this disclosure will now be described in detail with reference to the accompanying drawings.

[0040] Figure 2 This is a flowchart of a category embedding method in a machine recognition scenario according to an exemplary embodiment of this disclosure.

[0041] refer to Figure 2 Category embedding methods in machine recognition scenarios may include:

[0042] Step S202: In response to the received classification task, the corpus is learned to determine supervision data, which has conceptual representations.

[0043] Step S204: Perform maximum likelihood estimation on the sample distribution of the supervised data.

[0044] Step S206: Determine the lower bound of the maximum likelihood estimation based on the dynamic routing algorithm in the capsule network, and determine the loss function of the classification model based on the lower bound of the maximum likelihood estimation.

[0045] Step S208: Input the supervised data into the classification model to obtain sample representation.

[0046] Step S210: Construct a category embedding set based on the concept representation, and fuse the sample representation with the category embedding set to achieve category embedding for the classification task.

[0047] In this embodiment of the disclosure, supervised data is determined by learning from the corpus, and a set of category embeddings is constructed based on the concept representation. The sample representations are then fused with the set of category embeddings to achieve category embedding for the classification task. This improves the representation quality and generalization of category embeddings, thereby enhancing the generalization of machine recognition.

[0048] The following section provides a detailed explanation of each step in the category embedding method for machine recognition scenarios.

[0049] In one exemplary embodiment of this disclosure, such as Figure 3 As shown, in response to the received classification task, the corpus is learned to determine the supervised data, including:

[0050] Step S302: Perform unsupervised task learning and / or supervised task learning on corpora from multiple specified domains to obtain a sample dataset. The specified domains include at least one of e-commerce, finance, news, law, computer science, electronics, chemistry, and biology.

[0051] Step S304: Determine positive and negative samples based on the sample dataset.

[0052] Step S306: The contrastive learning algorithm is used to perform contrastive learning on the positive and negative samples to determine the supervised data.

[0053] In the embodiment described above, when generating positive samples based on unsupervised task learning, the corpus is input into the model to obtain its own encoding, and then the dropout rate is adjusted, and the sample is fed into the model again to obtain its own encoding, which is itself a positive example.

[0054] In the embodiment described, when generating positive examples based on supervised task learning, if there are 4 samples A, B, C, and D in a batch, since all samples are labeled, for A, if only C matches its own label, then C is a positive example. Similarly, a positive example of C is also A.

[0055] In one exemplary embodiment of this disclosure, such as Figure 4 As shown, determining positive and negative samples based on the sample dataset includes:

[0056] Step S402: If the sample dataset was determined through the unsupervised task learning, then positive samples in the sample dataset are determined by two dropout algorithms.

[0057] Step S404: Determine the other samples in the sample dataset besides the positive samples as the negative samples.

[0058] In the above embodiment, if the data in the sample dataset is unsupervised data, when generating positive samples, existing mature methods can be used, such as two dropouts. When generating negative samples, each sample in each batch, except for itself and positive samples, is treated as a negative sample.

[0059] Dropout can be used as a trick for training deep neural networks. In each training batch, by ignoring half of the feature detectors (setting half of the hidden layer nodes to 0), overfitting can be significantly reduced. This approach reduces the interaction between feature detectors (hidden layer nodes), where some detectors depend on others to function.

[0060] In one exemplary embodiment of this disclosure, such as Figure 5 As shown, determining the positive and negative samples based on the sample dataset and performing maximum likelihood estimation of the sample distribution of the supervised data includes:

[0061] Step S502: If the sample dataset was determined through the supervised learning task, then determine multiple labels under the supervised learning task.

[0062] Step S504: Determine that multiple sample data under the same label are the positive sample and the negative sample to each other.

[0063] In the described embodiment, if the sample dataset consists of supervised data, samples with the same label can be considered positive examples when generating positive samples; otherwise, they are treated as negative examples. Specifically, a batch size of B is randomly sampled. Then it is fed into a pre-trained language model to obtain the output representation. ,in , is the average of the last layer output of BERT along the token dimension. In the case of unsupervised data, the dropout ratio needs to be adjusted, and S is fed back into the pre-trained model to obtain the output representation. ,in , 1≤i≤B. For each sample representation , This will serve as a positive example, and This will serve as a negative example, representing each sample in the context of supervised data. , As a positive example As a negative example.

[0064] In one exemplary embodiment of this disclosure, such as Figure 6 As shown, the maximum likelihood estimation of the sample distribution of the supervised data includes:

[0065] Step S602: Perform likelihood estimation on the logarithm of the supervised data to determine the sample distribution.

[0066] Step S604: Introduce conceptual variables as latent variables and maximize the result of the likelihood estimation calculation.

[0067] In the aforementioned embodiments, concepts are introduced and represented, and these concepts are used as latent variables for inference. The aim is to ensure maximum likelihood estimation of the sample distribution while identifying common conceptual knowledge among the samples. In category embedding schemes for machine recognition scenarios, estimating class-conditional probabilities is extremely difficult due to factors such as the probability density function containing all the information of a random variable, the limited sample data, and the potentially large dimension of the feature vector x. Directly estimating the density function of the class-conditional probability is challenging. However, the embodiments of this disclosure transform the estimation of the completely unknown probability density into the estimation of parameters through maximum likelihood estimation, thus converting the probability density estimation problem into a parameter estimation problem and simplifying the calculation process of class-conditional probabilities.

[0068] In one exemplary embodiment of this disclosure, such as Figure 7 As shown, maximizing the result of the likelihood estimation by introducing conceptual variables as latent variables includes:

[0069] Step S702: Introduce conceptual variables as latent variables and infer the sample distribution of the latent variables.

[0070] Step S704: Determine the cluster centers of the latent variable distribution based on the inference results.

[0071] Step S706: The cluster center is determined as the high-level capsule in the capsule network by a dynamic routing algorithm, and the other cluster samples in the latent variable distribution, excluding the cluster center, are determined as the low-level capsule in the capsule network.

[0072] Step S708: Maximize the result of the likelihood estimation calculation based on the iteration result of the capsule network.

[0073] In the aforementioned embodiment, C can be defined as a set of K cluster centroids using a clustering algorithm, thus obtaining... To ensure the stability of clustering, the number of samples to be clustered should be as large as possible. A data volume of 50 times the batch size can be selected and input into the current model. Only forward calculation is performed here, and gradient updates are not required. The clustering algorithm still selects the dynamic routing algorithm in capsule networks, treating the center point as the high-level capsule and the samples to be clustered as the low-level capsule.

[0074] In one exemplary embodiment of this disclosure, such as Figure 8 As shown, constructing a category embedding set based on the concept representation and fusing the sample representation with the category embedding set to achieve the category embedding for the classification task includes:

[0075] Step S802: Construct the category embedding set based on the concept representation.

[0076] Step S804: The sample representation set is determined as the lower-level capsule of the capsule network by the dynamic routing algorithm, and the category embedding set is determined as the higher-level capsule of the capsule network.

[0077] Step S806: The sample representation and the category embedding set are fused to achieve the category embedding of the classification task.

[0078] Combination Figure 9 and Figure 10 As shown, the first phase 1002 and the second phase 1004 of the category embedding framework 1000 defined in this disclosure are specifically described:

[0079] Step 1 S902, collect the first stage of corpus:

[0080] 1) In the first stage, unsupervised task learning is mainly used, so there are no special requirements for the training corpus. In order to achieve the goal of forming general conceptual knowledge, text data from multiple fields will be collected for learning. In order to balance training cost and generality, existing datasets in e-commerce, finance, news, law and other fields are selected as the corpus for the first stage, including tasks such as Chinese semantic matching, Chinese language reasoning, and Chinese text classification.

[0081] 2) Summarize the corpus to obtain a first-stage dataset, represented as follows: ,in It represents a single sample, and N represents the number of samples.

[0082] Step 2 S904, conduct the first phase of training:

[0083] First, the dictionary is constructed. Since the pre-trained language model BERT is used, the dictionary that comes with BERT is adopted, and its size is [size missing]. ;

[0084] Secondly, for a sample Each character is segmented using WordPiece and converted into its corresponding ID value from the dictionary, resulting in an ID sequence. , where M is The length after word segmentation To take values ​​in The sequence consists of positive integers between a certain value and a certain value, padded with zeros. The maximum length is set to T, resulting in the final sample sequence. ;

[0085] Furthermore, in order to construct the contrastive learning task, positive and negative pairs need to be generated. If the data is unsupervised, existing mature methods can be used to generate positive examples, such as two dropouts. To generate negative examples, all samples in each batch, except for themselves and positive examples, are treated as negative examples. If the data consists of supervised data, to generate positive examples, samples under the same label can be considered as positive examples of each other, otherwise, they are treated as negative examples.

[0086] Specifically, a batch of samples with a size of B is randomly sampled. Then it is fed into a pre-trained language model to obtain the output representation. ,in , is the average of the last layer output of BERT along the token dimension. In the case of unsupervised data, the dropout ratio needs to be adjusted, and S is fed back into the pre-trained model to obtain the output representation. ,in For each sample representation , This will serve as a positive example, and This will serve as a negative example; in the case of supervised data, each sample will be represented as a negative example. , As a positive example As a negative example.

[0087] Then, using the positive and negative examples, comparative learning of representations is performed. The loss function under unsupervised data is... The loss function under supervised data is :

[0088] , expression (1)

[0089] , expression (2)

[0090] Furthermore, a concept formation task is constructed, which involves introducing and representing concepts. Concepts are used as latent variables for inference, with the aim of ensuring maximum likelihood estimation of the sample distribution while identifying common conceptual knowledge among the samples.

[0091] Furthermore, using Let represent the sample random variable, whose sample representation is as in the previous step. That is, observation data, using Let the concept random variable, i.e. the latent variable, be represented by the log-likelihood estimate of the sample as shown in expression (3):

[0092] , expression (3)

[0093] After introducing the concept of implicit variables, as shown in expression (4) and expression (5),

[0094]

[0095]

[0096] because Since the information is unknown, an analytical form cannot be obtained; therefore, a distribution is introduced. The above expression is rewritten as shown in expression (6):

[0097]

[0098]

[0099] The inequality is derived from Jensen's inequality. To ensure equality holds, the following condition must be met: This holds true for all different c, that is, U is a constant. The integral of both sides with respect to c is shown in expression (7):

[0100]

[0101] Therefore, we get Therefore, it is determined .

[0102] The derivation shows that by introducing conceptual variables as latent variables, the log-likelihood is maximized by continuously maximizing a lower bound.

[0103] In one embodiment of this disclosure, the process is accomplished using EM (Expectation Maximization Algorithm), which begins with the inference of the distribution of latent variables, i.e. Since we want C to represent a general concept, we use a clustering algorithm to define C as a set of K cluster centers, that is... To ensure clustering stability, the number of samples to be clustered should be as large as possible, with a batch size of 50 times the normal size. This data should be input into the current model, and only forward calculations are performed here; gradient updates are not required.

[0104] In one embodiment of this disclosure, the clustering algorithm still selects the dynamic routing algorithm in capsule networks, treating the center point as a high-level capsule and the samples to be clustered as low-level capsules. After r iterations, the low-level capsules in the r-th iteration can be considered as being on top of the high-level capsules. Normalized weights as Therefore, the lower bound of the log-likelihood function that needs optimization, after ignoring the constant term, is shown in expression (8):

[0105]

[0106]

[0107] Next, proceed to step M. We can assume prior knowledge. The problem is that the distribution is uniformly distributed U(0,K). Once its representation is obtained, the lower bound of the log-likelihood can be maximized using gradient descent. At this point, we can assume that... It follows a Gaussian distribution, with mean c and variance . As shown in expression (9):

[0108]

[0109] in It is the normalization factor, which can be expressed as After normalizing z and c, the complete lower bound is shown in expression (10):

[0110]

[0111]

[0112] in, This parameter describes the density of the current class centroids, and is set to an empirical value here. Therefore, the final loss function for the concept formation task is shown in expression (11):

[0113]

[0114] Step 3, S906: Collect the second-stage corpus:

[0115] The second stage, or downstream task stage, involves training data that serves as supervision data for the corresponding task. This is represented as... ,in It is text. For the corresponding label, N represents the number of samples, and G represents the number of categories.

[0116] Step 4, S908, conduct the second phase of training:

[0117] The dictionary is constructed using the pre-trained language model BERT, so it uses the dictionary that comes with BERT, and its size is [size missing]. ;

[0118] For a sample Each character is segmented using WordPiece and converted into its corresponding ID value from the dictionary, resulting in an ID sequence. , where M is The length after word segmentation To take values ​​in The sequence consists of positive integers between a certain value and a certain value, padded with zeros. The maximum length is set to T, resulting in the final sample sequence.

[0119] The samples are fed into the model to obtain a representation of the samples. .

[0120] Construct a set of category embeddings using the concept representations obtained from one-stage training. The first stage yields a set of conceptual representations. Treating the set as a low-level capsule and the category embeddings as high-level capsules, the category embeddings are still obtained through a dynamic routing algorithm. .

[0121] The sample representations and category embeddings are fused. Here, a simple fusion method, namely the cross-attention mechanism, is chosen to obtain the final fused representation. as follows:

[0122] ;

[0123] ;

[0124] ;

[0125] .

[0126] The classifier chosen is a linear classifier, and its weight matrix is... So, what is the final probability? As shown in expression (12):

[0127]

[0128] Cross-entropy is used as the loss function as shown in expression (13):

[0129]

[0130] Here, CE represents the cross-entropy function, y is the true label, and gradient descent training is performed using an Adam-based optimizer.

[0131] Corresponding to the method embodiments described above, this disclosure also provides a category embedding device for machine recognition scenarios, which can be used to execute the method embodiments described above.

[0132] Figure 11 This is a block diagram of a category embedding device in a machine recognition scenario according to an exemplary embodiment of this disclosure.

[0133] refer to Figure 11The category embedding device 1100 in the machine recognition scenario may include:

[0134] The learning module 1102 is configured to learn from the corpus in response to a received classification task to determine supervised data, which has conceptual representations.

[0135] The learning module 1102 is further configured to perform maximum likelihood estimation on the sample distribution of the supervised data.

[0136] The determination module 1104 is configured to determine the lower bound of the maximum likelihood estimate based on the dynamic routing algorithm in the capsule network, and determine the loss function of the classification model based on the lower bound of the maximum likelihood estimate.

[0137] The learning module 1102 is further configured to input the supervised data into the classification model to obtain sample representations.

[0138] Embedding module 1106 is configured to construct a category embedding set based on the concept representation and fuse the sample representation with the category embedding set to achieve category embedding for the classification task.

[0139] In one exemplary embodiment of this disclosure, the learning module 1102 is further configured to: perform unsupervised task learning and / or supervised task learning on corpora of multiple specified domains to obtain a sample dataset, wherein the specified domains include at least one of e-commerce, finance, news, law, computer, electronics, chemistry, and biology; determine positive samples and negative samples based on the sample dataset; and use the contrastive learning algorithm to perform contrastive learning on the positive samples and the negative samples to determine the supervised data.

[0140] In an exemplary embodiment of this disclosure, the learning module 1102 is further configured to: if the sample dataset is determined through the unsupervised task learning, then determine the positive samples in the sample dataset through two dropout algorithms; and determine the other samples in the sample dataset besides the positive samples as the negative samples.

[0141] In an exemplary embodiment of this disclosure, the learning module 1102 is further configured to: if the sample dataset is determined through the supervised learning task, then determine multiple labels under the supervised learning task; and determine multiple sample data under the same label as the positive sample and the negative sample.

[0142] In an exemplary embodiment of this disclosure, the category embedding device 1100 in the machine recognition scenario is further configured to: perform likelihood estimation calculation on the logarithm of the supervised data; introduce conceptual variables as latent variables, and maximize the result of the likelihood estimation calculation.

[0143] In an exemplary embodiment of this disclosure, the category embedding device 1100 in a machine recognition scenario is further configured to: introduce conceptual variables as latent variables and infer the sample distribution of the latent variables; determine the cluster centers of the latent variable distribution based on the inference results; determine the cluster centers as high-level capsules in the capsule network using a dynamic routing algorithm, and determine the other clustered samples in the latent variable distribution besides the cluster centers as low-level capsules in the capsule network; and maximize the result of the likelihood estimation calculation based on the iteration results of the capsule network.

[0144] In an exemplary embodiment of this disclosure, the embedding module 1106 is further configured to: construct a category embedding set based on the concept representation, and fuse the sample representation with the category embedding set to achieve category embedding of the classification task, including: constructing the category embedding set based on the concept representation; determining the sample representation set as a low-level capsule of the capsule network and the category embedding set as a high-level capsule of the capsule network through the dynamic routing algorithm; and fusing the sample representation and the category embedding set to achieve category embedding of the classification task.

[0145] Since the functions of the category embedding device 1100 in the machine recognition scenario have been described in detail in their corresponding method embodiments, they will not be repeated here.

[0146] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to embodiments of this disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.

[0147] In an exemplary embodiment of this disclosure, an electronic device capable of implementing the method is also provided.

[0148] Those skilled in the art will understand that various aspects of this disclosure can be implemented as a system, method, or program product. Therefore, various aspects of this disclosure can be specifically implemented in the following forms: a completely hardware implementation, a completely software implementation (including firmware, microcode, etc.), or a combination of hardware and software aspects, collectively referred to herein as a "circuit," "module," or "system."

[0149] The following reference Figure 12 To describe an electronic device 1200 according to such an embodiment of the present disclosure. Figure 12The electronic device 1200 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments disclosed herein.

[0150] like Figure 12 As shown, the electronic device 1200 is presented in the form of a general-purpose computing device. The components of the electronic device 1200 may include, but are not limited to: the at least one processing unit 1210, the at least one storage unit 1220, and a bus 1230 connecting different system components (including the storage unit 1220 and the processing unit 1210).

[0151] The storage unit stores program code that can be executed by the processing unit 1210, causing the processing unit 1210 to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure. For example, the processing unit 1210 can perform the method shown in the embodiments of this disclosure.

[0152] Storage unit 1220 may include readable media in the form of volatile storage units, such as random access memory (RAM) 12201 and / or cache memory 12202, and may further include read-only memory (ROM) 12203.

[0153] Storage unit 1220 may also include a program / utility 12204 having a set (at least one) of program modules 12205, such program modules 12205 including but not limited to: operating system, one or more application programs, other program modules and program data, each or some combination of these examples may include an implementation of a network environment.

[0154] Bus 1230 can represent one or more of several types of bus structures, including a memory cell bus or memory cell controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of the various bus structures.

[0155] Electronic device 1200 can also communicate with one or more external devices 1240 (e.g., keyboard, pointing device, Bluetooth device, etc.), one or more devices that enable a user to interact with electronic device 1200, and / or any device that enables electronic device 1200 to communicate with one or more other computing devices (e.g., router, modem, etc.). This communication can be performed via input / output (I / O) interface 1250. Furthermore, electronic device 1200 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 1260. As shown, network adapter 1260 communicates with other modules of electronic device 1200 via bus 1230. It should be understood that, although not shown in the figures, other hardware and / or software modules can be used in conjunction with electronic device 1200, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0156] From the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, terminal device, or network device, etc.) to execute the methods according to the embodiments of this disclosure.

[0157] In exemplary embodiments of this disclosure, a computer-readable storage medium is also provided, having stored thereon a program product capable of implementing the methods described herein. In some possible implementations, various aspects of this disclosure may also be implemented as a program product including program code that, when run on a terminal device, causes the terminal device to perform the steps described in the "Exemplary Methods" section of this specification according to various exemplary embodiments of this disclosure.

[0158] The program product for implementing the method according to embodiments of this disclosure may employ a portable compact disc read-only memory (CD-ROM) and include program code, and may run on a terminal device, such as a personal computer. However, the program product of this disclosure is not limited thereto. In this document, the readable storage medium may be any tangible medium that contains or stores a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0159] The program product may employ any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0160] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A readable signal medium may also be any readable medium other than a readable storage medium, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device.

[0161] The program code contained on the readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, fiber optic, RF, etc., or any suitable combination thereof.

[0162] Program code for performing the operations of this disclosure can be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java and C++, and conventional procedural programming languages ​​such as C or similar languages ​​for maximum likelihood estimation of the sample distribution of the supervised data. The program code can execute entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0163] Furthermore, the accompanying drawings are merely illustrative of the processes included in the method according to exemplary embodiments of this disclosure and are not intended to be limiting. It is readily understood that the processes shown in the drawings do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.

[0164] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and concept of this disclosure are indicated by the claims.

Claims

1. A category embedding method for machine recognition scenarios, characterized in that, include: In response to the received classification task, the corpus is learned to determine supervision data, which has conceptual representations; Performing maximum likelihood estimation on the sample distribution of the supervised data includes: Likelihood estimation is performed on the logarithm of the supervised data to determine the sample distribution; The conceptual variables are introduced as latent variables, and the sample distribution of the latent variables is inferred. The cluster centers of the latent variable distribution are determined based on the inference results; The cluster centers are determined as high-level capsules in the capsule network using a dynamic routing algorithm, and the other cluster samples in the latent variable distribution, excluding the cluster centers, are determined as low-level capsules in the capsule network. The result of the likelihood estimation is maximized based on the iterative results of the capsule network. The lower bound of the maximum likelihood estimate is determined based on the dynamic routing algorithm in the capsule network, and the loss function of the classification model is determined based on the lower bound of the maximum likelihood estimate. The supervised data is input into the classification model to obtain sample representations; A category embedding set is constructed based on the concept representation, and the sample representation is fused with the category embedding set to achieve the category embedding of the classification task.

2. The category embedding method in a machine recognition scenario as described in claim 1, characterized in that, In response to the received classification task, the corpus is learned to determine the supervised data, including: Unsupervised and / or supervised learning tasks are performed on corpora from multiple specified domains to obtain sample datasets. The specified domains include at least one of e-commerce, finance, news, law, computer science, electronics, chemistry, and biology. Based on the sample dataset, determine the positive and negative samples; A contrastive learning algorithm is used to compare and learn the positive and negative examples to determine the supervised data.

3. The category embedding method in a machine recognition scenario as described in claim 2, characterized in that, Determining positive and negative samples based on the aforementioned sample dataset includes: If the sample dataset was determined through the unsupervised task, then positive samples in the sample dataset are determined by two dropout algorithms. The samples in the sample dataset other than the positive samples are identified as the negative samples.

4. The category embedding method in a machine recognition scenario as described in claim 2, characterized in that, Determining positive and negative samples based on the aforementioned sample dataset includes: If the sample dataset was determined through the supervised learning task, then multiple labels under the supervised learning task are determined. Multiple sample data under the same label are identified as positive samples and negative samples.

5. The category embedding method in machine recognition scenarios as described in any one of claims 1-4, characterized in that, The category embedding set is constructed based on the concept representation, and the sample representation is fused with the category embedding set to achieve the category embedding of the classification task, including: The category embedding set is constructed based on the conceptual representation; the sample representation set is determined as the lower-level capsule of the capsule network by the dynamic routing algorithm, and the category embedding set is determined as the higher-level capsule of the capsule network. The sample representations and the category embedding set are fused to achieve the category embedding for the classification task.

6. A category embedding device for machine recognition scenarios, characterized in that, include: The learning module is configured to learn from the corpus in response to a received classification task to determine supervisory data, which has conceptual representations. The learning module is further configured to perform maximum likelihood estimation on the sample distribution of the supervised data, including: Likelihood estimation is performed on the logarithm of the supervised data to determine the sample distribution; The conceptual variables are introduced as latent variables, and the sample distribution of the latent variables is inferred. The cluster centers of the latent variable distribution are determined based on the inference results; The cluster centers are determined as high-level capsules in the capsule network using a dynamic routing algorithm, and the other cluster samples in the latent variable distribution, excluding the cluster centers, are determined as low-level capsules in the capsule network. The result of the likelihood estimation is maximized based on the iterative results of the capsule network. The determination module is configured to determine the lower bound of the maximum likelihood estimate based on the dynamic routing algorithm in the capsule network, and determine the loss function of the classification model based on the lower bound of the maximum likelihood estimate; The learning module is further configured to input the supervised data into the classification model to obtain sample representations; The embedding module is configured to construct a category embedding set based on the concept representation and fuse the sample representation with the category embedding set to achieve category embedding for the classification task.

7. An electronic device, characterized in that, include: Memory; as well as A processor coupled to the memory, the processor being configured to execute the category embedding method for machine recognition scenarios as described in any one of claims 1-5 based on instructions stored in the memory.

8. A computer-readable storage medium having a program stored thereon that, when executed by a processor, implements the category embedding method for machine recognition scenarios as described in any one of claims 1-5.

Images