Method and device for recognizing text

Abstract

This invention discloses a method and apparatus for text recognition, relating to the field of deep learning technology. One specific embodiment of the method includes: inputting the text to be recognized into a pre-trained encoder to output a general semantic vector of the text; inputting the general semantic vector of the text to be recognized into a trained feature projection layer to output a recognition result of the text; wherein, the training process of the feature projection layer employs regularization terms to constrain each basis vector of the feature subspace, ensuring that the basis vectors of the feature subspace are mutually perpendicular. This embodiment can solve the technical problem of poor interpretability of feature vectors.

Description

Technical Field

[0001] This invention relates to the field of deep learning technology, and more particularly to a method and apparatus for recognizing text. Background Technology

[0002] A semantic encoder is a tool that encodes text, mapping the input text into vectors in a semantic space. Once we have these semantic vectors, we can use them to train models for subsequent business-related tasks, such as semantic similarity calculation, topic detection, and sentiment analysis.

[0003] Current semantic encoders are typically obtained through pre-trained language models, with the BERT (Bidirectional Encoder Representations from Transformers) family of language models being a prime example. These models have demonstrated good performance after fine-tuning across various task scenarios, making the pre-trained model + fine-tuning training model a typical solution to related problems.

[0004] The fine-tuning process involves how to project the general semantic vector output by the general semantic encoder into a task-specific space. Existing solutions use a linear transformation to achieve the projection.

[0005] In the process of realizing this invention, the inventors discovered at least the following problems in the prior art:

[0006] Existing technical solutions primarily focus on obtaining good semantic vectors, without imposing restrictions on the feature projection process. This leads to a lack of constraints on the basis vectors of the semantic space themselves, resulting in non-orthogonality of the basis vectors. Consequently, confusion arises between the different dimensions of the semantic vectors, leading to poor interpretability of the feature vectors, increasing the error in calculating semantic similarity, and thus increasing the difficulty of subsequent learning tasks. Summary of the Invention

[0007] In view of this, embodiments of the present invention provide a method and apparatus for recognizing text to solve the technical problem of poor interpretability of feature vectors.

[0008] To achieve the above objectives, according to one aspect of the present invention, a method for recognizing text is provided, comprising:

[0009] The text to be recognized is input into a pre-trained encoder to output a general semantic vector of the text to be recognized;

[0010] The general semantic vector of the text to be identified is input into a trained feature projection layer to output the recognition result of the text to be identified.

[0011] The training process of the feature projection layer employs regularization terms to constrain each basis vector of the feature subspace, ensuring that the basis vectors of the feature subspace are mutually perpendicular.

[0012] Optionally, before inputting the text to be recognized into a pre-trained encoder to output a general semantic vector of the text to be recognized, the method further includes:

[0013] The sample text is input into a pre-trained encoder to output a general semantic vector of the sample text;

[0014] The general semantic vector of the sample text and its corresponding label are input into the neural network. At the same time, regularization terms are used to constrain each basis vector of the feature subspace so that the basis vectors of the feature subspace are perpendicular to each other. After iterative training, the neural network obtains the feature projection layer.

[0015] Optionally, a regularization term is used to constrain each basis vector of the feature subspace so that the basis vectors of the feature subspace are mutually perpendicular, including:

[0016] Initialize the parameter matrix; wherein, the parameter matrix is ​​the parameter of the feature projection layer;

[0017] The difference between the parameter matrix and the identity matrix is ​​calculated to obtain the regularization term;

[0018] The parameters of the feature projection layer are optimized with the goal of making the regularization term close to zero.

[0019] Optionally, the difference between the parameter matrix and the identity matrix is ​​calculated to obtain the regularization term, including:

[0020] The parameter matrix is ​​normalized to obtain the weight matrix;

[0021] Multiplying the transpose of the weight matrix with the weight matrix yields a dot product matrix; wherein the dot product matrix represents the dot product between the basis vectors of the feature subspace.

[0022] The difference between the dot product matrix and the identity matrix is ​​calculated to obtain the regularization term.

[0023] Optionally, the parameter matrix is ​​normalized to obtain a weight matrix, including:

[0024] For each column vector in the parameter matrix, calculate the magnitude of the column vector;

[0025] For each element in the column vector, divide the element by the magnitude to obtain a normalized column vector, thereby obtaining the weight matrix.

[0026] Alternatively, the difference between the dot product matrix and the identity matrix can be calculated using the following method:

[0027] Calculate the mean square between the dot product matrix and the identity matrix;

[0028] or,

[0029] The difference between the dot product matrix and the identity matrix is ​​calculated based on the loss function.

[0030] Optionally, calculating the difference between the dot product matrix and the identity matrix based on the loss function includes:

[0031] The difference between the dot product matrix and the identity matrix is ​​calculated using the following formula:

[0032] mean[I-softmax(a*A)*I]

[0033] Where a is a hyperparameter controlling the strength of the regularization term, I is the identity matrix, and A is the dot product matrix.

[0034] Additionally, according to another aspect of the present invention, a text recognition device is provided, comprising:

[0035] The encoding module is used to input the text to be recognized into a pre-trained encoder to output a general semantic vector of the text to be recognized;

[0036] The recognition module is used to input the general semantic vector of the text to be recognized into the trained feature projection layer to output the recognition result of the text to be recognized.

[0037] The training process of the feature projection layer employs regularization terms to constrain each basis vector of the feature subspace, ensuring that the basis vectors of the feature subspace are mutually perpendicular.

[0038] It also includes a training module for:

[0039] Before inputting the text to be recognized into the pre-trained encoder to output the general semantic vector of the text to be recognized, the sample text is input into the pre-trained encoder to output the general semantic vector of the sample text.

[0040] The general semantic vector of the sample text and its corresponding label are input into the neural network. At the same time, regularization terms are used to constrain each basis vector of the feature subspace so that the basis vectors of the feature subspace are perpendicular to each other. After iterative training, the neural network obtains the feature projection layer.

[0041] Optionally, the training module is further configured to:

[0042] Initialize the parameter matrix; wherein, the parameter matrix is ​​the parameter of the feature projection layer;

[0043] The difference between the parameter matrix and the identity matrix is ​​calculated to obtain the regularization term;

[0044] The parameters of the feature projection layer are optimized with the goal of making the regularization term close to zero.

[0045] Optionally, the training module is further configured to:

[0046] The parameter matrix is ​​normalized to obtain the weight matrix;

[0047] Multiplying the transpose of the weight matrix with the weight matrix yields a dot product matrix; wherein the dot product matrix represents the dot product between the basis vectors of the feature subspace.

[0048] The difference between the dot product matrix and the identity matrix is ​​calculated to obtain the regularization term.

[0049] Optionally, the training module is further configured to:

[0050] For each column vector in the parameter matrix, calculate the magnitude of the column vector;

[0051] For each element in the column vector, divide the element by the magnitude to obtain a normalized column vector, thereby obtaining the weight matrix.

[0052] Optionally, the training module is further configured to:

[0053] Calculate the mean square between the dot product matrix and the identity matrix;

[0054] or,

[0055] The difference between the dot product matrix and the identity matrix is ​​calculated based on the loss function.

[0056] Optionally, the training module is further configured to:

[0057] The difference between the dot product matrix and the identity matrix is ​​calculated using the following formula:

[0058] mean[I-softmax(a*A)*I]

[0059] Where a is a hyperparameter controlling the strength of the regularization term, I is the identity matrix, and A is the dot product matrix.

[0060] According to another aspect of the present invention, an electronic device is also provided, comprising:

[0061] One or more processors;

[0062] Storage device for storing one or more programs.

[0063] When the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any of the above embodiments.

[0064] According to another aspect of the present invention, a computer-readable medium is also provided, on which a computer program is stored, which, when executed by a processor, implements the methods described in any of the above embodiments.

[0065] One embodiment of the above invention has the following advantages or beneficial effects: because it employs a technique of using regularization terms to constrain the basis vectors of the feature subspace during the training of the feature projection layer, ensuring that the basis vectors of the feature subspace are mutually perpendicular, it overcomes the technical problem of poor interpretability of feature vectors in the prior art. By adding orthogonal basis vector constraints to the feature projection layer, this embodiment of the invention makes the meaning of the feature vectors in the feature subspace clearer, enhances interpretability, reduces the error in calculating semantic similarity, and thus reduces the difficulty of subsequent learning tasks.

[0066] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0067] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein:

[0068] Figure 1 This is a schematic diagram of the main flow of a text recognition method according to an embodiment of the present invention;

[0069] Figure 2 This is a schematic diagram of a pre-trained encoder and a feature projection layer according to an embodiment of the present invention;

[0070] Figure 3 This is a schematic diagram of the main flow of a method for recognizing text according to a possible embodiment of the present invention;

[0071] Figure 4 This is a schematic diagram of the main process of constraining the basis vectors of the feature subspace with regularization terms according to a possible embodiment of the present invention;

[0072] Figure 5 This is a schematic diagram of the main modules of a text recognition device according to an embodiment of the present invention;

[0073] Figure 6 This is an exemplary system architecture diagram in which embodiments of the present invention can be applied;

[0074] Figure 7 This is a schematic diagram of the structure of a computer system suitable for implementing terminal devices or servers of the present invention. Detailed Implementation

[0075] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0076] The core of existing technical solutions focuses on how to obtain a good general semantic vector, and how this semantic vector can exhibit excellent performance in subsequent fine-tuning tasks. However, there is an implicit concept that has not been addressed: the basis vectors of the semantic space. If the basis vectors of the semantic space are mutually orthogonal, then the various dimensions of the semantic vector are mutually independent. However, if the basis vectors are non-orthogonal, then there is confusion between the various dimensions of the semantic vector. This leads to poor interpretability of the feature vector, increases the error in calculating semantic similarity, and thus increases the difficulty of subsequent learning tasks.

[0077] Existing technical solutions primarily focus on obtaining good semantic vectors, without imposing restrictions on the feature projection process. This leads to a lack of constraints on the semantic space basis vectors themselves, while subsequent tasks often require features to be as decoupled as possible. For example, in topic classification, each topic should be independent; in sentiment classification, positive and negative sentiments should be mutually exclusive. The most typical drawback is that we often want to use sentence vectors to calculate sentence similarity, whether using Euclidean distance or cosine distance. These methods rely on the assumption that the dimensions of the vectors are independent, which is not actually the case.

[0078] Fine-tuning tasks typically involve two modules: a pre-trained model module and a fine-tuning module. The general semantic vectors extracted by the pre-trained model possess strong generalization performance, but without fine-tuning in a specific domain, they are unlikely to achieve good learning results. The fine-tuning module projects these general semantic vectors onto a specific domain, forming domain-specific feature vectors, thus reducing the difficulty of subsequent learning tasks. These domain-specific feature vectors indicate which features are of greater interest in certain domains.

[0079] Furthermore, sentence vectors are often used to calculate similarity for clustering or matching. In such cases, which semantic dimensions we focus on becomes crucial. For example: s1 - I like to eat apples, s2 - I don't like to eat apples, s3 - I like to eat bananas. From a topic perspective:

[0080] sim topic(s1, s2) > sim topic (s1, s3),

[0081] However, from an emotional perspective:

[0082] sim sentiment (s1, s2) < sim sentiment (s1, s3).

[0083] This example illustrates that semantic similarity is ambiguous when separated from a specific task. Only by projecting general semantic vectors onto the space we are interested in can we obtain a clearly defined semantic similarity. This is also why we need to perform feature projection, as mentioned earlier, regarding which features are of greater interest to us.

[0084] Assuming a pre-trained model provides an n-dimensional general semantic vector s, existing neural network-based feature projection methods simply append a linear layer to s:

[0085] s p =Ws,

[0086] Where W is an m*n dimensional matrix, the function of which can be understood as: selecting m feature directions in the n-dimensional general feature space as basis vectors, and projecting s onto the feature subspace with these m basis vectors as basis vectors, to obtain s p Generally speaking, after learning, this feature subspace will contain the features that the task is interested in.

[0087] However, existing schemes impose almost no restrictions on W, meaning there are no constraints between the m eigenvectors, and they may even be parallel, which would lead to s p Some feature dimensions are related, which leads to the repeated calculation of the contribution of certain feature dimensions when calculating similarity.

[0088] For example, in a 2-dimensional eigenspace, s p1 =[a1, b1], s p2 = [a2, b2], assuming the basis vectors of the eigenspace are e1 and e2, then in the eigenspace s p1 s p2 The actual form is:

[0089] s p1 = a1*e1 + b1*e2,

[0090] s p2 = a2*e1+b2*e2.

[0091] Based on this, calculate s p1 s p2 Cosine similarity:

[0092]

[0093] We can obtain:

[0094]

[0095] It is evident that the above calculation formula is more accurate than commonly understood. It's much more complicated.

[0096] This implicitly assumes that e1·e2 = 0, meaning e1 and e2 are perpendicular. In other words, when e1 and e2 are not perpendicular, the similarity calculated using the above formula will contain errors. These errors stem from repeated calculations of features in certain dimensions (the parallel components of e1 and e2). Therefore, to reduce the error in calculating semantic similarity, the parallel components of e1 and e2 should be as small as possible.

[0097] To address the technical problems existing in the prior art, the embodiments of the present invention constrain each basis vector of the feature subspace during the model training process, so that each basis vector of the feature subspace is perpendicular to each other, thereby achieving decoupling and compression between the sub-dimensions of the feature vectors, thus reducing the difficulty of subsequent learning tasks.

[0098] Figure 1 This is a schematic diagram illustrating the main flow of a text recognition method according to an embodiment of the present invention. As one embodiment of the present invention, such as... Figure 1 As shown, the method for recognizing text may include:

[0099] Step 101: Input the text to be recognized into a pre-trained encoder to output a general semantic vector of the text to be recognized.

[0100] First, such as Figure 2 As shown, the text to be recognized is input into a pre-trained encoder, which can be a language model of the BERT (Bidirectional Encoder Representations from Transformers) family. The pre-trained encoder outputs a general semantic vector of the text to be recognized.

[0101] Step 102: Input the general semantic vector of the text to be identified into the trained feature projection layer to output the recognition result of the text to be identified.

[0102] The general semantic vectors extracted by the pre-trained encoder have strong generalization performance and need to be further recognized through a feature projection layer. Specifically, such as... Figure 2As shown, the general semantic vector output by the pre-trained encoder is input into the trained feature projection layer, and the feature projection layer outputs the recognition result of the text to be recognized. It should be noted that the recognition result is related to the recognition task, and the recognition result can be similarity calculation result, topic detection result, sentiment analysis result, etc.

[0103] It should be noted that the feature projection layer needs to be trained in advance, and during the training of the feature projection layer, regularization terms are used to constrain each basis vector of the feature subspace so that the basis vectors of the feature subspace are mutually perpendicular.

[0104] Optionally, before step 101, the method further includes: inputting sample text into a pre-trained encoder to output a general semantic vector of the sample text; inputting the general semantic vector of the sample text and its corresponding label into a neural network, while using regularization terms to constrain each basis vector of the feature subspace so that the basis vectors of the feature subspace are mutually perpendicular; and obtaining a feature projection layer after iterative training of the neural network. Before step 101, the feature projection layer needs to be trained first. Specifically, the sample text is input into the pre-trained encoder, which outputs the general semantic vector corresponding to each sample text. Then, the general semantic vector corresponding to each sample text and its corresponding label are input into the neural network for training. During the training of the neural network, regularization terms are used to constrain each basis vector of the feature subspace so that the basis vectors of the feature subspace are mutually perpendicular. The feature projection layer is obtained after iterative training of the neural network.

[0105] To achieve the requirement that the basis vectors in the feature subspace are perpendicular to each other, this embodiment of the invention introduces a regularization method in the projection layer. This regularization method adds a constraint to the projection layer, which requires that the basis vectors be as perpendicular as possible.

[0106] Optionally, a regularization term is used to constrain each basis vector of the feature subspace to make the basis vectors of the feature subspace mutually perpendicular, including: initializing a parameter matrix; wherein the parameter matrix is ​​the parameter of the feature projection layer; calculating the difference between the parameter matrix and the identity matrix to obtain the regularization term; and optimizing the parameter of the feature projection layer with the goal of making the regularization term close to zero. Specifically, a parameter matrix W = [e1, e2, ..., e] of m n-dimensional vectors is initialized. mThe parameter matrix is ​​used as the parameter to be optimized in the feature projection layer. Then, the difference between the parameter matrix and the identity matrix is ​​calculated, and this difference is used as a regularization term. Finally, during the optimization of each parameter in the feature projection layer, supervised iterative training is performed with the regularization term approaching zero, combined with the model's training objective. In the embodiments of this invention, the neural network acts as the projection layer, projecting the general semantic vector onto a feature subspace with m basis vectors. The basis vectors in the feature subspace are approximately perpendicular to each other, thus achieving decoupling and compression between the sub-dimensions of the feature vectors, thereby reducing the difficulty of subsequent learning tasks.

[0107] Optionally, calculating the difference between the parameter matrix and the identity matrix to obtain a regularization term includes: normalizing the parameter matrix to obtain a weight matrix; multiplying the transpose of the weight matrix with the weight matrix to obtain a dot product matrix; wherein the dot product matrix represents the dot product between the basis vectors of the feature subspace; and calculating the difference between the dot product matrix and the identity matrix to obtain a regularization term. In this embodiment of the invention, the parameter matrix is ​​normalized by the modulus to obtain a normalized weight matrix W′=[e′1,e′2,...,e′ m Then, the transpose of the weight matrix W′ is... T Multiplying this by the weight matrix W′ yields the dot product matrix, i.e., W′ T W′=A, where A (m*m dimensional matrix) can be understood as the dot product between each basis vector in the eigenspace. Finally, the difference between the dot product matrix A and the identity matrix I is calculated, and this difference is the regularization term.

[0108] Optionally, the parameter matrix is ​​normalized to obtain a weight matrix, including: for each column vector in the parameter matrix, calculating the magnitude of the column vector; for each element in the column vector, dividing the element by the magnitude to obtain a normalized column vector, thereby obtaining the weight matrix. Specifically, e is first calculated. i The length of the module is determined, and then e is used with that length. i Normalization is performed to obtain the normalized weight matrix W′=[e′1,e′2,...,e′ m Normalizing the weight matrix first ensures that the difference between the dot product matrix and the identity matrix can be accurately calculated, that is, the regularization term can be accurately calculated.

[0109] Alternatively, the difference between the dot product matrix and the identity matrix can be calculated using the following method:

[0110] Calculate the mean square between the dot product matrix and the identity matrix; or, calculate the difference between the dot product matrix and the identity matrix based on the loss function.

[0111] There are several ways to calculate the difference between the dot product matrix and the identity matrix. For example, the mean square can be used to calculate the difference between the dot product matrix and the identity matrix, i.e., mean[(AI)]. 2 Alternatively, a loss function can be used to calculate the difference between the dot product matrix and the identity matrix. In each iteration, the regularization term gradually approaches zero.

[0112] Optionally, calculating the difference between the dot product matrix and the identity matrix based on the loss function includes: calculating the difference between the dot product matrix and the identity matrix using the following formula:

[0113] mean[I-softmax(a*A)*I]

[0114] Where a is a hyperparameter controlling the strength of the regularization term, I is the identity matrix, and A is the dot product matrix.

[0115] In each iteration, a regularization term can be added to the loss function to optimize the parameter matrix.

[0116] After multiple iterations, the optimized parameter matrix W is obtained. Applying the optimized parameter matrix W to the general semantic vector can realize the feature projection from n dimensions to m dimensions, and the feature dimensions are as perpendicular as possible to each other.

[0117] The embodiments of the present invention constrain each basis vector of the feature subspace so that they are mutually perpendicular, thereby achieving independence between feature dimensions and giving semantic vectors a clearer meaning and interpretability.

[0118] Based on the various embodiments described above, it can be seen that the embodiments of the present invention solve the technical problem of poor interpretability of feature vectors in the prior art by using regularization terms to constrain each basis vector of the feature subspace during the training of the feature projection layer, so that the basis vectors of the feature subspace are mutually perpendicular. By adding orthogonal basis vector constraints to the feature projection layer, the embodiments of the present invention make the meaning of the feature vectors in the feature subspace clearer, enhance interpretability, reduce the error in calculating semantic similarity, and thus reduce the difficulty of subsequent learning tasks.

[0119] Figure 3 This is a schematic diagram of the main flow of a text recognition method according to a possible embodiment of the present invention. As another embodiment of the present invention, such as... Figure 3 As shown, the method for recognizing text may include:

[0120] Step 301: Input the sample text into the pre-trained encoder to output the general semantic vector of the sample text.

[0121] Step 302: Input the general semantic vector of the sample text and its corresponding label into the neural network. At the same time, use regularization terms to constrain each basis vector of the feature subspace so that the basis vectors of the feature subspace are perpendicular to each other. After iterative training, the neural network obtains the feature projection layer.

[0122] The feature projection layer needs to be trained in advance. During the training process, a regularization term is used to constrain each basis vector in the feature subspace to ensure that the basis vectors in the feature subspace are mutually perpendicular. This embodiment of the invention uses a neural network to optimize the parameters of the feature projection layer, and obtains the feature projection layer through iterative training.

[0123] Step 303: Input the text to be recognized into the pre-trained encoder to output a general semantic vector of the text to be recognized.

[0124] After training the feature projection layer, the task-related text to be recognized is input into the pre-trained encoder. The pre-trained encoder can be a language model from the BERT family. The pre-trained encoder outputs a general semantic vector of the text to be recognized.

[0125] Step 304: Input the general semantic vector of the text to be identified into the trained feature projection layer to output the recognition result of the text to be identified.

[0126] The general semantic vectors extracted by the pre-trained encoder have strong generalization performance and need to be further recognized by a feature projection layer. The feature projection layer projects features onto the general semantic vectors and then outputs the recognition result of the text to be recognized.

[0127] In product customer reviews, information often involves various dimensions. We aim to extract valuable information to help improve our services. For example, a review might be about the product's color, casing texture, sound effects, or service-related attributes. To determine the attributes of a review, a logical design task based on an embodiment of the present invention can be used to classify the attributes. Furthermore, similarity calculations can be performed on reviews based on sentence vectors in the feature subspace to aid in classification.

[0128] After using the optimized feature projection layer of this invention, the classification performance was improved to a certain extent, as shown in the table below:

[0129]

[0130] Furthermore, the specific implementation details of the text recognition method in one of the reference embodiments of the present invention have been described in detail in the text recognition method described above, so the content will not be repeated here.

[0131] Figure 4This is a schematic diagram illustrating the main process of constraining the basis vectors of the feature subspace using regularization terms according to a possible embodiment of the present invention. As another embodiment of the present invention, such as... Figure 4 As shown, constraining the basis vectors of the feature subspace with regularization terms to make the basis vectors of the feature subspace mutually perpendicular may include the following steps:

[0132] Step 401, initialize the parameter matrix; wherein, the parameter matrix is ​​the parameter of the feature projection layer.

[0133] Step 402: For each column vector in the parameter matrix, calculate the magnitude of the column vector.

[0134] Step 403: For each element in the column vector, divide the element by the modulus to obtain the normalized column vector, thereby obtaining the weight matrix.

[0135] Step 404: Multiply the transpose of the weight matrix with the weight matrix to obtain a dot product matrix; wherein the dot product matrix represents the dot product between the basis vectors of the feature subspace.

[0136] Step 405: Calculate the difference between the dot product matrix and the identity matrix to obtain the regularization term.

[0137] Step 406: Optimize the parameters of the feature projection layer with the goal of making the regularization term close to zero.

[0138] Furthermore, the specific implementation details of the text recognition method in one of the reference embodiments of the present invention have been described in detail in the text recognition method described above, so the content will not be repeated here.

[0139] Figure 5 This is a schematic diagram of the main modules of a text recognition device according to an embodiment of the present invention. Figure 5 As shown, the text recognition device 500 includes an encoding module 501 and a recognition module 502; wherein, the encoding module 501 is used to input the text to be recognized into a pre-trained encoder to output a general semantic vector of the text to be recognized; the recognition module 502 is used to input the general semantic vector of the text to be recognized into a trained feature projection layer to output the recognition result of the text to be recognized; wherein, the training process of the feature projection layer uses a regularization term to constrain each basis vector of the feature subspace so that the basis vectors of the feature subspace are mutually perpendicular.

[0140] It also includes a training module for:

[0141] Before inputting the text to be recognized into the pre-trained encoder to output the general semantic vector of the text to be recognized, the sample text is input into the pre-trained encoder to output the general semantic vector of the sample text.

[0142] The general semantic vector of the sample text and its corresponding label are input into the neural network. At the same time, regularization terms are used to constrain each basis vector of the feature subspace so that the basis vectors of the feature subspace are perpendicular to each other. After iterative training, the neural network obtains the feature projection layer.

[0143] Optionally, the training module is further configured to:

[0144] Initialize the parameter matrix; wherein, the parameter matrix is ​​the parameter of the feature projection layer;

[0145] The difference between the parameter matrix and the identity matrix is ​​calculated to obtain the regularization term;

[0146] The parameters of the feature projection layer are optimized with the goal of making the regularization term close to zero.

[0147] Optionally, the training module is further configured to:

[0148] The parameter matrix is ​​normalized to obtain the weight matrix;

[0149] Multiplying the transpose of the weight matrix with the weight matrix yields a dot product matrix; wherein the dot product matrix represents the dot product between the basis vectors of the feature subspace.

[0150] The difference between the dot product matrix and the identity matrix is ​​calculated to obtain the regularization term.

[0151] Optionally, the training module is further configured to:

[0152] For each column vector in the parameter matrix, calculate the magnitude of the column vector;

[0153] For each element in the column vector, divide the element by the magnitude to obtain a normalized column vector, thereby obtaining the weight matrix.

[0154] Optionally, the training module is further configured to:

[0155] Calculate the mean square between the dot product matrix and the identity matrix;

[0156] or,

[0157] The difference between the dot product matrix and the identity matrix is ​​calculated based on the loss function.

[0158] Optionally, the training module is further configured to:

[0159] The difference between the dot product matrix and the identity matrix is ​​calculated using the following formula:

[0160] mean[I-softmax(a*A)*I]

[0161] Where a is a hyperparameter controlling the strength of the regularization term, I is the identity matrix, and A is the dot product matrix.

[0162] Based on the various embodiments described above, it can be seen that the embodiments of the present invention solve the technical problem of poor interpretability of feature vectors in the prior art by using regularization terms to constrain each basis vector of the feature subspace during the training of the feature projection layer, so that the basis vectors of the feature subspace are mutually perpendicular. By adding orthogonal basis vector constraints to the feature projection layer, the embodiments of the present invention make the meaning of the feature vectors in the feature subspace clearer, enhance interpretability, reduce the error in calculating semantic similarity, and thus reduce the difficulty of subsequent learning tasks.

[0163] It should be noted that the specific implementation of the text recognition device described in this invention has been described in detail in the text recognition method described above, so the content will not be repeated here.

[0164] Figure 6 An exemplary system architecture 600 is shown, in which the method or apparatus for recognizing text can be applied according to embodiments of the present invention.

[0165] like Figure 6 As shown, system architecture 600 may include terminal devices 601, 602, and 603, a network 604, and a server 605. Network 604 serves as the medium for providing communication links between terminal devices 601, 602, and 603 and server 605. Network 604 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0166] Users can use terminal devices 601, 602, and 603 to interact with server 605 via network 604 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 601, 602, and 603, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0167] Terminal devices 601, 602, and 603 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0168] Server 605 can be a server that provides various services, such as a backend management server that supports shopping websites browsed by users using terminal devices 601, 602, and 603 (this is just an example). The backend management server can analyze and process data such as received item information query requests, and then feed the processing results back to the terminal devices.

[0169] It should be noted that the text recognition method provided in this embodiment of the invention is generally executed by server 605, and correspondingly, the text recognition device is generally located in server 605. The text recognition method provided in this embodiment of the invention can also be executed by terminal devices 601, 602, and 603, and correspondingly, the text recognition device can be located in terminal devices 601, 602, and 603.

[0170] It should be understood that Figure 6 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0171] The following is for reference. Figure 7 It shows a schematic diagram of the structure of a computer system 700 suitable for implementing a terminal device of the present invention. Figure 7 The terminal device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of the present invention.

[0172] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 702 or programs loaded from storage section 708 into random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the system 700. The CPU 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0173] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0174] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 709, and / or installed from removable medium 711. When the computer program is executed by central processing unit (CPU) 701, it performs the functions defined above in the system of this invention.

[0175] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0176] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer programs according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0177] The modules described in the embodiments of the present invention can be implemented in software or hardware. The described modules can also be housed in a processor; for example, a processor can be described as including an encoding module, a recognition module, and a training module, wherein the names of these modules do not necessarily limit the module itself.

[0178] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, which, when executed by the device, implement the following method: inputting text to be recognized into a pre-trained encoder to output a general semantic vector of the text to be recognized; inputting the general semantic vector of the text to be recognized into a trained feature projection layer to output a recognition result of the text to be recognized; wherein the training process of the feature projection layer uses regularization terms to constrain the basis vectors of the feature subspace so that the basis vectors of the feature subspace are mutually perpendicular.

[0179] According to the technical solution of the present invention, by employing a regularization term to constrain each basis vector of the feature subspace during the training of the feature projection layer, so that the basis vectors of the feature subspace are mutually perpendicular, the technical problem of poor interpretability of feature vectors in the prior art is overcome. The present invention, by adding orthogonal basis vector constraints to the feature projection layer, makes the meaning of the feature vectors in the feature subspace clearer, enhances interpretability, reduces the error in calculating semantic similarity, and thus reduces the difficulty of subsequent learning tasks.

[0180] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for recognizing text, characterized in that, include: The sample text is input into a pre-trained encoder to output a general semantic vector of the sample text; The general semantic vector of the sample text and its corresponding label are input into the neural network. At the same time, regularization terms are used to constrain each basis vector of the feature subspace so that the basis vectors of the feature subspace are perpendicular to each other. The neural network obtains the feature projection layer after iterative training. The text to be recognized is input into a pre-trained encoder to output a general semantic vector of the text to be recognized; The general semantic vector of the text to be identified is input into a trained feature projection layer to output the recognition result of the text to be identified. The training process of the feature projection layer employs regularization terms to constrain each basis vector of the feature subspace, ensuring that the basis vectors of the feature subspace are mutually perpendicular.

2. The method according to claim 1, characterized in that, Constraining the basis vectors of the feature subspace with regularization terms to make the basis vectors of the feature subspace mutually perpendicular includes: Initialize the parameter matrix; wherein, the parameter matrix is ​​the parameter of the feature projection layer; The difference between the parameter matrix and the identity matrix is ​​calculated to obtain the regularization term; The parameters of the feature projection layer are optimized with the goal of making the regularization term close to zero.

3. The method according to claim 2, characterized in that, Calculate the difference between the parameter matrix and the identity matrix to obtain the regularization term, including: The parameter matrix is ​​normalized to obtain the weight matrix; Multiplying the transpose of the weight matrix with the weight matrix yields a dot product matrix; wherein the dot product matrix represents the dot product between the basis vectors of the feature subspace. The difference between the dot product matrix and the identity matrix is ​​calculated to obtain the regularization term.

4. The method according to claim 3, characterized in that, The parameter matrix is ​​normalized to obtain the weight matrix, which includes: For each column vector in the parameter matrix, calculate the magnitude of the column vector; For each element in the column vector, divide the element by the magnitude to obtain a normalized column vector, thereby obtaining the weight matrix.

5. The method according to claim 3, characterized in that, The difference between the dot product matrix and the identity matrix is ​​calculated using the following method: Calculate the mean square between the dot product matrix and the identity matrix; or, The difference between the dot product matrix and the identity matrix is ​​calculated based on the loss function.

6. The method according to claim 5, characterized in that, The difference between the dot product matrix and the identity matrix is ​​calculated based on the loss function, including: The difference between the dot product matrix and the identity matrix is ​​calculated using the following formula: mean[I-softmax(a*A)*I] Where a is a hyperparameter controlling the strength of the regularization term, I is the identity matrix, and A is the dot product matrix.

7. A device for recognizing text, characterized in that, include: The encoding module is used to input the text to be recognized into a pre-trained encoder to output a general semantic vector of the text to be recognized; The recognition module is used to input the general semantic vector of the text to be recognized into the trained feature projection layer to output the recognition result of the text to be recognized. The training process of the feature projection layer uses a regularization term to constrain each basis vector of the feature subspace so that the basis vectors of the feature subspace are perpendicular to each other. The training module is used for: Before inputting the text to be recognized into the pre-trained encoder to output the general semantic vector of the text to be recognized, the sample text is input into the pre-trained encoder to output the general semantic vector of the sample text. The general semantic vector of the sample text and its corresponding label are input into the neural network. At the same time, regularization terms are used to constrain each basis vector of the feature subspace so that the basis vectors of the feature subspace are perpendicular to each other. After iterative training, the neural network obtains the feature projection layer.

8. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-6.

9. A computer-readable medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-6.

Images