Text semantic representation method and device based on information compression and spatial decomposition

Abstract

The invention discloses a text semantic representation method and device based on information compression and spatial decomposition. The method comprises the following steps: performing text semantic mining in statements on a to-be-represented text to obtain node vector representation of the to-be-represented text; performing cross-statement text semantic mining on the node vector representation of the to-be-represented text to obtain sentence node vector representation of the to-be-represented text; and determining text semantic representation of the to-be-represented text according to the sentence node vector representation.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and more specifically, to a text semantic representation method and device based on information compression and space decomposition. Background Art

[0002] Natural language processing technology is a very critical and rapidly developing field in artificial intelligence and computer science, dedicated to enabling computers to understand, interpret, generate and respond to human language. Since the 1950s, natural language processing technology has evolved from rule-based systems to statistical learning methods to the current deep learning revolution. In particular, the emergence of deep learning technology has enabled pre-trained language models such as BERT and GPT to learn the deep semantic features of language through massive data, greatly promoting the ability of language understanding, generation and interaction. The application areas of natural language processing technology are very wide, including but not limited to machine translation, speech recognition, sentiment analysis, text summarization and question-answering systems. These technologies are constantly changing the way we communicate with machines and improving the efficiency of information acquisition and processing.

[0003] Text semantic representation technology is an advanced natural language processing method that captures and represents the semantic information of text by converting text data into a vector form that can be understood by computers. This technology involves word embedding, context embedding, sentence and document level representation, enabling computers to perform complex language tasks such as sentiment analysis, machine translation, information retrieval and text classification. In addition, this technology can analyze the semantic similarity and relevance between texts, providing support for question-answering systems, recommendation systems and content management. With the continuous advancement of technology, text semantic representation technology plays an increasingly critical role in improving the automation and intelligence level of information processing, and is widely used in many fields such as market research, customer service and data analysis. The bag-of-words model ignores the order of words in the text. In this model, a document is represented as a long vector that contains the counts of all words in the document, but these counts are unordered. This means that, for example, the two sentences "the cat is sitting on the mat" and "the cat is sitting on the mat" will get exactly the same vector representation in the bag-of-words model, although they may be semantically different. This practice of ignoring word order may cause the model to fail to capture some important information in the text, such as modification relations and grammatical structures. Secondly, the bag-of-words model may be affected by the size of the vocabulary. As the vocabulary continues to grow, the dimensions that the model needs to handle will also increase, which may lead to the curse of dimensionality, that is, the performance of the model drops sharply as the dimension increases. In addition, a very large vocabulary may contain many rare words that are not important for text representation, and the presence of these words may interfere with the learning process of the model. The bag-of-words model may also be affected by uncommon words. In practical applications, there may be some rare words in the document, which occupy a position in the vocabulary but rarely appear during model training and prediction. This may cause the model to generate noise when processing these uncommon words, thereby affecting the performance of the model. In addition, the bag-of-words model cannot handle the synonym problem well. Since the bag-of-words model is based on word counts, it cannot recognize words with similar meanings. For example, "car" and "sedan" will be regarded as completely different words in the bag-of-words model, even though they can be used interchangeably in many cases. Finally, the bag-of-words model cannot capture the contextual information in the text. In natural language, the meaning of a word often depends on the context in which it is located. The bag-of-words model cannot represent this contextual relationship, so it may not accurately understand the meaning of the text. One of the main drawbacks of the Word2Vec model is that it cannot effectively handle the problem of polysemy. In Word2Vec, each word is mapped to a unique vector, which means that the same word in different contexts (such as "bank" can refer to the riverbank or the bank) will be given the same vector representation, thus ignoring the differences in their meanings in different contexts. Secondly, the Word2Vec model relies on local context information during training, usually through a fixed-size window to capture the words around the word.This approach may not capture broader contextual information, such as the semantic information of an entire sentence or paragraph, which may limit the model's ability to gain a deep semantic understanding of the text. In addition, the training of the Word2Vec model usually requires a lot of computing resources and time, especially when processing a large corpus. Although the model can be used for a variety of NLP tasks after training, the training process itself may be time-consuming. Another limitation of the Word2Vec model is that it is difficult to adapt to new or rare words. Since the model learns word vectors based on statistical co-occurrence information, words that rarely appear in the training data may not be able to obtain effective vector representations. Finally, the static nature of the Word2Vec model means that it cannot dynamically adjust word vectors based on specific tasks or contexts. This is in contrast to some later models such as BERT, which are able to dynamically adjust word representations based on specific tasks and contexts. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present invention provides a text semantic representation method and device based on information compression and space decomposition.

[0005] According to one aspect of the present invention, a text semantic representation method based on information compression and space decomposition is provided, comprising:

[0006] Perform text semantic mining on the text to be represented to obtain the node vector representation of the text to be represented;

[0007] Perform cross-sentence text semantic mining on the node vector representation of the text to be represented, and obtain the sentence node vector representation of the text to be represented;

[0008] According to the sentence node vector representation, the text semantic representation of the text to be represented is determined.

[0009] Optionally, the text to be represented is subjected to sentence semantic mining to obtain a node vector representation of the text to be represented, including:

[0010] Design an encoder for intra-sentence text semantic mining of a single sentence;

[0011] The intra-sentence text semantic mining encoder is used to perform intra-sentence text semantic mining on the text to be represented to obtain the node vector representation of the text to be represented.

[0012] Optionally, the training parameter optimization expression of the sentence text semantic mining encoder is:

[0013]

[0014] In the formula, represents mutual information, T Θ represents the encoder, Θ represents the trainable parameters of the encoder, and L i(X) and S j (X) represents the i-th relevant feature and j-th irrelevant feature of X, respectively. is the weight parameter, X represents a single sentence, Represents the text to be represented, and Ns represents the number of irrelevant features.

[0015] Optionally, cross-sentence text semantic mining is performed on the node vector representation of the text to be represented to obtain the sentence node vector representation of the text to be represented, including:

[0016] Design a routing network for text decomposition embedding space;

[0017] Design a dedicated projector for each representation in the text decomposition embedding space;

[0018] Based on the routing network and the dedicated projector, an embedding representation space decomposition model is constructed;

[0019] The embedding representation space decomposition model is used to perform cross-sentence text semantic mining on the node vector representation to obtain the sentence node vector representation of the text to be represented.

[0020] Optionally, the embedding representation space decomposition model is used to perform cross-sentence text semantic mining on the node vector representation to obtain the sentence node vector representation of the text to be represented, including:

[0021] A routing network is used to represent the context of the node vector as an embedding space vector.

[0022] The classification labels are respectively attached to the embedding space vectors of the context to obtain additional vectors;

[0023] A dedicated projector is used to perform spatial decomposition on the additional vectors to obtain the sentence node vector representation of the text to be represented.

[0024] Optionally, an additional vector based on the Softmax probability The upper or lower range group of:

[0025]

[0026] In the formula, G 1 (·) is the group classification of the linear layer; j = 1 or 2 is used to represent sentence 1 or sentence 2; is the additional vector; h xj is the embedding space vector;

[0027] The loss function of the routing network is:

[0028]

[0029] Where BCE is the binary cross entropy loss; y indicates whether the sentence pair is an upper bound or a lower bound.

[0030] Optionally, the expression for a dedicated projector is:

[0031]

[0032] In the formula, β j From max get The highest probability; z xj is a representation mapped to the upper range or lower range subspace; j = 1 or 2 is used to represent sentence 1 or sentence 2;

[0033] The loss function of the dedicated projector is:

[0034]

[0035] In the formula, y sim is the STS score; G 2 (concat(z x1 ,z x2 )) indicates that the linear layer G2 converts z x1 With z x2 Connect them; BCE is the binary cross entropy loss.

[0036] Optionally, the loss function for the embedding representation space decomposition model is:

[0037]

[0038] In the formula, α 1 and α 2 is the loss weight obtained by adjusting the development set.

[0039] According to another aspect of the present invention, there is provided a text semantic representation device based on information compression and space decomposition, comprising:

[0040] The first acquisition module is used to perform text semantic mining on the text to be represented and obtain the node vector representation of the text to be represented;

[0041] The second acquisition module is used to perform cross-sentence text semantic mining on the node vector representation of the text to be represented, and obtain the sentence node vector representation of the text to be represented;

[0042] The determination module is used to determine the text semantic representation of the text to be represented according to the sentence node vector representation.

[0043] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the method described in any one of the above aspects of the present invention.

[0044] According to another aspect of the present invention, an electronic device is provided, comprising: a processor; a memory for storing instructions executable by the processor; the processor is configured to read the executable instructions from the memory and execute the instructions to implement the method described in any one of the above aspects of the present invention.

[0045] Thus, the present application achieves efficient intra-sentence text semantic mining through innovative encoder design and information theory optimization, and effectively mines cross-sentence text context information through the combination of routing network and dedicated projector. This technical solution not only improves the efficiency and quality of information compression, but also adapts to a variety of application scenarios such as bandwidth-limited transmission, private data sharing, and unbiased model development through flexible parameter adjustment. In addition, through the hard selection process and supervised contrastive learning, the present invention enhances the model's understanding of semantics, and through end-to-end learning objectives, it achieves effective training of representation learning and classification loss. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] A more complete understanding of exemplary embodiments of the present invention may be obtained by referring to the following drawings:

[0047] Figure 1 It is a flowchart of a text semantic representation method based on information compression and space decomposition provided by an exemplary embodiment of the present invention;

[0048] Figure 2 It is a structural schematic diagram of a text semantic representation device based on information compression and space decomposition provided by an exemplary embodiment of the present invention;

[0049] Figure 3 This is a structure of an electronic device provided by an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0050] Below, the exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described here.

[0051] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless specifically stated otherwise.

[0052] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present invention are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.

[0053] It should also be understood that, in the embodiments of the present invention, “plurality” may refer to two or more than two, and “at least one” may refer to one, two or more than two.

[0054] It should also be understood that any component, data or structure mentioned in the embodiments of the present invention can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0055] In addition, the term "and / or" in the present invention is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present invention generally indicates that the associated objects before and after are in an "or" relationship.

[0056] It should also be understood that the description of the various embodiments of the present invention focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0057] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0058] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0059] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0060] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0061] Embodiments of the present invention can be applied to electronic devices such as terminal devices, computer systems, servers, etc., which can operate with many other general or special computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments and / or configurations suitable for use with electronic devices such as terminal devices, computer systems, servers, etc. include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.

[0062] Electronic devices such as terminal devices, computer systems, servers, etc. can be described in the general context of computer system executable instructions (such as program modules) executed by computer systems. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.

[0063] Exemplary Methods

[0064] Figure 1 is a flow chart of a text semantic representation method based on information compression and space decomposition provided by an exemplary embodiment of the present invention. This embodiment can be applied to electronic devices, such as Figure 1 As shown, the text semantic representation method 100 based on information compression and space decomposition includes the following steps:

[0065] Step 101, performing sentence semantic mining on the text to be represented to obtain a node vector representation of the text to be represented;

[0066] Step 102, performing cross-sentence text semantic mining on the node vector representation of the text to be represented, and obtaining the sentence node vector representation of the text to be represented;

[0067] Step 103: Determine the text semantic representation of the text to be represented based on the sentence node vector representation.

[0068] Specifically, the technical solution of this application provides an efficient, accurate and privacy-focused solution for the field of text semantic mining through innovative encoder design, information theory evaluation, routing network and dedicated projector application, as well as end-to-end training methods. In terms of intra-sentence text semantic mining, the patent introduces an encoder that achieves information compression, key information retention and sensitive information filtering by optimizing trainable parameters to improve the information theory compression effect and ensure data privacy and fairness. The encoder uses a fixed architecture and non-negative parameters to estimate and optimize mutual information through an information theory evaluator, thereby filtering out sensitive information while retaining key information. In terms of cross-sentence text semantic mining, the patent proposes a combination of a routing network and a dedicated projector for mining and understanding contextual relationships between sentence pairs. The routing network enhances the understanding of sentence pair relationships by introducing additional contextual information, while the dedicated projector decomposes the embedded representation into upper range or lower range subspaces through a hard selection process, avoiding subspace overlap and improving model performance. In addition, the patent also proposes an improved learning objective to improve the semantic understanding of the representation generated by the projector through linear similarity prediction, avoiding the performance damage that may be caused by batch contrastive learning.

[0069] The technical solution of this application includes two aspects: intra-sentence text semantic mining and cross-sentence text semantic mining. The details are as follows.

[0070] 1. Semantic mining of text within sentences

[0071] For a single sentence, design an encoder So that when applied to each sentence When , the encoded sentence satisfies certain theoretical information properties. Here, Θ represents the trainable parameters of the encoder, L i (X) and S j (X) respectively represent the i-th relevant (common) feature and j-th irrelevant (sensitive) feature of X. The architecture of the encoder is fixed, and its trainable parameters are optimized as follows:

[0072]

[0073] parameter takes non-negative values ​​and is adjusted to achieve a desired balance between competing objectives in the optimization problem, while the cardinality of Z determines the compression level of the encoder.

[0074] The first term in the formula is dedicated to information compression, to maximize the mutual information between the compressed sentence and its original representation. Information-theoretic compression outperforms classical approaches where simple distortion measures are minimized (e.g., mean squared error after decompression). The second term in the formula is dedicated to utility, making the embedding more representative of the selected information by focusing adjustable attention on certain parts of the sentence content. One benefit of this scheme is that it preserves all critical information in the face of unavoidable loss of information. The last term in the formula is related to filtering out sensitive information from the encoded embedding to ensure that it is not used by unauthorized users when the data is released publicly (privacy), or to ensure that bias is not included in the model trained using the encoded representation (fairness).

[0075] The above problems can be solved by setting the design parameters appropriately. When , the problem reduces to a task-independent information-theoretic compression. When γ = 0, the problem is task-oriented, essentially filtering out sensitive information from the dataset. Finally, when γ = 0, the task-oriented problem is a privacy-utility tradeoff, where the sentences in the dataset are encoded so that they can be used to train a model to recognize public information (utility purposes). However, they are much less useful for training a model to recognize certain sensitive information (for privacy purposes) for the image domain with one public label and one private label.

[0076] The architecture and training process of the model mainly include components: sentence encoder and information theoretic evaluator. The trained encoder can be used on any dataset from the distribution used in the training phase to tailor its information theoretic properties; for various applications such as bandwidth-constrained transmission, private data sharing or unbiased model development.

[0077] The encoder consists of a ready-made sentence embedding, which receives the tokens of a sentence in sequence and outputs a vector representing the entire sentence. The sentence embedding is fed into a neural network, which projects the original sentence embedding into a low-dimensional embedding with certain information-theoretic properties. The neural network is the trainable part of the encoder, which is designed using an information-theoretic evaluator. The information-theoretic evaluator consists of an estimator for the mutual information term in the optimization problem. The mutual information is defined between two random variables A and B, and is empirically estimated using samples drawn from a joint distribution following the Donsker-Varadhan formula for mutual information. The formula states that the mutual information can be obtained by finding a function that maximizes an expression that depends only on the first-order moment of the probability term. Based on this, the first-order moment can be empirically determined using the mean of the samples. Furthermore, the set of functions that perform the search can be modeled with a neural network, thereby modeling any well-behaved function with appropriate weights. Finally, the optimization is performed using stochastic gradient descent, where at each step, a batch of samples in a public dataset are used to estimate the expectation and its gradient. The maximum converged value then represents the mutual information.

[0078] 2. Cross-sentence text semantic mining

[0079] In order to mine the textual context information between sentences, the embedding representation space is decomposed into a routing network and a dedicated projector as follows.

[0080] 1). Routing network

[0081] The routing network can be used as a decomposed embedding space, representing the context as h x , introducing the [CLS] label into the routing network. This additional information allows the routing mechanism to understand the relationship between sentence pairs (sent1, sent2). By incorporating the global representation, the sentence pair input is element-wise additional Represent the routing network as a linear layer G 1 (·) group classification, and It is based on Softmax probability The upper or lower range group of:

[0082]

[0083] Where j represents sent1 (j=1) or sent2 (j=2), and then from sent1h x1 and sent2 h x2 Calculation. To help the routing network classify the input, binary cross entropy (BCE) is used as shown below.

[0084]

[0085] Where y indicates whether the sentence pair is an upper bound or a lower bound. Classification loss Use as part of overall learning objectives.

[0086] 2). Dedicated projector

[0087] The main goal of the dedicated projector is to transform the embedding representation h xj Decompose into upper or lower range subspaces. Previous work in embedding space decomposition produces composite representations, where the final representation is obtained by computing the vector sum from multiple projectors’ outputs. We find that this soft selection approach leads to overlap between subspaces, which is detrimental to model performance. We therefore derive a hard selection procedure where a dedicated projector has a separate projection for each class and only one projection is used for each representation. In particular, two dedicated projectors, an upper bound projector Upper(·) and a lower range projector Lower(·), represent h xj Mapped to the upper or lower range subspace as follows:

[0088]

[0089] where β j is from get The highest probability, and h xj is the representation mapped to the upper range or lower range subspace. Using the output of different projectors, we obtain the representation pairs of upper range and lower range samples, h x1 and h x2 However, h xj is generated by the random weights of the specialized projectors. We need a way to improve the representation of the projectors.

[0090] One of the key parts of this approach is to improve the representation z produced by the specialized projectors Upper(·) or Lower(·) xj A common practice is to apply supervised contrastive learning to pairwise representations. However, we find that batch contrastive learning hurts the performance of projectors because it requires combining representations from different projectors (Upper(·) and Lower(·)) in the same mini-batch. Therefore, a more appropriate learning objective is designed for the classification and ranking mechanism, which is to predict the linear similarity for each projector separately. Specifically, a linear layer G2 is used to transform z x1 With z x2 concatenate, where the output of the linear is a regression number from zero (dissimilarity) to one (similarity). Then, the difference between the output and ysim (STS score) is minimized using the BCE loss as follows:

[0091]

[0092] The final training loss is an end-to-end example of representation learning and classification loss:

[0093]

[0094] Parameter α 1 and α 2 is the loss weight obtained by adjusting the development set.

[0095] 1. A method for intra-sentence text semantic mining: An encoder is designed to encode a single sentence so that the encoded sentence satisfies specific information theoretic properties. The encoder contains trainable parameters that are adjusted through an optimization process to achieve goals such as information compression, utility maximization, and sensitive information filtering. The architecture of the encoder is fixed, but the parameters are trainable, and non-negative values ​​are optimized to balance different objectives. By adjusting the design parameters, issues such as information theoretic compression, task relevance, privacy, and fairness can be addressed. The encoder consists of sentence embeddings and a neural network that projects the sentence embeddings into a low-dimensional embedding with specific information theoretic properties. An information theoretic estimator is used to estimate the mutual information and optimized by stochastic gradient descent.

[0096] 2. A method for cross-sentence text semantic mining: The text context information between sentences is mined through routing networks and dedicated projectors. The routing network is used to decompose the embedding space, understand the relationship between sentence pairs, and classify through binary cross entropy. The dedicated projector decomposes the embedding representation into an upper range or lower range subspace, adopts a hard selection process, and uses only one projection for each representation. In order to improve semantic understanding, supervised contrastive learning is adopted, but batch contrastive learning is avoided, and linear similarity prediction is performed for each projector instead. The final training loss combines representation learning and classification loss to form an end-to-end learning paradigm.

[0097] Thus, the present application achieves efficient intra-sentence text semantic mining through innovative encoder design and information theory optimization, and effectively mines cross-sentence text context information through the combination of routing network and dedicated projector. This technical solution not only improves the efficiency and quality of information compression, but also adapts to a variety of application scenarios such as bandwidth-limited transmission, private data sharing, and unbiased model development through flexible parameter adjustment. In addition, through the hard selection process and supervised contrastive learning, the present invention enhances the model's understanding of semantics, and through end-to-end learning objectives, it achieves effective training of representation learning and classification loss.

[0098] Exemplary Devices

[0099] Figure 2FIG. 1 is a schematic diagram of a text semantic representation device based on information compression and space decomposition provided by an exemplary embodiment of the present invention. Figure 2 As shown, the device 200 includes:

[0100] A first acquisition module 210 is used to perform sentence semantic mining on the text to be represented and obtain a node vector representation of the text to be represented;

[0101] The second acquisition module 220 is used to perform cross-sentence text semantic mining on the node vector representation of the text to be represented, and obtain the sentence node vector representation of the text to be represented;

[0102] The determination module 230 is used to determine the text semantic representation of the text to be represented according to the sentence node vector representation.

[0103] Optionally, the first acquisition module 210 includes:

[0104] Design an encoder for intra-sentence text semantic mining of a single sentence;

[0105] The intra-sentence text semantic mining encoder is used to perform intra-sentence text semantic mining on the text to be represented to obtain the node vector representation of the text to be represented.

[0106] Optionally, the training parameter optimization expression of the sentence text semantic mining encoder is:

[0107]

[0108] In the formula, represents mutual information, T Θ represents the encoder, Θ represents the trainable parameters of the encoder, and L i (X) and S j (X) represents the i-th relevant feature and j-th irrelevant feature of X, respectively. is the weight parameter, X represents a single sentence, Represents the text to be represented, and Ns represents the number of irrelevant features.

[0109] Optionally, the second acquisition module 220 includes:

[0110] Design a routing network for text decomposition embedding space;

[0111] Design a dedicated projector for each representation in the text decomposition embedding space;

[0112] Based on the routing network and the dedicated projector, an embedding representation space decomposition model is constructed;

[0113] The embedding representation space decomposition model is used to perform cross-sentence text semantic mining on the node vector representation to obtain the sentence node vector representation of the text to be represented.

[0114] Optionally, the embedding representation space decomposition model is used to perform cross-sentence text semantic mining on the node vector representation to obtain the sentence node vector representation of the text to be represented, including:

[0115] A routing network is used to represent the context of the node vector as an embedding space vector.

[0116] The classification labels are respectively attached to the embedding space vectors of the context to obtain additional vectors;

[0117] A dedicated projector is used to perform spatial decomposition on the additional vectors to obtain the sentence node vector representation of the text to be represented.

[0118] Optionally, an additional vector based on the Softmax probability The upper or lower range group of:

[0119]

[0120] In the formula, G 1 (·) is the group classification of the linear layer; j = 1 or 2 is used to represent sentence 1 or sentence 2; is the additional vector; h xj is the embedding space vector;

[0121] The loss function of the routing network is:

[0122]

[0123] Where BCE is the binary cross entropy loss; y indicates whether the sentence pair is an upper bound or a lower bound.

[0124] Optionally, the expression for a dedicated projector is:

[0125]

[0126] In the formula, β j From max get The highest probability; z xj is a representation mapped to the upper range or lower range subspace; j = 1 or 2 is used to represent sentence 1 or sentence 2;

[0127] The loss function of the dedicated projector is:

[0128]

[0129] In the formula, y sim is the STS score; G2 (concat(z x1 ,z x2 )) indicates that the linear layer G2 converts z x1 With z x2 Connect them; BCE is the binary cross entropy loss.

[0130] Optionally, the loss function for the embedding representation space decomposition model is:

[0131]

[0132] In the formula, α 1 and α 2 is the loss weight obtained by adjusting the development set.

[0133] Exemplary Electronic Devices

[0134] Figure 3 This is a structure of an electronic device provided by an exemplary embodiment of the present invention. Figure 3 As shown, the electronic device 30 includes one or more processors 31 and a memory 32 .

[0135] The processor 31 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0136] The memory 32 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 31 may run the program instructions to implement the methods of the software programs of the various embodiments of the present invention described above and / or other desired functions. In one example, the electronic device may also include: an input device 33 and an output device 34, which are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0137] In addition, the input device 33 may also include, for example, a keyboard, a mouse, and the like.

[0138] The output device 34 can output various information to the outside. The output device 34 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.

[0139] Of course, to simplify, Figure 3 Only some of the components related to the present invention in the electronic device are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, the electronic device may further include any other appropriate components according to specific application conditions.

[0140] Exemplary computer program products and computer-readable storage media

[0141] In addition to the above-mentioned methods and devices, an embodiment of the present invention may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the method according to various embodiments of the present invention described in the above-mentioned "Exemplary Method" section of this specification.

[0142] The computer program product may be written in any combination of one or more programming languages ​​to write program code for performing the operations of the embodiments of the present invention, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0143] In addition, an embodiment of the present invention may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enable the processor to execute the steps of the method according to various embodiments of the present invention described in the above “Exemplary Method” section of this specification.

[0144] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, system or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0145] The basic principle of the present invention is described above in conjunction with specific embodiments. However, it should be pointed out that the advantages, strengths, effects, etc. mentioned in the present invention are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. must be possessed by each embodiment of the present invention. In addition, the specific details disclosed above are only for the purpose of illustration and facilitation of understanding, rather than limitation, and the above details do not limit the present invention to be implemented by adopting the above specific details.

[0146] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0147] The block diagrams of the devices, systems, equipment, and systems involved in the present invention are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, systems, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The word "such as" used here refers to the phrase "such as but not limited to", and can be used interchangeably with it.

[0148] The method and system of the present invention may be implemented in many ways. For example, the method and system of the present invention may be implemented by software, hardware, firmware or any combination of software, hardware, firmware. The above order of steps for the method is only for illustration, and the steps of the method of the present invention are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present invention may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present invention. Thus, the present invention also covers a recording medium storing a program for executing the method according to the present invention.

[0149] It should also be noted that in the system, device and method of the present invention, each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent schemes of the present invention. The above description of the disclosed aspects is provided to enable any technician in the field to make or use the present invention. Various modifications to these aspects are very obvious to those skilled in the art, and the general principles defined here can be applied to other aspects without departing from the scope of the present invention. Therefore, the present invention is not intended to be limited to the aspects shown here, but in accordance with the widest range consistent with the principles and novel features disclosed here.

[0150] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present invention to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.

Claims

1. A text semantic representation method based on information compression and space decomposition, characterized in that: include: Performing sentence semantic mining on the text to be represented to obtain a node vector representation of the text to be represented; Performing cross-sentence text semantic mining on the node vector representation of the text to be represented to obtain the sentence node vector representation of the text to be represented; According to the sentence node vector representation, a text semantic representation of the text to be represented is determined.

2. The method according to claim 1, characterized in that: Performing sentence semantic mining on the text to be represented to obtain a node vector representation of the text to be represented includes: Design an encoder for intra-sentence text semantic mining of a single sentence; The intra-sentence text semantic mining encoder is used to perform intra-sentence text semantic mining on the text to be represented, and the node vector representation of the text to be represented is obtained.

3. The method according to claim 2, characterized in that The training parameter optimization expression of the text semantic mining encoder in the sentence is: In the formula, represents mutual information, T Θ represents the encoder, Θ represents the trainable parameters of the encoder, L i (X) and S j (X) represents the i-th relevant feature and j-th irrelevant feature of X, respectively. is the weight parameter, X represents a single sentence, Represents the text to be represented, and Ns represents the number of irrelevant features.

4. The method according to claim 1, characterized in that Performing cross-sentence text semantic mining on the node vector representation of the text to be represented to obtain the sentence node vector representation of the text to be represented, including: Design a routing network for text decomposition embedding space; Design a dedicated projector for each representation in the text decomposition embedding space; Constructing the embedding representation space decomposition model according to the routing network and the dedicated projector; The embedding representation space decomposition model is used to perform cross-sentence text semantic mining on the node vector representation to obtain the sentence node vector representation of the text to be represented.

5. The method according to claim 4, characterized in that Using the embedding representation space decomposition model to perform cross-sentence text semantic mining on the node vector representation to obtain the sentence node vector representation of the text to be represented, including: The routing network is used to represent the context of the node vector as an embedding space vector respectively; The classification labels are respectively attached to the embedding space vectors of the context to obtain additional vectors; The dedicated projector is used to perform spatial decomposition on the additional vector to obtain the sentence node vector representation of the text to be represented.

6. The method according to claim 5, characterized in that The additional vector is based on the Softmax probability The upper or lower range group of: Where G1(·) is the group classification of the linear layer; j = 1 or 2 is used to represent sentence 1 or sentence 2; is the additional vector; h xj is the embedding space vector; The loss function of the routing network is: Where BCE is the binary cross entropy loss; y indicates whether the sentence pair is an upper bound or a lower bound.

7. The method according to claim 6, characterized in that The expression of the dedicated projector is: In the formula, β j From max get The highest probability; z xj is a representation mapped to the upper range or lower range subspace; j = 1 or 2 is used to represent sentence 1 or sentence 2; The loss function of the dedicated projector is: In the formula, y sim is the STS score; G2(concat(z x1 ,z x2 )) indicates that the linear layer G2 converts z x1 With z x2 Connect them; BCE is the binary cross entropy loss.

8. The method according to claim 7, characterized in that The loss function of the embedding representation space decomposition model is: Where α1 and α2 are the loss weights obtained by adjusting the development set.

9. A text semantic representation device based on information compression and spatial decomposition, characterized in that: include: A first acquisition module is used to perform sentence semantic mining on the text to be represented and obtain a node vector representation of the text to be represented; A second acquisition module is used to perform cross-sentence text semantic mining on the node vector representation of the text to be represented, and obtain the sentence node vector representation of the text to be represented; A determination module is used to determine the text semantic representation of the text to be represented based on the sentence node vector representation.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and the computer program is used to execute the method according to any one of claims 1 to 8.

11. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the method described in any one of claims 1-8.

Images