A text semantic representation method and device based on information compression and spatial decomposition

Abstract

The present invention discloses a text semantic representation method and device based on information compression and spatial decomposition. The method comprises: performing intra-sentence text semantic mining on the text to be represented to obtain a node vector representation of the text to be represented; performing cross-sentence text semantic mining on the node vector representation of the text to be represented to obtain a sentence node vector representation of the text to be represented; and determining a text semantic representation of the text to be represented based on the sentence node vector representation.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and more particularly, to a text semantic representation method and device based on information compression and spatial decomposition. Background Art

[0002] Natural language processing (NLP) is a critical and rapidly developing field in artificial intelligence and computer science, dedicated to enabling computers to understand, interpret, generate, and respond to human language. Since the 1950s, NLP has evolved from rule-based systems to statistical learning methods, and finally to the current deep learning revolution. The emergence of deep learning, in particular, has enabled pre-trained language models such as BERT and GPT to learn the deep semantic features of language from massive amounts of data, significantly advancing language understanding, generation, and interaction capabilities. NLP has a wide range of applications, including but not limited to machine translation, speech recognition, sentiment analysis, text summarization, and question-answering systems. These technologies are continuously changing the way we communicate with machines, improving the efficiency of information acquisition and processing.

[0003] Text semantic representation technology is an advanced natural language processing method that captures and represents the semantic information of text by converting text data into a vector form that computers can understand. This technology involves word embeddings, contextual embeddings, and sentence- and document-level representations, enabling computers to perform complex language tasks such as sentiment analysis, machine translation, information retrieval, and text categorization. Furthermore, this technology can analyze the semantic similarity and relevance between texts, supporting question-answering systems, recommendation systems, and content management. With continuous technological advancements, text semantic representation technology is playing an increasingly critical role in improving the automation and intelligence of information processing, and is widely used in fields such as market research, customer service, and data analysis. The bag-of-words model ignores the order of words in a text. In this model, a document is represented as a long vector containing counts of all the words in the document, but these counts are unordered. This means that, for example, the sentences "the cat sat on the mat" and "the cat sat on the mat" will receive exactly the same vector representation in the bag-of-words model, despite potentially different semantic meanings. This disregard for word order can result in the model failing to capture important information in the text, such as modifiers and grammatical structure. Secondly, the bag-of-words model may be affected by vocabulary size. As the vocabulary continues to grow, the number of dimensions the model must handle also increases, potentially leading to the curse of dimensionality, where model performance plummets as the number of dimensions increases. Furthermore, a very large vocabulary may contain many rare words that are not essential for text representation, and their presence may interfere with the model's learning process. The bag-of-words model may also be affected by uncommon words. In real applications, documents may contain rare words that occupy a position in the vocabulary but rarely appear during model training and prediction. This can cause noise in the model's processing of these rare words, affecting model performance. Furthermore, the bag-of-words model does not handle synonyms well. Because the bag-of-words model is based on word counts, it cannot identify words with similar meanings. For example, "car" and "sedan" would be treated as completely different words in the bag-of-words model, even though they can be used interchangeably in many contexts. Finally, the bag-of-words model fails to capture contextual information in the text. In natural language, the meaning of a word often depends on the context in which it is used. The bag-of-words model cannot represent this contextual relationship and may therefore fail to accurately understand the meaning of the text. A major drawback of the Word2Vec model is its inability to effectively handle polysemy. In Word2Vec, each word is mapped to a unique vector, meaning that the same word in different contexts (e.g., "bank" can refer to a riverbank or a bank) is assigned the same vector representation, ignoring the differences in their meanings in different contexts. Furthermore, the Word2Vec model relies on local contextual information during training, typically capturing surrounding words through a fixed-size window.This approach may not capture broader contextual information, such as the semantic information of an entire sentence or paragraph, which may limit the model's ability to gain a deep semantic understanding of the text. In addition, training the Word2Vec model typically requires a lot of computing resources and time, especially when processing large-scale corpora. Although the model can be used for various NLP tasks after training, the training process itself can be time-consuming. Another limitation of the Word2Vec model is that it has difficulty adapting to new or rare words. Because the model learns word vectors based on statistical co-occurrence information, words that rarely appear in the training data may not receive effective vector representations. Finally, the static nature of the Word2Vec model means that it cannot dynamically adjust word vectors based on specific tasks or contexts. This is in contrast to later models such as BERT, which can dynamically adjust word representations based on specific tasks and contexts. Summary of the Invention

[0004] In view of the deficiencies of the prior art, the present invention provides a text semantic representation method and device based on information compression and spatial decomposition.

[0005] According to one aspect of the present invention, a text semantic representation method based on information compression and spatial decomposition is provided, comprising:

[0006] Perform text semantic mining on the text to be represented to obtain the node vector representation of the text to be represented;

[0007] Perform cross-sentence text semantic mining on the node vector representation of the text to be represented to obtain the sentence node vector representation of the text to be represented;

[0008] According to the sentence node vector representation, the text semantic representation of the text to be represented is determined.

[0009] Optionally, performing sentence semantic mining on the text to be represented to obtain a node vector representation of the text to be represented includes:

[0010] Design an encoder for intra-sentence text semantic mining of a single sentence;

[0011] The intra-sentence text semantic mining encoder is used to perform intra-sentence text semantic mining on the text to be represented to obtain the node vector representation of the text to be represented.

[0012] Optionally, the training parameter optimization expression of the sentence text semantic mining encoder is:

[0013]

[0014] Where, represents mutual information, T Θ represents the encoder, Θ represents the trainable parameters of the encoder, L i(X) and S j (X) represents the i-th relevant feature and the j-th irrelevant feature of X, respectively. is the weight parameter, X represents a single sentence, Represents the text to be represented, and Ns represents the number of irrelevant features.

[0015] Optionally, cross-sentence text semantic mining is performed on the node vector representation of the text to be represented to obtain the sentence node vector representation of the text to be represented, including:

[0016] Design a routing network for text decomposition embedding space;

[0017] Design a dedicated projector for each representation in the text factorization embedding space;

[0018] Based on the routing network and the dedicated projector, an embedding representation space decomposition model is constructed;

[0019] The embedding representation space decomposition model is used to perform cross-sentence text semantic mining on the node vector representation to obtain the sentence node vector representation of the text to be represented.

[0020] Optionally, using an embedding representation space decomposition model to perform cross-sentence text semantic mining on the node vector representation to obtain the sentence node vector representation of the text to be represented, including:

[0021] A routing network is used to represent the context of the node vector as an embedding space vector.

[0022] Append the classification labels to the context embedding space vectors to obtain additional vectors;

[0023] A dedicated projector is used to perform spatial decomposition on the additional vectors to obtain the sentence node vector representation of the text to be represented.

[0024] Optionally, an additional vector is based on the Softmax probability The upper or lower range group of:

[0025]

[0026] Where G1(·) is the group classification of the linear layer; j = 1 or 2 is used to represent sentence 1 or sentence 2; is an additional vector; h xj is the embedding space vector;

[0027] The loss function of the routing network is:

[0028]

[0029] Where BCE is the binary cross entropy loss; y indicates whether the sentence pair is an upper bound or a lower bound.

[0030] Optionally, the expression for a specialized projector is:

[0031]

[0032] Where, β j From max get The highest probability of z xj is a representation mapped to the upper or lower range subspace; j = 1 or 2 is used to represent sentence 1 or sentence 2;

[0033] The loss function of the dedicated projector is:

[0034]

[0035] Where y sim is the STS score; G2(concat(z x1 ,z x2 )) indicates that the linear layer G2 converts z x1 With z x2 Connect them; BCE is the binary cross entropy loss.

[0036] Optionally, the loss function for the embedding representation space decomposition model is:

[0037]

[0038] Where α1 and α2 are the loss weights obtained by adjusting the development set.

[0039] According to another aspect of the present invention, a text semantic representation device based on information compression and spatial decomposition is provided, comprising:

[0040] The first acquisition module is used to perform text semantic mining on the text to be represented and obtain the node vector representation of the text to be represented;

[0041] The second acquisition module is used to perform cross-sentence text semantic mining on the node vector representation of the text to be represented, and obtain the sentence node vector representation of the text to be represented;

[0042] The determination module is used to determine the text semantic representation of the text to be represented based on the sentence node vector representation.

[0043] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the method according to any one of the above aspects of the present invention.

[0044] According to another aspect of the present invention, an electronic device is provided, comprising: a processor; a memory for storing instructions executable by the processor; and the processor for reading the executable instructions from the memory and executing the instructions to implement the method described in any one of the above aspects of the present invention.

[0045] Thus, this application achieves efficient intra-sentence text semantic mining through innovative encoder design and information-theoretic optimization, while effectively mining cross-sentence text context information through the combination of routing network and dedicated projector. This technical solution not only improves the efficiency and quality of information compression, but also adapts to a variety of application scenarios such as bandwidth-limited transmission, private data sharing, and unbiased model development through flexible parameter adjustment. In addition, through the hard selection process and supervised contrastive learning, the present invention enhances the model's understanding of semantics and achieves effective training of representation learning and classification loss through end-to-end learning objectives. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] A more complete understanding of exemplary embodiments of the present invention may be obtained by referring to the following drawings:

[0047] Figure 1 1 is a flow chart of a text semantic representation method based on information compression and spatial decomposition provided by an exemplary embodiment of the present invention;

[0048] Figure 2 1 is a schematic structural diagram of a text semantic representation device based on information compression and spatial decomposition provided by an exemplary embodiment of the present invention;

[0049] Figure 3 This is a structure of an electronic device provided by an exemplary embodiment of the present invention. DETAILED DESCRIPTION

[0050] Below, the exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described herein.

[0051] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless specifically stated otherwise.

[0052] Those skilled in the art will understand that the terms "first" and "second" in the embodiments of the present invention are only used to distinguish different steps, devices or modules, and neither represent any specific technical meaning nor indicate the necessary logical order between them.

[0053] It should also be understood that, in the embodiments of the present invention, “a plurality of” may refer to two or more than two, and “at least one” may refer to one, two or more than two.

[0054] It should also be understood that any component, data or structure mentioned in the embodiments of the present invention can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.

[0055] In addition, the term "and / or" in this invention merely describes an association relationship between related objects, indicating that three possible relationships exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. Furthermore, the character " / " in this invention generally indicates that the related objects are in an "or" relationship.

[0056] It should also be understood that the description of the various embodiments of the present invention focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced with each other. For the sake of brevity, they will not be described one by one.

[0057] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0058] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0059] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.

[0060] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0061] Embodiments of the present invention can be applied to electronic devices such as terminal devices, computer systems, and servers, and can operate in conjunction with numerous other general-purpose or specialized computing system environments or configurations. Examples of well-known terminal devices, computing systems, environments, and / or configurations suitable for use with terminal devices, computer systems, servers, and other electronic devices include, but are not limited to, personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, minicomputer systems, mainframe computer systems, and distributed cloud computing technology environments including any of the above.

[0062] Electronic devices such as terminal devices, computer systems, and servers can be described in the general context of computer system-executable instructions (such as program modules) executed by a computer system. Generally, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked via a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media, including storage devices.

[0063] Exemplary Methods

[0064] Figure 1 This is a flow chart of a text semantic representation method based on information compression and spatial decomposition provided by an exemplary embodiment of the present invention. This embodiment can be applied to electronic devices, such as Figure 1 As shown, the text semantic representation method 100 based on information compression and space decomposition includes the following steps:

[0065] Step 101: perform sentence semantic mining on the text to be represented to obtain a node vector representation of the text to be represented;

[0066] Step 102: Perform cross-sentence text semantic mining on the node vector representation of the text to be represented to obtain the sentence node vector representation of the text to be represented;

[0067] Step 103: Determine the text semantic representation of the text to be represented based on the sentence node vector representation.

[0068] Specifically, the technical solution of this application provides an efficient, accurate, and privacy-preserving solution for text semantic mining through the application of innovative encoder design, information-theoretic evaluation, routing networks, and dedicated projectors, as well as an end-to-end training method. For intra-sentence text semantic mining, the patent describes an encoder that optimizes trainable parameters to achieve information compression, key information retention, and sensitive information filtering, improving information-theoretic compression while ensuring data privacy and fairness. The encoder utilizes a fixed architecture and non-negative parameters, and uses an information-theoretic evaluator to estimate and optimize mutual information, thereby preserving key information while filtering out sensitive information. For cross-sentence text semantic mining, the patent proposes a combination of a routing network and a dedicated projector to mine and understand the contextual relationships between sentence pairs. The routing network enhances understanding of sentence pair relationships by introducing additional contextual information, while the dedicated projector decomposes the embedded representation into upper or lower range subspaces through a hard selection process, avoiding subspace overlap and improving model performance. Furthermore, the patent proposes an improved learning objective that improves semantic understanding of the representations produced by the projector through linear similarity prediction, avoiding the performance penalties that may be associated with batch contrastive learning.

[0069] The technical solution of this application includes two aspects: intra-sentence text semantic mining and cross-sentence text semantic mining. The details are as follows.

[0070] 1. Semantic mining of text within sentences

[0071] For a single sentence, design an encoder So that when applied to each sentence When , the encoded sentence satisfies certain theoretical information properties. Here, Θ represents the trainable parameters of the encoder, L i (X) and S j (X) represents the i-th relevant (common) feature and the j-th irrelevant (sensitive) feature of X. The architecture of the encoder is fixed, and its trainable parameters are optimized as follows:

[0072]

[0073] parameter It takes non-negative values ​​and is tuned to achieve a desired balance between competing objectives in the optimization problem, while the cardinality of Z determines the compression level of the encoder.

[0074] Among them, the first term in the formula is dedicated to information compression to maximize the mutual information between the compressed sentence and its original representation. Information-theoretic compression is superior to classical methods in which simple distortion measures are minimized (for example, the mean squared error after decompression). The second term in the formula is dedicated to utility, making the embedding more representative of the selected information by focusing adjustable attention on certain parts of the sentence content. One benefit of this scheme is that it can retain all key information in the case of inevitable loss of information. The last term in the formula is related to filtering out sensitive information from the encoded embedding to ensure that unauthorized users will not use this sensitive information when the data is released publicly (privacy), or to ensure that bias is not contained in the model trained using the encoded representation (fairness).

[0075] The above problems can be solved by setting the design parameters appropriately. When , the problem reduces to a task-independent information-theoretic compression. When γ = 0, the problem is task-oriented, essentially filtering out sensitive information from the dataset. Finally, when γ = 0, the task-oriented problem is a privacy-utility tradeoff, where sentences in the dataset are encoded so that they can be used to train models to recognize public information (for utility purposes). However, they are much less useful for training models to recognize certain sensitive information (for privacy purposes) for image domains with one public label and one private label.

[0076] The model's architecture and training process primarily consist of components: a sentence encoder and an information-theoretic evaluator. The trained encoder can be used on any dataset from the distribution used during training to tailor its information-theoretic properties; this can be used for a variety of applications, such as bandwidth-constrained transmission, private data sharing, or unbiased model development.

[0077] The encoder consists of a ready-made sentence embedding, which receives sentence tokens sequentially and outputs a vector representing the entire sentence. The sentence embeddings are fed into a neural network, which projects the original sentence embeddings into a low-dimensional embedding with certain information-theoretic properties. The neural network, the trainable part of the encoder, is designed using an information-theoretic estimator. This estimator consists of an estimator for the mutual information term in the optimization problem. Mutual information is defined between two random variables A and B according to the Donsker-Varadhan formula for mutual information, which is empirically estimated using samples drawn from the joint distribution. This formula states that mutual information can be obtained by finding a function that maximizes an expression that depends only on the first-order moment of the probability term. Based on this, the first-order moment can be empirically determined using the mean of the samples. Furthermore, the set of functions being searched can be modeled using a neural network, allowing any well-behaved function with appropriate weights to be modeled. Finally, optimization is performed using stochastic gradient descent, where at each step, the expectation and its gradient are estimated using a batch of samples from a public dataset. The maximum converged value then represents the mutual information.

[0078] 2. Cross-sentence text semantic mining

[0079] In order to mine the textual context information between sentences, the embedding representation space is decomposed into a routing network and a dedicated projector as follows.

[0080] 1) Routing network

[0081] The routing network can be used as a decomposed embedding space, representing the context as h x , [CLS] labels are introduced into the routing network. This additional information allows the routing mechanism to understand the relationship between sentence pairs (sent1, sent2). By incorporating global representations, the sentence pair input is element-wise additional Represent the routing network as a group classification from a linear layer G1(·), and It is based on Softmax probability The upper or lower range group of:

[0082]

[0083] Where j represents sent1 (j=1) or sent2 (j=2), and then x1 and sent2 h x2 To help the routing network classify the input, binary cross entropy (BCE) is used as shown below.

[0084]

[0085] Where y indicates whether the sentence pair is an upper bound or a lower bound. Use as part of overall learning objectives.

[0086] 2). Dedicated projector

[0087] The main goal of the dedicated projector is to transform the embedding representation h xj Decompose into upper or lower range subspaces. Previous work in embedding space decomposition has produced composite representations, where the final representation is obtained by computing the sum of vectors from multiple projectors. We find that this soft selection approach leads to overlap between subspaces, which is detrimental to model performance. Therefore, we derive a hard selection procedure where a dedicated projector has a separate projection for each class and only one projection is used for each representation. In particular, two dedicated projectors, an upper bound projector Upper(·) and a lower range projector Lower(·), represent h xj Mapped to the upper or lower range subspace as follows:

[0088]

[0089] where β j It is from get The highest probability, and h xj is the representation mapped to the upper range or lower range subspace. Using the output of different projectors, we obtain the representation pairs of upper range and lower range samples, h x1 and h x2 However, h xj is generated by the random weights of the specialized projectors. We need a way to improve the representation of the projectors.

[0090] One of the key parts of this method is to improve the representation z produced by the specialized projectors Upper(·) or Lower(·) xj A common approach is to apply supervised contrastive learning to pairwise representations. However, we find that batch contrastive learning hurts the performance of the projector because it requires combining representations from different projectors (Upper(·) and Lower(·)) in the same mini-batch. Therefore, a more appropriate learning objective is designed for the classification and ranking mechanism, which is to predict the linear similarity for each projector separately. Specifically, a linear layer G2 is used to transform z x1 With z x2 Connect them together, where the output of the linear is a regression number from zero (dissimilar) to one (similar). Then, use the BCE loss to minimize the difference between the output and ysim (STS score) as follows:

[0091]

[0092] Final training loss is an end-to-end example of representation learning and classification loss:

[0093]

[0094] The parameters α1 and α2 are the loss weights obtained by adjusting the development set.

[0095] 1. A method for intra-sentence text semantic mining: An encoder is designed to encode a single sentence so that the encoded sentence satisfies specific information-theoretic properties. The encoder contains trainable parameters that are adjusted through an optimization process to achieve goals such as information compression, utility maximization, and sensitive information filtering. The encoder architecture is fixed, but the parameters are trainable, and non-negative optimization is used to balance different objectives. By adjusting the design parameters, problems such as information-theoretic compression, task relevance, privacy, and fairness can be addressed. The encoder consists of a sentence embedding and a neural network that projects the sentence embedding into a low-dimensional embedding with specific information-theoretic properties. An information-theoretic estimator is used to estimate the mutual information, and optimization is performed via stochastic gradient descent.

[0096] 2. A method for cross-sentence text semantic mining: This paper explores the textual contextual information between sentences through a routing network and a dedicated projector. The routing network is used to decompose the embedding space, understand the relationship between sentence pairs, and perform classification through binary cross-entropy. The dedicated projector decomposes the embedding representation into an upper-range or lower-range subspace, using a hard selection process and only using one projection for each representation. To improve semantic understanding, supervised contrastive learning is adopted, but batch contrastive learning is avoided, and instead linear similarity prediction is performed for each projector. The final training loss combines representation learning and classification loss to form an end-to-end learning paradigm.

[0097] Thus, this application achieves efficient intra-sentence text semantic mining through innovative encoder design and information-theoretic optimization, while effectively mining cross-sentence text context information through the combination of routing network and dedicated projector. This technical solution not only improves the efficiency and quality of information compression, but also adapts to a variety of application scenarios such as bandwidth-limited transmission, private data sharing, and unbiased model development through flexible parameter adjustment. In addition, through the hard selection process and supervised contrastive learning, the present invention enhances the model's understanding of semantics and achieves effective training of representation learning and classification loss through end-to-end learning objectives.

[0098] Exemplary devices

[0099] Figure 2FIG is a structural diagram of a text semantic representation device based on information compression and space decomposition provided by an exemplary embodiment of the present invention. Figure 2 As shown, the apparatus 200 includes:

[0100] A first acquisition module 210 is configured to perform sentence semantic mining on the text to be represented and obtain a node vector representation of the text to be represented;

[0101] The second acquisition module 220 is used to perform cross-sentence text semantic mining on the node vector representation of the text to be represented, and obtain the sentence node vector representation of the text to be represented;

[0102] The determination module 230 is configured to determine the text semantic representation of the text to be represented based on the sentence node vector representation.

[0103] Optionally, the first acquisition module 210 includes:

[0104] Design an encoder for intra-sentence text semantic mining of a single sentence;

[0105] The intra-sentence text semantic mining encoder is used to perform intra-sentence text semantic mining on the text to be represented to obtain the node vector representation of the text to be represented.

[0106] Optionally, the training parameter optimization expression of the sentence text semantic mining encoder is:

[0107]

[0108] Where, represents mutual information, T Θ represents the encoder, Θ represents the trainable parameters of the encoder, L i (X) and S j (X) represents the i-th relevant feature and the j-th irrelevant feature of X, respectively. is the weight parameter, X represents a single sentence, Represents the text to be represented, and Ns represents the number of irrelevant features.

[0109] Optionally, the second acquisition module 220 includes:

[0110] Design a routing network for text decomposition embedding space;

[0111] Design a dedicated projector for each representation in the text factorization embedding space;

[0112] Based on the routing network and the dedicated projector, an embedding representation space decomposition model is constructed;

[0113] The embedding representation space decomposition model is used to perform cross-sentence text semantic mining on the node vector representation to obtain the sentence node vector representation of the text to be represented.

[0114] Optionally, using an embedding representation space decomposition model to perform cross-sentence text semantic mining on the node vector representation to obtain the sentence node vector representation of the text to be represented, including:

[0115] A routing network is used to represent the context of the node vector as an embedding space vector.

[0116] Append the classification labels to the context embedding space vectors to obtain additional vectors;

[0117] A dedicated projector is used to perform spatial decomposition on the additional vectors to obtain the sentence node vector representation of the text to be represented.

[0118] Optionally, an additional vector is based on the Softmax probability The upper or lower range group of:

[0119]

[0120] Where G1(·) is the group classification of the linear layer; j = 1 or 2 is used to represent sentence 1 or sentence 2; is an additional vector; h xj is the embedding space vector;

[0121] The loss function of the routing network is:

[0122]

[0123] Where BCE is the binary cross entropy loss; y indicates whether the sentence pair is an upper bound or a lower bound.

[0124] Optionally, the expression for a specialized projector is:

[0125]

[0126] Where, β j From max get The highest probability of z xj is a representation mapped to the upper or lower range subspace; j = 1 or 2 is used to represent sentence 1 or sentence 2;

[0127] The loss function of the dedicated projector is:

[0128]

[0129] Where y sim is the STS score; G2(concat(zx1 ,z x2 )) indicates that the linear layer G2 converts z x1 With z x2 Connect them; BCE is the binary cross entropy loss.

[0130] Optionally, the loss function for the embedding representation space decomposition model is:

[0131]

[0132] Where α1 and α2 are the loss weights obtained by adjusting the development set.

[0133] Exemplary electronic devices

[0134] Figure 3 This is the structure of an electronic device provided by an exemplary embodiment of the present invention. Figure 3 As shown, the electronic device 30 includes one or more processors 31 and a memory 32 .

[0135] The processor 31 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0136] The memory 32 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may, for example, include read-only memory (ROM), a hard disk, a flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 31 may execute the program instructions to implement the methods of the software programs of the various embodiments of the present invention described above and / or other desired functions. In one example, the electronic device may further include: an input device 33 and an output device 34, which are interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0137] In addition, the input device 33 may also include, for example, a keyboard, a mouse, and the like.

[0138] The output device 34 can output various information to the outside. The output device 34 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto.

[0139] Of course, to simplify, Figure 3Only some of the components related to the present invention in the electronic device are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, the electronic device may further include any other appropriate components according to specific application conditions.

[0140] Exemplary computer program products and computer-readable storage media

[0141] In addition to the above-mentioned methods and devices, an embodiment of the present invention may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to perform the steps of the method according to various embodiments of the present invention described in the above "Exemplary Method" section of this specification.

[0142] The computer program product may be written in any combination of one or more programming languages ​​to implement the operations of embodiments of the present invention, including object-oriented programming languages ​​such as Java, C++, and conventional procedural programming languages ​​such as C or similar programming languages. The program code may be executed entirely on the user's computing device, partially on the user's computing device, as a stand-alone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0143] In addition, an embodiment of the present invention may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enable the processor to execute the steps of the method according to various embodiments of the present invention described in the above "Exemplary Method" section of this specification.

[0144] The computer-readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can, for example, include but is not limited to a system, system or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof.

[0145] The basic principles of the present invention have been described above in conjunction with specific embodiments. However, it should be noted that the advantages, strengths, and effects mentioned in the present invention are merely illustrative and non-limiting, and should not be construed as necessarily possessed by each embodiment of the present invention. Furthermore, the specific details disclosed above are provided for illustrative purposes and to facilitate understanding, and are not intended to be limiting. These details do not necessarily limit the present invention to being implemented using these specific details.

[0146] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. References to the same or similar parts between the various embodiments are sufficient. For system embodiments, since they are generally similar to the method embodiments, their description is relatively simple. For relevant parts, references to the description of the method embodiments are sufficient.

[0147] The block diagrams of the devices, systems, equipment, and systems involved in the present invention are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As will be appreciated by those skilled in the art, these devices, systems, equipment, and systems can be connected, arranged, or configured in any manner. Words such as "including," "comprising," "having," and the like are open-ended words, meaning "including but not limited to," and can be used interchangeably therewith. The words "or" and "and" used herein refer to the words "and / or" and can be used interchangeably therewith, unless the context clearly indicates otherwise. The word "such as" used herein refers to the phrase "such as but not limited to," and can be used interchangeably therewith.

[0148] The method and system of the present invention may be implemented in many ways. For example, the method and system of the present invention may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above sequence of steps for the method is for illustration only, and the steps of the method of the present invention are not limited to the sequence specifically described above, unless otherwise specified. In addition, in some embodiments, the present invention may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present invention. Thus, the present invention also covers recording media that store programs for executing the method according to the present invention.

[0149] It should also be noted that, in the system, device and method of the present invention, each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent schemes of the present invention. The above description of the disclosed aspects is provided to enable any technician in this field to make or use the present invention. Various modifications to these aspects will be very obvious to those skilled in the art, and the general principles defined here can be applied to other aspects without departing from the scope of the present invention. Therefore, the present invention is not intended to be limited to the aspects shown here, but according to the widest scope consistent with the principles disclosed here and novel features.

[0150] The above description has been presented for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of the present invention to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A text semantic representation method based on information compression and spatial decomposition, characterized in that: include: Performing sentence semantic mining on the text to be represented to obtain a node vector representation of the text to be represented; Performing cross-sentence text semantic mining on the node vector representation of the text to be represented to obtain a sentence node vector representation of the text to be represented; Determining a text semantic representation of the text to be represented based on the sentence node vector representation; Performing cross-sentence text semantic mining on the node vector representation of the text to be represented to obtain the sentence node vector representation of the text to be represented, including: Design a routing network for text decomposition embedding space, which is used to decompose the embedding space, understand the relationship between sentence pairs, and perform classification through binary cross entropy; Design a dedicated projector for each representation in the text decomposition embedding space, which decomposes the embedding representation into an upper range or lower range subspace, using a hard selection process to use only one projection per representation; Constructing an embedding representation space decomposition model based on the routing network and the dedicated projector; The embedding representation space decomposition model is used to perform cross-sentence text semantic mining on the node vector representation to obtain the sentence node vector representation of the text to be represented.

2. The method according to claim 1, characterized in that Performing sentence semantic mining on the text to be represented to obtain a node vector representation of the text to be represented includes: Design an encoder for intra-sentence text semantic mining of a single sentence; The intra-sentence text semantic mining encoder is used to perform intra-sentence text semantic mining on the text to be represented to obtain the node vector representation of the text to be represented.

3. The method according to claim 2, characterized in that The training parameter optimization expression of the text semantic mining encoder in the sentence is: Where, represents mutual information, T Θ represents the encoder, Θ represents the trainable parameters of the encoder, L i (X) and S j (X) represents the i-th relevant feature and the j-th irrelevant feature of X, respectively. is the weight parameter, X represents a single sentence, Represents the text to be represented, N s Represents the number of irrelevant features, N l Indicates the number of relevant features.

4. The method according to claim 1, wherein Using the embedding representation space decomposition model to perform cross-sentence text semantic mining on the node vector representation to obtain the sentence node vector representation of the text to be represented, including: Using the routing network, the contexts of the node vectors are represented as embedding space vectors respectively; Appending the classification labels to the embedding space vectors of the context to obtain additional vectors; The dedicated projector is used to perform spatial decomposition on the additional vector to obtain the sentence node vector representation of the text to be represented.

5. The method according to claim 4, characterized in that The additional vector is based on the Softmax probability The upper or lower range group of: Where G1(·) is the group classification of the linear layer; j = 1 or 2 is used to represent sentence 1 or sentence 2; is an additional vector; h xj is the embedding space vector; The loss function of the routing network is: Where BCE is the binary cross entropy loss; Indicates whether the sentence pair is an upper bound or a lower bound.

6. The method according to claim 5, characterized in that The expression of the dedicated projector is: Where, β j It is from get The highest probability of z xj is a representation mapped to the upper or lower range subspace; j = 1 or 2 is used to represent sentence 1 or sentence 2; The loss function of the dedicated projector is: Where y sim is the STS score; G2(concat(z x1 ,z x2 )) indicates that the linear layer G2 converts z x1 With z x2 Connect them; BCE is the binary cross entropy loss.

7. The method according to claim 6, characterized in that The loss function of the embedding representation space decomposition model is: Where α1 and α2 are the loss weights obtained by adjusting the development set.

8. A text semantic representation device based on information compression and spatial decomposition, characterized in that: include: A first acquisition module is used to perform sentence semantic mining on the text to be represented and obtain a node vector representation of the text to be represented; A second acquisition module is used to perform cross-sentence text semantic mining on the node vector representation of the text to be represented, and obtain the sentence node vector representation of the text to be represented; A determination module, configured to determine a text semantic representation of the text to be represented based on the sentence node vector representation; The second acquisition module includes: Design a routing network for text decomposition embedding space, which is used to decompose the embedding space, understand the relationship between sentence pairs, and perform classification through binary cross entropy; Design a dedicated projector for each representation in the text decomposition embedding space, which decomposes the embedding representation into an upper range or lower range subspace, using a hard selection process to use only one projection per representation; Constructing an embedding representation space decomposition model based on the routing network and the dedicated projector; The embedding representation space decomposition model is used to perform cross-sentence text semantic mining on the node vector representation to obtain the sentence node vector representation of the text to be represented.

9. A computer-readable storage medium, characterized in that The storage medium stores a computer program, and the computer program is used to execute the method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor; The processor is configured to read the executable instructions from the memory and execute the instructions to implement the method according to any one of claims 1 to 7.

Images