Text embedding vector generation method and device, computer device and storage medium
By fusing the guiding signal matrix and sparse representation vector in the pre-trained language model, the problem of inaccurate keyword representation by text embedding vectors in existing technologies is solved, thereby improving the accuracy of text embedding vectors.
Patent Information
- Application Number
- CN202610553143.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-04-24
- Publication Date
- 2026-08-25
AI Technical Summary
Existing pre-trained language models struggle to accurately represent keywords and technical terms when generating text embedding vectors, resulting in insufficient accuracy of the text embedding vectors.
By obtaining the text segmentation sequence of the text to be processed and matching it with the domain keyword dictionary, a guiding signal matrix is constructed. This matrix is then used to guide the generation of dense representation vectors in a pre-trained language model. Finally, sparse representation vectors are combined to perform semantic smoothing and fusion, resulting in text embedding vectors.
It improves the accuracy and sensitivity of capturing keywords and technical terms, enhances the semantic coverage and recall of synonyms and low-frequency words, and improves the accuracy of text embedding vectors.
Smart Images
Figure CN122635339A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, computer device, and storage medium for generating text embedding vectors. Background Technology
[0002] Text embedding is a technique that maps text data (words, sentences, paragraphs, etc.) into fixed-length real vectors, so that semantically similar texts are close in distance in the vector space.
[0003] In related technologies, pre-trained language models are used to generate text embedding vectors. Because this model uses a fully intrinsic self-attention mechanism to model the contextual relationships of the input text, the generated text embedding vectors are not accurate in representing keywords, entities, or technical terms. Summary of the Invention
[0004] Therefore, it is necessary to provide a text embedding vector generation method, apparatus, computer device, and computer-readable storage medium that can improve the accuracy of text embedding vectors, in order to address the above-mentioned technical problems.
[0005] Firstly, this application provides a method for generating text embedding vectors, the method comprising:
[0006] Obtain the text segmentation sequence of the text to be processed, match each segment of the text segmentation sequence with each keyword in the preset domain keyword dictionary, and obtain the first sparse representation vector; the domain keyword dictionary includes each keyword and the semantic weight of each keyword, and the non-zero dimension of the first sparse representation vector corresponds to the semantic weight of the successfully matched keyword.
[0007] Based on the first sparse representation vector, construct the guiding signal matrix corresponding to the text segmentation sequence;
[0008] The text segmentation sequence and the guiding signal matrix are input into a pre-trained language model to obtain a dense representation vector;
[0009] Based on the dense representation vector, semantic smoothing is performed on the first sparse representation vector to obtain the second sparse representation vector.
[0010] The dense representation vector and the second sparse representation vector are fused to obtain the text embedding vector corresponding to the text to be processed.
[0011] In one embodiment, each segment of the text segmentation sequence is matched with each keyword of a preset domain keyword dictionary to obtain a first sparse representation vector, including:
[0012] Obtain the initial sparse representation vector; the number of dimensions of the initial sparse representation vector is the same as the size of the domain keyword dictionary, and the dimension value of each dimension of the initial sparse representation vector is zero;
[0013] Each segment of the text segmentation sequence is matched with each keyword in the preset domain keyword dictionary, and the semantic weight of the successfully matched keywords is extracted from the domain keyword dictionary.
[0014] Based on the order of keywords in the domain keyword dictionary, the semantic weights of the successfully matched keywords are filled into the corresponding dimensions of the initial sparse representation vector to obtain the first sparse representation vector.
[0015] In one embodiment, before filling the semantic weights of the successfully matched keywords into the corresponding dimensions of the initial sparse representation vector according to the order of the keywords in the domain keyword dictionary to obtain the first sparse representation vector, the method further includes:
[0016] Obtain the word frequency of each word in the text segmentation sequence;
[0017] The semantic weight of the successfully matched keywords is adjusted based on the word frequency of the word segments corresponding to the successfully matched keywords.
[0018] In one embodiment, a guiding signal matrix corresponding to the text segmentation sequence is constructed based on the first sparse representation vector, including:
[0019] Obtain a set of target keywords associated with each word in the text segmentation sequence from the domain keyword dictionary; the keyword set includes at least one target keyword associated with each word segmentation.
[0020] Obtain the semantic weights of the target keywords associated with each segment in the text segmentation sequence from the first sparse representation vector;
[0021] Based on the semantic weights of the target keywords associated with each segmented word, the element values of each matrix element in the guiding signal matrix are obtained; the number of rows and columns of the guiding signal matrix are the same as the sequence length of the text segmentation sequence.
[0022] In one embodiment, semantic smoothing is performed on the first sparse representation vector based on the dense representation vector to obtain the second sparse representation vector, including:
[0023] Calculate the semantic relevance between the dense representation vector and each keyword in the domain keyword dictionary;
[0024] Based on semantic relevance, the dimension values in the first sparse representation vector are adjusted to obtain the second sparse representation vector.
[0025] In one embodiment, the dense representation vector and the second sparse representation vector are fused to obtain the text embedding vector corresponding to the text to be processed, including:
[0026] The second sparse representation vector is projected to the same dimension as the dense representation vector to obtain the third sparse representation vector.
[0027] The third sparse representation vector is concatenated with the dense representation vector to obtain the concatenated vector;
[0028] The concatenated vector is input into the fully connected layer to obtain the text embedding vector corresponding to the text to be processed.
[0029] In one embodiment, the dense representation vector and the second sparse representation vector are fused to obtain the text embedding vector corresponding to the text to be processed, including:
[0030] Based on the preset gating network, generate the first gating weight corresponding to the dense representation vector and the second gating weight corresponding to the second sparse representation vector;
[0031] Based on the first and second gating weights, the dense representation vector and the second sparse representation vector are weighted and summed to obtain the text embedding vector corresponding to the text to be processed.
[0032] Secondly, this application also provides a text embedding vector generation apparatus, the apparatus comprising:
[0033] The first sparse representation vector acquisition module is used to obtain the text segmentation sequence of the text to be processed, match each segment of the text segmentation sequence with each keyword in the preset domain keyword dictionary, and obtain the first sparse representation vector. The domain keyword dictionary includes each keyword and the semantic weight of each keyword. The non-zero dimension of the first sparse representation vector corresponds to the semantic weight of the successfully matched keyword.
[0034] The guiding signal matrix construction module is used to construct the guiding signal matrix corresponding to the text segmentation sequence based on the first sparse representation vector.
[0035] The dense representation vector acquisition module is used to input the text segmentation sequence and the guiding signal matrix into the pre-trained language model to obtain dense representation vectors.
[0036] The second sparse representation vector acquisition module is used to perform semantic smoothing on the first sparse representation vector based on the dense representation vector to obtain the second sparse representation vector.
[0037] The text embedding vector acquisition module is used to fuse the dense representation vector and the second sparse representation vector to obtain the text embedding vector corresponding to the text to be processed.
[0038] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method steps of the first aspect.
[0039] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method steps of the first aspect.
[0040] The aforementioned text embedding vector generation method, apparatus, computer device, and computer-readable storage medium obtain a text segmentation sequence of the text to be processed, match each segment of the text segmentation sequence with each keyword in a pre-defined domain keyword dictionary to obtain a first sparse representation vector. The domain keyword dictionary includes each keyword and its semantic weight. The non-zero dimension of the first sparse representation vector corresponds to the semantic weight of the successfully matched keyword. Based on the first sparse representation vector, a guiding signal matrix corresponding to the text segmentation sequence is constructed. The text segmentation sequence and the guiding signal matrix are input into a pre-trained language model to obtain a dense representation vector. Based on the dense representation vector, the first sparse representation vector is semantically smoothed to obtain a second sparse representation vector. The dense representation vector and the second sparse representation vector are fused to obtain the text embedding vector corresponding to the text to be processed. As can be seen from the above, by generating the first sparse representation vector through the domain keyword dictionary and using the constructed guiding signal matrix to provide real-time guidance during the dense semantic encoding process of the pre-trained language model, the generated dense representation vector is injected with key entity information from the source, thereby improving the accuracy and sensitivity of capturing keywords, entities, or technical terms. Based on the dense representation vector, semantic smoothing is applied to the first sparse representation vector. This assigns semantic weights to semantically related words (such as synonyms) that do not appear in the text, enhancing the semantic coverage and recall capabilities for synonyms and low-frequency words. The dense and second sparse representation vectors are then fused to obtain the text embedding vector corresponding to the text to be processed. This combination of the precise matching capability of sparse representation and the deep semantic representation capability of dense representation improves the accuracy of the text embedding vector. Attached Figure Description
[0041] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.
[0042] Figure 1 This is an application environment diagram of the text embedding vector generation method in one embodiment;
[0043] Figure 2 This is a flowchart illustrating a text embedding vector generation method in one embodiment;
[0044] Figure 3 This is a schematic diagram of the process for obtaining the first sparse representation vector in one embodiment;
[0045] Figure 4 This is a structural block diagram of a text embedding vector generation device in one embodiment;
[0046] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation
[0047] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0048] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.
[0049] The text embedding vector generation method provided in this application embodiment can be applied to, for example... Figure 1In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. Terminal 102 acquires the text segmentation sequence of the text to be processed, matches each segment of the text segmentation sequence with each keyword in a preset domain keyword dictionary to obtain a first sparse representation vector; the domain keyword dictionary includes each keyword and its semantic weight, and the non-zero dimension of the first sparse representation vector corresponds to the semantic weight of the successfully matched keyword; based on the first sparse representation vector, a guiding signal matrix corresponding to the text segmentation sequence is constructed; the text segmentation sequence and the guiding signal matrix are input into a pre-trained language model to obtain a dense representation vector; based on the dense representation vector, the first sparse representation vector is semantically smoothed to obtain a second sparse representation vector; the dense representation vector and the second sparse representation vector are fused to obtain the text embedding vector corresponding to the text to be processed. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, drones, low-altitude aircraft, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, and projection equipment. Portable wearable devices can include smartwatches, smart bracelets, and head-mounted displays. Head-mounted displays can be virtual reality (VR) devices, augmented reality (AR) devices, and smart glasses. Server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.
[0050] In one embodiment, such as Figure 2 As shown, a text embedding vector generation method is provided. This embodiment applies this method to... Figure 1 Taking terminal 102 as an example, the method includes the following steps:
[0051] Step S210: Obtain the text segmentation sequence of the text to be processed, match each segment of the text segmentation sequence with each keyword in the preset domain keyword dictionary, and obtain the first sparse representation vector; the domain keyword dictionary includes each keyword and the semantic weight of each keyword, and the non-zero dimension of the first sparse representation vector corresponds to the semantic weight of the successfully matched keyword.
[0052] Each keyword in the pre-defined domain keyword dictionary is associated with a predefined semantic weight, which can be derived based on domain expert knowledge, the IDF value of the vocabulary, or its importance in the domain corpus. In practical applications, a pre-constructed domain keyword dictionary can be designed for specific downstream tasks (such as patent retrieval and medical literature analysis).
[0053] Sparse representation is a vector representation method with typically high dimensionality, but the vast majority of its elements are zero, with only a few elements having non-zero values. It is often used to accurately reflect the presence of specific keywords.
[0054] The first sparse representation vector has the same number of dimensions as the domain keyword dictionary. The dimensions of each element in the first sparse representation vector include both zero and non-zero values. The non-zero values in the first sparse representation vector correspond to the semantic weights of successfully matched keywords.
[0055] In this embodiment, the text to be processed undergoes standardization processes such as cleaning (e.g., removing irrelevant characters) and word segmentation to obtain a text segmentation sequence. The presence of each word segment from the text segmentation sequence in a preset domain keyword dictionary is detected, and keywords matching the words in the text segmentation sequence are identified as successfully matched keywords. The semantic weights of the successfully matched keywords in the domain keyword dictionary are obtained, and a first sparse representation vector is obtained based on these semantic weights.
[0056] Step S220: Construct the guiding signal matrix corresponding to the text segmentation sequence based on the first sparse representation vector.
[0057] The number of rows and columns in the guiding signal matrix corresponds to the sequence length of the text segmentation sequence. Specifically, if the sequence length of the text segmentation sequence is L, then the number of rows and columns in the guiding signal matrix is also L.
[0058] In this embodiment of the application, for the i-th word (as the attention query object) and the j-th word (as the attention object) in the text segmentation sequence, the value of the guiding signal matrix Guide(S)_ij reflects the prior influence of the semantic weights of all keywords associated with the j-th word on the attention of the i-th word.
[0059] Step S230: Input the text segmentation sequence and the guiding signal matrix into the pre-trained language model to obtain a dense representation vector.
[0060] Among them, pre-trained language models are based on the Transformer architecture, including but not limited to BERT and Sentence-BERT models. With its core "self-attention mechanism" as its core, the pre-trained language model can dynamically model the global contextual relationships of the input text, thereby generating low-dimensional and dense context embedding vectors.
[0061] Dense representation is a vector representation method with relatively low dimensionality, where all or most elements are non-zero real numbers. It is often used to capture complex semantic and contextual information.
[0062] In this embodiment, in the self-attention calculation module of the pre-trained language model, after completing the query-key dot product calculation and before the Softmax normalization operation, an external signal injection interface is added to receive an external guiding signal matrix. The text segmentation sequence is input to the encoder of the pre-trained language model. When the encoded data flows through the self-attention layer configured with the external signal injection interface, the guiding signal matrix is injected in real time, performing externally guided attention calculation. The encoder ultimately outputs a dense representation vector guided by keyword information from the very beginning of its generation. Specifically, the corrected attention is... .in, It is about balancing the original attention with the guiding signal matrix. The scaling factor (which can be fixed or learnable).
[0063] Step S240: Based on the dense representation vector, perform semantic smoothing on the first sparse representation vector to obtain the second sparse representation vector.
[0064] In this embodiment of the application, the semantic information contained in the dense representation vector is used to calculate its semantic relevance with each keyword in the domain keyword dictionary, and the first sparse representation vector S is semantically smoothed (for example, by assigning small semantic weights to synonyms with high relevance) to generate a semantically enhanced sparse representation vector (the second sparse representation vector).
[0065] Step S250: The dense representation vector and the second sparse representation vector are fused to obtain the text embedding vector corresponding to the text to be processed.
[0066] In this embodiment of the application, the dense representation vector and the second sparse representation vector can be concatenated or weighted and summed to obtain the text embedding vector corresponding to the text to be processed.
[0067] The aforementioned text embedding vector generation method obtains the text segmentation sequence of the text to be processed, matches each segment with each keyword in a pre-defined domain keyword dictionary to obtain a first sparse representation vector. The domain keyword dictionary includes each keyword and its semantic weight; the non-zero dimension of the first sparse representation vector corresponds to the semantic weight of the successfully matched keyword. Based on the first sparse representation vector, a guiding signal matrix corresponding to the text segmentation sequence is constructed. The text segmentation sequence and the guiding signal matrix are input into a pre-trained language model to obtain a dense representation vector. Based on the dense representation vector, the first sparse representation vector is semantically smoothed to obtain a second sparse representation vector. The dense representation vector and the second sparse representation vector are fused to obtain the text embedding vector corresponding to the text to be processed. As can be seen from the above, by generating the first sparse representation vector through the domain keyword dictionary and using the constructed guiding signal matrix to provide real-time guidance during the dense semantic encoding process in the pre-trained language model, the generated dense representation vector is injected with key entity information from the source, thereby improving the accuracy and sensitivity of capturing keywords, entities, or technical terms. Based on the dense representation vector, semantic smoothing is applied to the first sparse representation vector. This process assigns weights to semantically related words (such as synonyms) that do not appear in the text, enhancing the semantic coverage and recall capabilities for synonyms and low-frequency words. The dense and second sparse representation vectors are then fused to obtain the text embedding vector corresponding to the text to be processed. This combination of the precise matching capability of sparse representation and the deep semantic representation capability of dense representation improves the accuracy of the text embedding vector.
[0068] In one embodiment, such as Figure 3 As shown, each segment of the text word segmentation sequence is matched with each keyword in a pre-defined domain keyword dictionary to obtain the first sparse representation vector, including:
[0069] Step S310: Obtain the initial sparse representation vector; the number of dimensions of the initial sparse representation vector is the same as the size of the domain keyword dictionary, and the dimension value of each dimension of the initial sparse representation vector is zero.
[0070] In this embodiment of the application, taking the size of the domain keyword dictionary as V as an example, the dimension values of the V dimensions of the initial sparse representation vector are all 0.
[0071] Step S320: Match each segment of the text segmentation sequence with each keyword in the preset domain keyword dictionary, and extract the semantic weight of the successfully matched keywords from the domain keyword dictionary.
[0072] Step S330: According to the order of each keyword in the domain keyword dictionary, fill the semantic weight of the successfully matched keyword into the corresponding dimension of the initial sparse representation vector to obtain the first sparse representation vector.
[0073] In this embodiment of the application, taking the text segmentation sequence as [artificial intelligence, algorithm, deep learning] and the domain keyword dictionary as {"artificial intelligence": 0.8, "neural network model": 0.5, "machine learning": 0.9, "deep learning": 0.7} as an example, the successfully matched keywords are "artificial intelligence" and "deep learning", with corresponding semantic weights of 0.8 and 0.7, and the first sparse representation vector is [0.8, 0, 0, 0.7].
[0074] In one embodiment, before filling the semantic weights of the successfully matched keywords into the corresponding dimensions of the initial sparse representation vector according to the order of the keywords in the domain keyword dictionary to obtain the first sparse representation vector, the method further includes:
[0075] Step S332: Obtain the word frequency of each word in the text segmentation sequence;
[0076] Step S334: Adjust the semantic weight of the successfully matched keywords based on the word frequency of the word segments corresponding to the successfully matched keywords.
[0077] In this embodiment of the application, the formula for adjusting semantic weights is: .in, The semantic weight of successfully matched keywords, Let be the word frequency of the keyword in the text segmentation sequence, and log represent the logarithmic function.
[0078] The embodiments of this application adjust the semantic weights of successfully matched keywords based on word frequency, which can improve the accuracy of the first sparse representation vector.
[0079] In one embodiment, a guiding signal matrix corresponding to the text segmentation sequence is constructed based on the first sparse representation vector, including:
[0080] Step S222: Obtain the set of target keywords associated with each segment in the text segmentation sequence from the domain keyword dictionary; the keyword set includes at least one target keyword associated with each segment.
[0081] In this embodiment, a set of target keywords associated with each segment in the text segmentation sequence is obtained through precise matching, part-of-speech tagging, sub-word matching, or semantic similarity. Precise matching refers to segmenting words that are exactly equal to keywords in the domain keyword dictionary. Part-of-speech tagging involves restoring the word form of the segment (e.g., "running" → "run") before matching. Sub-word matching uses BPE or WordPiece segmentation to match sub-word units (e.g., "unhappiness" → ["un", "happiness"]). Semantic similarity involves calculating the cosine similarity between the segment and the keyword using word vectors (e.g., GloVe) and then filtering by a threshold (e.g., cosine similarity > 0.8).
[0082] Step S224: Obtain the semantic weights of the target keywords associated with each segment in the text segmentation sequence from the first sparse representation vector;
[0083] Step S226: Based on the semantic weights of the target keywords associated with each segmented word, obtain the element values of each matrix element in the guiding signal matrix; the number of rows and columns of the guiding signal matrix are the same as the sequence length of the text segmentation sequence.
[0084] In this embodiment of the application, the expression for the element value of each matrix element in the guiding signal matrix is as follows: Where Keywords(j) represents the set of target keywords associated with the j-th word segment. It is the semantic weight of the k-th target keyword in the first sparse representation vector. It is a global scaling factor (which can be fixed or learned).
[0085] In one embodiment, semantic smoothing is performed on a first sparse representation vector based on a dense representation vector to obtain a second sparse representation vector, including:
[0086] Step S242: Calculate the semantic relevance between the dense representation vector and each keyword in the domain keyword dictionary.
[0087] In dense representation vectors, each dimension typically does not directly correspond to interpretable features, but rather implicitly encodes semantic information through the training process. For example, dense vectors of similar words (such as "car" and "automobile") are close in distance in the vector space (high cosine similarity), while vectors of antonyms (such as "good" and "bad") may be in opposite directions but located in the same semantic region. Therefore, although the numerical values of dense vectors themselves have no direct semantic meaning, their overall distribution reflects the semantic relationships between words.
[0088] In this embodiment, a pre-trained word embedding model is used to convert each keyword in the domain keyword dictionary into a corresponding dense vector. The cosine similarity (semantic relevance) between the dense representation vector and the dense vector of each keyword in the domain keyword dictionary is calculated.
[0089] Step S244: Adjust the dimension values in the first sparse representation vector according to the semantic relevance to obtain the second sparse representation vector.
[0090] In this embodiment, the second sparse representation vector is obtained by multiplying the dimension value of each dimension in the first sparse representation vector with the semantic relevance corresponding to each dimension. Here, the semantic relevance corresponding to each dimension refers to the semantic relevance between the dense vector of the keyword corresponding to that dimension and the dense representation vector.
[0091] In one embodiment, the dense representation vector and the second sparse representation vector are fused to obtain the text embedding vector corresponding to the text to be processed, including:
[0092] Step S251: Project the second sparse representation vector to the same dimension as the dense representation vector to obtain the third sparse representation vector;
[0093] Step S252: Concatenate the third sparse representation vector with the dense representation vector to obtain the concatenated vector;
[0094] Step S253: Input the concatenated vector into the fully connected layer to obtain the text embedding vector corresponding to the text to be processed.
[0095] In this embodiment, the second sparse representation vector is mapped to the same dimension as the dense representation vector through a linear layer to obtain a third sparse representation vector. The third sparse representation vector and the dense representation vector are concatenated along the feature dimension to obtain a concatenated vector. A fully connected layer is then used to perform feature mapping on the concatenated vector to obtain the text embedding vector corresponding to the text to be processed.
[0096] In one embodiment, the dense representation vector and the second sparse representation vector are fused to obtain the text embedding vector corresponding to the text to be processed, including:
[0097] Step S254: Based on the preset gating network, generate the first gating weight corresponding to the dense representation vector and the second gating weight corresponding to the second sparse representation vector.
[0098] Step S255: Based on the first gating weight and the second gating weight, perform a weighted summation on the dense representation vector and the second sparse representation vector to obtain the text embedding vector corresponding to the text to be processed.
[0099] In this embodiment, the first gating weight is multiplied by the dense representation vector to obtain a first product result. The second gating weight is multiplied by the second sparse representation vector to obtain a second product result. The first product result and the second product result are added together to obtain the text embedding vector corresponding to the text to be processed.
[0100] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.
[0101] Based on the same inventive concept, this application also provides a text embedding vector generation apparatus for implementing the text embedding vector generation method described above. The solution provided by this apparatus is similar to the implementation described in the above method; therefore, the specific limitations in one or more text embedding vector generation apparatus embodiments provided below can be found in the limitations of the text embedding vector generation method described above, and will not be repeated here.
[0102] In one exemplary embodiment, please refer to Figure 4 A text embedding vector generation apparatus is provided, the apparatus comprising:
[0103] The first sparse representation vector acquisition module 410 is used to acquire the text segmentation sequence of the text to be processed, match each segment of the text segmentation sequence with each keyword in the preset domain keyword dictionary, and obtain the first sparse representation vector; the domain keyword dictionary includes each keyword and the semantic weight of each keyword, and the non-zero dimension of the first sparse representation vector corresponds to the semantic weight of the successfully matched keyword.
[0104] The guiding signal matrix construction module 420 is used to construct the guiding signal matrix corresponding to the text segmentation sequence based on the first sparse representation vector;
[0105] The dense representation vector acquisition module 430 is used to input the text segmentation sequence and the guiding signal matrix into the pre-trained language model to obtain the dense representation vector;
[0106] The second sparse representation vector acquisition module 440 is used to perform semantic smoothing processing on the first sparse representation vector based on the dense representation vector to obtain the second sparse representation vector.
[0107] The text embedding vector acquisition module 450 is used to fuse the dense representation vector and the second sparse representation vector to obtain the text embedding vector corresponding to the text to be processed.
[0108] In one embodiment, each segment of the text segmentation sequence is matched with each keyword of a preset domain keyword dictionary to obtain a first sparse representation vector, including:
[0109] Obtain the initial sparse representation vector; the number of dimensions of the initial sparse representation vector is the same as the size of the domain keyword dictionary, and the dimension value of each dimension of the initial sparse representation vector is zero;
[0110] Each segment of the text segmentation sequence is matched with each keyword in the preset domain keyword dictionary, and the semantic weight of the successfully matched keywords is extracted from the domain keyword dictionary.
[0111] Based on the order of keywords in the domain keyword dictionary, the semantic weights of the successfully matched keywords are filled into the corresponding dimensions of the initial sparse representation vector to obtain the first sparse representation vector.
[0112] In one embodiment, before filling the semantic weights of the successfully matched keywords into the corresponding dimensions of the initial sparse representation vector according to the order of the keywords in the domain keyword dictionary to obtain the first sparse representation vector, the method further includes:
[0113] Obtain the word frequency of each word in the text segmentation sequence;
[0114] The semantic weight of the successfully matched keywords is adjusted based on the word frequency of the word segments corresponding to the successfully matched keywords.
[0115] In one embodiment, a guiding signal matrix corresponding to the text segmentation sequence is constructed based on the first sparse representation vector, including:
[0116] Obtain a set of target keywords associated with each word in the text segmentation sequence from the domain keyword dictionary; the keyword set includes at least one target keyword associated with each word segmentation.
[0117] Obtain the semantic weights of the target keywords associated with each segment in the text segmentation sequence from the first sparse representation vector;
[0118] Based on the semantic weights of the target keywords associated with each segmented word, the element values of each matrix element in the guiding signal matrix are obtained; the number of rows and columns of the guiding signal matrix are the same as the sequence length of the text segmentation sequence.
[0119] In one embodiment, semantic smoothing is performed on the first sparse representation vector based on the dense representation vector to obtain the second sparse representation vector, including:
[0120] Calculate the semantic relevance between the dense representation vector and each keyword in the domain keyword dictionary;
[0121] Based on semantic relevance, the dimension values in the first sparse representation vector are adjusted to obtain the second sparse representation vector.
[0122] In one embodiment, the dense representation vector and the second sparse representation vector are fused to obtain the text embedding vector corresponding to the text to be processed, including:
[0123] The second sparse representation vector is projected to the same dimension as the dense representation vector to obtain the third sparse representation vector.
[0124] The third sparse representation vector is concatenated with the dense representation vector to obtain the concatenated vector;
[0125] The concatenated vector is input into the fully connected layer to obtain the text embedding vector corresponding to the text to be processed.
[0126] In one embodiment, the dense representation vector and the second sparse representation vector are fused to obtain the text embedding vector corresponding to the text to be processed, including:
[0127] Based on the preset gating network, generate the first gating weight corresponding to the dense representation vector and the second gating weight corresponding to the second sparse representation vector;
[0128] Based on the first and second gating weights, the dense representation vector and the second sparse representation vector are weighted and summed to obtain the text embedding vector corresponding to the text to be processed.
[0129] Each module in the aforementioned text embedding vector generation device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.
[0130] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 5As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When executed by the processor, the computer program implements a text embedding vector generation method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.
[0131] Those skilled in the art will understand that Figure 5 The structures shown are merely block diagrams of some structures related to the present application and do not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than shown in the figures, or combine certain components, or have different component arrangements. In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, which, when executed by the processor, causes the processor to perform the steps of the text embedding vector generation method described above. The steps of the text embedding vector generation method described here may be steps from one of the text embedding vector generation methods in the various embodiments described above.
[0132] In one embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, causes the processor to perform the steps of the text embedding vector generation method described above. The steps of the text embedding vector generation method described here can be steps from one of the text embedding vector generation methods in the various embodiments described above.
[0133] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, causes the processor to perform the steps of the text embedding vector generation method described above. The steps of the text embedding vector generation method described here may be steps from one of the text embedding vector generation methods in the various embodiments described above.
[0134] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.
[0135] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.
[0136] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.
[0137] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.
Claims
1. A method for generating text embedding vectors, characterized in that, The method includes: Obtain the text segmentation sequence of the text to be processed, and match each segment of the text segmentation sequence with each keyword of the preset domain keyword dictionary to obtain a first sparse representation vector; the domain keyword dictionary includes each keyword and the semantic weight of each keyword, and the non-zero dimension of the first sparse representation vector corresponds to the semantic weight of the successfully matched keyword. Based on the first sparse representation vector, construct the guiding signal matrix corresponding to the text segmentation sequence; The text segmentation sequence and the guiding signal matrix are input into a pre-trained language model to obtain a dense representation vector. Based on the dense representation vector, the first sparse representation vector is semantically smoothed to obtain the second sparse representation vector. The dense representation vector and the second sparse representation vector are fused to obtain the text embedding vector corresponding to the text to be processed.
2. The method according to claim 1, characterized in that, The step of matching each segment of the text segmentation sequence with each keyword of a preset domain keyword dictionary to obtain a first sparse representation vector includes: Obtain an initial sparse representation vector; the number of dimensions of the initial sparse representation vector is the same as the size of the domain keyword dictionary, and the dimension value of each dimension of the initial sparse representation vector is zero; Each segment of the text segmentation sequence is matched with each keyword in a preset domain keyword dictionary, and the semantic weight of the successfully matched keywords is extracted from the domain keyword dictionary. Based on the order of the keywords in the domain keyword dictionary, the semantic weights of the successfully matched keywords are filled into the corresponding dimensions of the initial sparse representation vector to obtain the first sparse representation vector.
3. The method according to claim 2, characterized in that, Before obtaining the first sparse representation vector by filling the semantic weights of the successfully matched keywords into the corresponding dimensions of the initial sparse representation vector according to the order of the keywords in the domain keyword dictionary, the method further includes: Obtain the word frequency of each word in the text segmentation sequence; The semantic weight of the successfully matched keyword is adjusted based on the word frequency of the word segment corresponding to the successfully matched keyword.
4. The method according to claim 1, characterized in that, The step of constructing the guiding signal matrix corresponding to the text segmentation sequence based on the first sparse representation vector includes: Obtain a set of target keywords associated with each word in the text segmentation sequence from the domain keyword dictionary; the keyword set includes at least one target keyword associated with each word segmentation. Obtain the semantic weights of the target keywords associated with each segment in the text segmentation sequence from the first sparse representation vector; Based on the semantic weights of the target keywords associated with each segmented word, the element values of each matrix element in the guiding signal matrix are obtained; the number of rows and columns of the guiding signal matrix are the same as the sequence length of the text segmentation sequence.
5. The method according to claim 1, characterized in that, The step of performing semantic smoothing on the first sparse representation vector based on the dense representation vector to obtain the second sparse representation vector includes: Calculate the semantic relevance between the dense representation vector and each keyword in the domain keyword dictionary; Based on the semantic relevance, the dimension values in the first sparse representation vector are adjusted to obtain the second sparse representation vector.
6. The method according to any one of claims 1 to 5, characterized in that, The step of fusing the dense representation vector and the second sparse representation vector to obtain the text embedding vector corresponding to the text to be processed includes: The second sparse representation vector is projected onto the same dimension as the dense representation vector to obtain the third sparse representation vector; The third sparse representation vector is concatenated with the dense representation vector to obtain the concatenated vector; The concatenated vector is input into a fully connected layer to obtain the text embedding vector corresponding to the text to be processed.
7. The method according to any one of claims 1 to 5, characterized in that, The step of fusing the dense representation vector and the second sparse representation vector to obtain the text embedding vector corresponding to the text to be processed includes: Based on the preset gating network, the first gating weight corresponding to the dense representation vector and the second gating weight corresponding to the second sparse representation vector are generated; Based on the first gating weight and the second gating weight, the dense representation vector and the second sparse representation vector are weighted and summed to obtain the text embedding vector corresponding to the text to be processed.
8. A text embedding vector generation device, characterized in that, The device includes: The first sparse representation vector acquisition module is used to acquire the text segmentation sequence of the text to be processed, match each segment of the text segmentation sequence with each keyword of the preset domain keyword dictionary, and obtain the first sparse representation vector; the domain keyword dictionary includes each keyword and the semantic weight of each keyword, and the non-zero dimension of the first sparse representation vector corresponds to the semantic weight of the successfully matched keyword. The guiding signal matrix construction module is used to construct the guiding signal matrix corresponding to the text segmentation sequence based on the first sparse representation vector. The dense representation vector acquisition module is used to input the text segmentation sequence and the guiding signal matrix into a pre-trained language model to obtain a dense representation vector. The second sparse representation vector acquisition module is used to perform semantic smoothing processing on the first sparse representation vector based on the dense representation vector to obtain the second sparse representation vector. The text embedding vector acquisition module is used to fuse the dense representation vector and the second sparse representation vector to obtain the text embedding vector corresponding to the text to be processed.
9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.