Text vector representation method and device, electronic equipment and storage medium

Abstract

The invention relates to a text vector representation method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a to-be-represented text which comprises a plurality of text segments; determining a static embedding vector and a position vector of each text fragment; according to the static embedding vectors and the position vectors of all the text segments, determining the incidence relation between the text segments; and determining a text vector of the to-be-expressed text according to the static embedding vectors, the position vectors and the incidence relation of all the text segments. By means of the method, the text features represented by the text segments are reserved, the incidence relation between the text segments is combined, and accurate text vector representation can be conducted on the to-be-represented text more accurately.

Description

Technical Field

[0001] The present invention relates to the field of text processing, and in particular to a text vector representation method, device, electronic device and storage medium. Background Art

[0002] Retrieval Augmented Generation is a technology applied in the field of knowledge content retrieval. It combines vector databases and large language models, and aims to enhance the generation capabilities of large models through retrieval technology. The process of building a vector database is: document preparation -> document segmentation -> slice vectorization -> storage. One of the focuses of RAG technology is to build a vector database, convert each document fragment into a high-dimensional vector representation, and store it for subsequent queries.

[0003] The current mainstream vector representation methods mainly rely on pre-trained language models, such as BERT, RoBERTa, etc. These models can capture the static semantic features of text, but they usually ignore the dynamic relationship between text fragments and the influence of their location. Specifically, the existing technology has the following advantages and disadvantages:

[0004] Lack of context sensitivity: Static embeddings cannot fully reflect the associations between different parts within the same document, which may lead to inaccurate retrieval results.

[0005] Loss of local information: When a document is segmented into multiple fragments, a single fragment may lose its contextual information in the entire document. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a text vector representation method, device, electronic device and storage medium, aiming to solve at least one of the above technical problems.

[0007] In the first aspect, the technical solution of the present invention to solve the above technical problem is as follows: a text vector representation method, the method comprising:

[0008] Acquire a text to be represented, wherein the text to be represented includes a plurality of text segments;

[0009] Determine a static embedding vector and a position vector of each of the text segments, wherein each of the static embedding vectors represents a text vector of the corresponding text segment itself, and each of the position vectors represents a position of the corresponding text segment in the text to be represented;

[0010] Determine the association relationship between each of the text segments according to the static embedding vectors and position vectors of all the text segments;

[0011] The text vector of the text to be represented is determined according to the static embedding vectors, position vectors and association relationships of all text segments.

[0012] The beneficial effect of the present invention is that in the present application scheme, by integrating the position of the document fragment in the text to be represented, the association relationship between each of the text fragments can be accurately determined, so that the present scheme can better understand and utilize the association relationship between each text fragment in the text to be represented, thereby improving the expression accuracy of the text vector.

[0013] Based on the above technical solution, the present invention can also be improved as follows.

[0014] Further, determining the position vector of each of the text segments includes:

[0015] For each of the text segments, based on the position of the text segment in the text to be represented, a position vector of the text segment is determined by adopting a sine position coding method or a cosine position coding method.

[0016] Further, for each of the text segments, if the position of the text segment in the text to be represented is an even position, then determining the position vector of the text segment based on the position of the text segment in the text to be represented by using a sine position encoding method or a cosine position encoding method includes:

[0017] Based on the position of the text segment in the text to be represented, a position vector of the text segment is determined by using a sinusoidal position encoding method;

[0018] For each of the text segments, if the position of the text segment in the text to be represented is an odd position, then based on the position of the text segment in the text to be represented, the position vector of the text segment is determined by using a sine position encoding method or a cosine position encoding method, including:

[0019] Based on the position of the text segment in the text to be represented, a position vector of the text segment is determined by using a cosine position encoding method.

[0020] Further, determining the association relationship between the text segments according to the static embedding vectors and position vectors of all the text segments includes:

[0021] According to the static embedding vectors and position vectors of all text fragments, a multi-head self-attention mechanism is used to determine the association relationship between each of the text fragments.

[0022] Furthermore, the multi-head self-attention mechanism is used to determine the association relationship between the text fragments according to the static embedding vectors and position vectors of all the text fragments, including:

[0023] For each of the text fragments, a multi-head self-attention mechanism is used to determine the query vector and key vector corresponding to the text fragment according to the static embedding vectors and position vectors of all text fragments, wherein the query vector represents the text features of the text fragment, and the key vector represents the association relationship between the text fragment and each text fragment in the text to be represented.

[0024] Further, determining the text vector of the text to be represented according to the static embedding vectors, position vectors and association relationships of all text segments includes:

[0025] For each of the text fragments, according to the query vector and the key vector corresponding to the text fragment, a multi-head self-attention mechanism is used to determine the value vector corresponding to the text fragment, wherein the value vector represents the text vector combined with the context information corresponding to the text fragment;

[0026] The text vector of the text to be represented is determined according to the query vectors, key vectors and value vectors corresponding to all the text segments.

[0027] In a second aspect, in order to solve the above technical problem, the present invention further provides a text vector representation device, the device comprising:

[0028] An acquisition module, used for acquiring a text to be represented, wherein the text to be represented includes a plurality of text segments;

[0029] An initial vector representation module, used to determine a static embedding vector and a position vector of each of the text segments, wherein each of the static embedding vectors represents a text vector of the corresponding text segment itself, and each of the position vectors represents a position of the corresponding text segment in the text to be represented;

[0030] An association relationship determination module, used to determine the association relationship between each of the text segments according to the static embedding vectors and position vectors of all the text segments;

[0031] The text vector representation module is used to determine the text vector of the text to be represented based on the static embedding vectors, position vectors and association relationships of all text fragments.

[0032] In a third aspect, in order to solve the above-mentioned technical problem, the present invention also provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the text vector representation method of the present application is implemented.

[0033] In a fourth aspect, in order to solve the above-mentioned technical problem, the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the text vector representation method of the present application is implemented.

[0034] Additional aspects and advantages of the present application will be partially given in the following description, which will become apparent from the following description, or will be understood through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for describing the embodiments of the present invention are briefly introduced below.

[0036] Figure 1 A flowchart of a text vector representation method provided by one embodiment of the present invention;

[0037] Figure 2 A schematic diagram of the structure of a text vector representation device provided by one embodiment of the present invention;

[0038] Figure 3 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0039] The principles and features of the present invention are described below. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0040] The technical solution of the present invention and how the technical solution of the present invention solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0041] The solution provided by the embodiment of the present invention can be applied to any application scenario where accurate text vector representation of the text to be represented is required. The solution provided by the embodiment of the present invention can be executed by any electronic device, for example, it can be a user's terminal device, including at least one of the following: a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a smart TV, and a smart car device.

[0042] The embodiment of the present invention provides a possible implementation method, such as Figure 1 As shown, a flowchart of a text vector representation method is provided. The solution can be executed by any electronic device, for example, a terminal device, or a terminal device and a server. For the convenience of description, the method provided by the embodiment of the present invention will be described below by taking the terminal device as the execution subject as an example. Figure 1 The method may include the following steps:

[0043] S10, obtaining a text to be represented, wherein the text to be represented includes a plurality of text segments;

[0044] S20, determining a static embedding vector and a position vector of each of the text segments, wherein each of the static embedding vectors represents a text vector of the corresponding text segment itself, and each of the position vectors represents a position of the corresponding text segment in the text to be represented;

[0045] S30, determining the association relationship between each of the text segments according to the static embedding vectors and position vectors of all the text segments;

[0046] S40, determining the text vector of the text to be represented according to the static embedding vectors, position vectors and association relationships of all text segments.

[0047] In the present application scheme, by integrating the position of the document fragment in the text to be represented, the association relationship between each of the text fragments can be accurately determined, so that the present scheme can better understand and utilize the association relationship between each of the text fragments in the text to be represented, thereby improving the expression accuracy of the text vector.

[0048] The solution of the present invention is further described below in conjunction with the following specific embodiments. In this embodiment, a text vector representation method provided in this embodiment may include the following steps:

[0049] S10, obtaining a text to be represented, wherein the text to be represented includes a plurality of text segments;

[0050] The text to be represented is a document that needs to be represented by a text vector. The text to be represented can be pre-divided into multiple text segments, and the specific division method will be described below.

[0051] S20, determining a static embedding vector and a position vector of each of the text segments, wherein each of the static embedding vectors represents a text vector of the corresponding text segment itself, and each of the position vectors represents a position of the corresponding text segment in the text to be represented;

[0052] Each of the static embedding vectors represents the text features expressed by the corresponding text segment itself. Absolute or relative position encoding can be used to represent the position of each text segment in the text to be represented.

[0053] Alternatively, a pre-trained language model (BERT) can be used to generate static embedding vectors for each text snippet.

[0054] Optionally, in the above S20, determining the position vector of each text segment includes:

[0055] S201 , for each of the text segments, based on the position of the text segment in the text to be represented, determine the position vector of the text segment by using a sine position coding method or a cosine position coding method.

[0056] Further, for each of the text segments, if the position of the text segment in the text to be represented is an even position, then determining the position vector of the text segment based on the position of the text segment in the text to be represented by using a sine position encoding method or a cosine position encoding method includes:

[0057] Based on the position of the text segment in the text to be represented, a position vector of the text segment is determined by using a sinusoidal position encoding method;

[0058] For each of the text segments, if the position of the text segment in the text to be represented is an odd position, then based on the position of the text segment in the text to be represented, the position vector of the text segment is determined by using a sine position encoding method or a cosine position encoding method, including:

[0059] Based on the position of the text segment in the text to be represented, a position vector of the text segment is determined by using a cosine position encoding method.

[0060] Specifically, for each of the text segments, the implementation method of determining the position vector of the text segment by using the sine position coding method or the cosine position coding method can be referred to the following formula:

[0061]

[0062] Among them, pos represents the position of a text segment in the text to be represented, d model It represents the embedding dimension (i.e. the dimension of the position vector), which is generally 512. i represents the dimension index in the position vector, 2i and 2i+1 represent even and odd positions respectively, and PE (pos，2i) Represents the position vector of the text segment corresponding to the even position, PE (pos，2i+1) The position vector representing the text fragment corresponding to the odd position.

[0063] S30, determining the association relationship between each of the text segments according to the static embedding vectors and position vectors of all the text segments;

[0064] The association relationship refers to the semantic association relationship between various text segments based on the context.

[0065] Optionally, an implementation of the above S30 is:

[0066] S301, based on the static embedding vectors and position vectors of all text fragments, a multi-head self-attention mechanism is used to determine the association relationship between each of the text fragments.

[0067] Furthermore, the above-mentioned multi-head self-attention mechanism is used to determine the association relationship between each of the text fragments based on the static embedding vectors and position vectors of all the text fragments, including:

[0068] For each of the text fragments, based on the static embedding vectors and position vectors of all text fragments, a multi-head self-attention mechanism is used to determine the query vector (which can be expressed as Q (Query)) and key vector (which can be expressed as K (Key)) corresponding to the text fragment, wherein the query vector represents the text features of the text fragment, and the key vector represents the association relationship between the text fragment and each text fragment in the text to be represented (including the text fragment itself).

[0069] Among them, the above-mentioned association relationship can be represented based on similarity.

[0070] For each text segment, the query vector corresponding to the text segment is a vector, and the key vector corresponding to the text segment can be a group of vectors.

[0071] Specifically, in the multi-head self-attention mechanism, it can be implemented through three elements: Q (Query), K (Key), and V (Value), which represent the query request, the relevance measure, and the actual data content, respectively. When the model receives the input, it converts the input vector into these three vectors and calculates the similarity scores between them. These scores are used to calculate the weighted sum of each position in the input, allowing the model to focus on those important positions. The benefit of this attention mechanism is that it can adaptively learn the key information in the input without manually specifying which parts are important.

[0072] Among them, Query (query vector): represents the request for obtaining information. In the model, Query can be a vector or a group of vectors, which is used to guide the model to focus on a specific part of the input data, that is, the vector to be queried. In the present application, it can represent the text vector corresponding to any text segment itself.

[0073] Key (key vector): a measure of relevance to the query. Key is also a vector or a vector set, which is used to calculate the similarity or relevance with the query, that is, to calculate the association between each text segment in the text to be represented and the text segment corresponding to the query. The greater the similarity, the greater the relevance.

[0074] Value (value vector): contains the actual data of the information to be extracted. Value also exists in the form of a vector or a vector set, and its content is closely related to the query and key. Specifically, the final output representation can be generated by multiplying the attention score with the value vector (as a weight) and performing a weighted sum.

[0075] S40, determining the text vector of the text to be represented according to the static embedding vectors, position vectors and association relationships of all text segments.

[0076] Among them, each text segment corresponds to a query vector, a key vector and a value vector, and through the query vector, the key vector and the value vector, the static embedding vector, the position vector and the association relationship corresponding to the text segment can be represented. In other words, based on the static embedding vector, the position vector and the association relationship corresponding to each text segment, the query vector, the key vector and the value vector corresponding to the text segment can be determined, so as to determine the text vector of the text to be represented based on the query vector, the key vector and the value vector of each text segment.

[0077] Optionally, in the above S40, determining the text vector of the text to be represented according to the static embedding vectors, position vectors and association relationships of all text segments includes:

[0078] For each of the text fragments, according to the query vector and the key vector corresponding to the text fragment, a multi-head self-attention mechanism is used to determine the value vector corresponding to the text fragment, wherein the value vector represents the text vector combined with the context information corresponding to the text fragment;

[0079] The text vector of the text to be represented is determined according to the query vectors, key vectors and value vectors corresponding to all the text segments.

[0080] Furthermore, the above-mentioned determining the text vector of the text to be represented according to the query vector, the key vector and the value vector corresponding to each of all the text segments includes:

[0081] For each text segment, perform linear transformation on the query vector, key vector and value vector corresponding to the text segment to obtain the linearly transformed query vector, key vector and value vector; obtain the text vector of the text segment through the multi-head attention formula based on the linearly transformed query vector, key vector and value vector;

[0082] Perform weighted summation on the text vectors corresponding to all text segments to obtain the text vector corresponding to the first head in the multi-head attention mechanism for the text to be represented;

[0083] The text vectors corresponding to all heads in the multi-head attention mechanism are averaged to obtain the text vector of the text to be represented.

[0084] The above process can be specifically described as follows:

[0085] The query vector, key vector, and value vector corresponding to a text fragment are represented as Q i , K j and V j , where i represents the index identifier of the current text segment, the current text segment can be any text segment in the text to be represented, and j represents the index identifier of any text segment in the text to be represented except the current text segment.

[0086] The query vector, key vector and value vector corresponding to the text segment are linearly transformed, and the query vector, key vector and value vector after linear transformation can be expressed as:

[0087]

[0088] Among them, W Q , W K and W V is a learnable weight matrix used to linearly transform the original embedding, E(s i ) represents the static embedding vector corresponding to the current text segment i, PE(i) represents the position vector corresponding to the current text segment i, and E(s j ) represents the static embedding vector corresponding to any text segment j, and PE(j) represents the position vector corresponding to any text segment j.

[0089] For the current text segment, the text vector of the current text segment can be obtained through the multi-head attention formula, which can be specifically achieved by the following formula:

[0090]

[0091] Among them, T represents transpose, d k is the dimension of the key vector K.

[0092] The above weighted summation of the text vectors corresponding to all text segments to obtain the text vector corresponding to the first head in the multi-head attention mechanism for the text to be represented can be achieved by the following formula:

[0093]

[0094] Among them, A i Represents the text vector corresponding to the first head in the multi-head attention mechanism.

[0095] The above average calculation of the text vectors corresponding to all heads (h heads) of the text to be represented in the multi-head attention mechanism can be achieved by the following formula to obtain the text vector of the text to be represented:

[0096]

[0097] Among them, MuA i A text vector representing the text to be represented.

[0098] By using multiple attention heads to capture different feature patterns, we can better learn the associations between different text fragments. After being processed by the multi-head self-attention mechanism, we finally get a richer text vector representation, which not only contains the information of the original fragment (the current text fragment), but also incorporates the context clues of the surrounding fragments (the text fragments associated with the current text fragment).

[0099] In the present application, the processing corresponding to the above multi-head attention mechanism can be described as the processing of the context-aware module, which captures the dependencies between the fragments through the multi-head self-attention mechanism. It receives the static embedding of the fragment and its position encoding as input, and outputs the enhanced vector (text vector of the text to be represented) representation.

[0100] Specifically, a text segment S in the text to be represented i The corresponding text vector obtained based on this solution can be expressed based on the following formula:

[0101] v i =E(s i )+C i (7)

[0102] Among them, S i represents a text segment in the text to be represented, E(s i ) represents the text segment S i Through the static embedding vector generated by BERT, C represents a context-aware module, C i Then it means the text segment S i The corresponding context-aware module, v i Represents a text segment S i The corresponding text vector.

[0103] Optionally, after obtaining the text vector of the text to be represented, the text vector of the text to be represented may be applied to the large model for semantic recognition.

[0104] Optionally, the text to be represented may be segmented in the following manner to obtain multiple text segments:

[0105] According to the paragraphs corresponding to the text to be represented, the text to be represented is preliminarily segmented to obtain multiple original blocks. For each of the original blocks, the optimal number of segments corresponding to the original block is determined. Based on the optimal number of segments, the original block is segmented to obtain at least two text segments corresponding to the original block.

[0106] The optimal number of cuts for the i-th original cut is calculated as follows:

[0107]

[0108] Among them, S i represents the optimal number of segments of the ith original segment, C represents the total number of characters corresponding to the ith original segment, and T i represents the information density of the i-th original cut, D i Indicates the topic change degree of the i-th original slice.

[0109] Furthermore, the adjustment parameters α, β and γ are determined as follows:

[0110] For each original block in the text to be represented, the original block is used as the current original block, the first information density of the text to be represented is determined, and it is determined whether the first information density is lower than the preset minimum information density. If so, it is further determined whether the current original block also includes a previous original block (the previous original block refers to the original block that appears in the previous text of the current original block, and the previous original block may contain content associated with the content of the current original block). If so, the first keyword of the title of the current original block and the second keyword of the current original block are extracted, and the third keyword of the title of the previous original block and the fourth keyword of the previous original block are extracted, the first correlation and the first similarity between the first keyword and the third keyword are determined, the second correlation and the second similarity between the second keyword and the fourth keyword are determined, a first similarity score is determined according to the first correlation and the first similarity, a second similarity score is determined according to the second correlation and the second similarity, a first total score of the current original block is determined based on the first similarity score and the second similarity score, and the range of α is determined based on the first total score.

[0111] The process of determining similarity scores is as follows:

[0112] The similarity score is determined by determining the weights corresponding to the correlation and similarity.

[0113] The process of determining the first total score of the current original cut block based on the first similarity score and the second similarity score is as follows:

[0114] Determine the first overall score based on the title weight and the text weight.

[0115] If the first information density is not lower than the preset minimum information density, or the current original block does not include the previous original block, the preset α is used as α in the optimal number of divisions of the current original block, and further judged whether β and γ need to be adjusted.

[0116] The preceding original segment refers to an original segment before the current original segment among a plurality of original segments arranged in the order of the text of the document to be segmented.

[0117] When any original cut contains more than a preset number of professional terms and more than a preset length of text description of technical specific content, the target number of professional terms is determined, and the target length of the text description is determined. Based on the table corresponding to the number of professional terms and the first weight, and the table corresponding to the length of the text description, the first target weight corresponding to the target number of professional terms and the second target weight corresponding to the target length of the text description are determined. According to the first target weight and the second target weight, combined with the number of professional terms and the text description, the second total score corresponding to the original cut is determined. The range of β is determined according to the second total score.

[0118] If any original block does not contain more than the preset number of professional terms, or does not contain a text description of technical specific content exceeding the preset length, the preset β is used as the β in the optimal number of divisions of the current original block, and further determines whether γ needs to be adjusted.

[0119] According to the third keyword and the fourth keyword, it is determined whether the current original block and the previous original block are completely different topics, that is, it is determined whether the overlap between the third keyword and the fourth keyword is lower than the preset overlap. If it is lower, γ is adjusted. If it is not lower than the preset overlap, the preset γ is used as the γ in the optimal number of segments of the current original block.

[0120] Based on Figure 1 Based on the same principle as the method shown in , the embodiment of the present invention also provides a text vector representation device 20, such as Figure 2 As shown in , the text vector representation device 20 may include an acquisition module 210, an initial vector representation module 220, an association relationship determination module 230 and a text vector representation module 240, wherein:

[0121] An acquisition module 210 is used to acquire a text to be represented, wherein the text to be represented includes a plurality of text segments;

[0122] An initial vector representation module 220, used to determine a static embedding vector and a position vector of each of the text segments, each of the static embedding vectors represents a text vector of the corresponding text segment itself, and each of the position vectors represents a position of the corresponding text segment in the text to be represented;

[0123] An association relationship determination module 230, configured to determine the association relationship between each of the text segments according to the static embedding vectors and position vectors of all the text segments;

[0124] The text vector representation module 240 is used to determine the text vector of the text to be represented according to the static embedding vectors, position vectors and association relationships of all text segments.

[0125] Optionally, when determining the position vector of each text segment, the initial vector representation module 220 is specifically configured to:

[0126] For each of the text segments, based on the position of the text segment in the text to be represented, a position vector of the text segment is determined by adopting a sine position coding method or a cosine position coding method.

[0127] Optionally, for each of the text segments, if the position of the text segment in the text to be represented is an even position, the initial vector representation module 220 determines the position vector of the text segment based on the position of the text segment in the text to be represented by using a sine position encoding method or a cosine position encoding method, specifically for:

[0128] Based on the position of the text segment in the text to be represented, a position vector of the text segment is determined by using a sinusoidal position encoding method;

[0129] For each of the text segments, if the position of the text segment in the text to be represented is an odd position, the initial vector representation module 220 determines the position vector of the text segment based on the position of the text segment in the text to be represented by using a sine position encoding method or a cosine position encoding method, specifically for:

[0130] Based on the position of the text segment in the text to be represented, a position vector of the text segment is determined by using a cosine position encoding method.

[0131] Optionally, when determining the association relationship between the text segments according to the static embedding vectors and position vectors of all the text segments, the association relationship determination module 230 is specifically configured to:

[0132] According to the static embedding vectors and position vectors of all text fragments, a multi-head self-attention mechanism is used to determine the association relationship between each of the text fragments.

[0133] Optionally, when the association relationship determination module 230 uses a multi-head self-attention mechanism to determine the association relationship between each of the text segments according to the static embedding vectors and position vectors of all the text segments, it is specifically used to:

[0134] For each of the text fragments, a multi-head self-attention mechanism is used to determine the query vector and key vector corresponding to the text fragment according to the static embedding vectors and position vectors of all text fragments, wherein the query vector represents the text features of the text fragment, and the key vector represents the association relationship between the text fragment and each text fragment in the text to be represented.

[0135] Optionally, when determining the text vector of the text to be represented according to the static embedding vectors, position vectors and association relationships of all text segments, the text vector representation module 240 is specifically used to:

[0136] For each of the text fragments, according to the query vector and the key vector corresponding to the text fragment, a multi-head self-attention mechanism is used to determine the value vector corresponding to the text fragment, wherein the value vector represents the text vector combined with the context information corresponding to the text fragment;

[0137] The text vector of the text to be represented is determined according to the query vectors, key vectors and value vectors corresponding to all the text segments.

[0138] The text vector representation device of the embodiment of the present invention can execute the text vector representation method provided by the embodiment of the present invention, and the implementation principle is similar. The actions performed by each module and unit in the text vector representation device in each embodiment of the present invention correspond to the steps in the text vector representation method in each embodiment of the present invention. For the detailed functional description of each module of the text vector representation device, please refer to the description in the corresponding text vector representation method shown in the previous text, which will not be repeated here.

[0139] Among them, the above-mentioned text vector representation device can be a computer program (including program code) running in a computer device, for example, the text vector representation device is an application software; the device can be used to execute the corresponding steps in the method provided in an embodiment of the present invention.

[0140] In some embodiments, the text vector representation device provided by the embodiment of the present invention can be implemented in a combination of software and hardware. As an example, the text vector representation device provided by the embodiment of the present invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the text vector representation method provided by the embodiment of the present invention. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs) or other electronic components.

[0141] In other embodiments, the text vector representation device provided by the embodiment of the present invention can be implemented in software. Figure 2 A text vector representation device stored in a memory is shown, which may be software in the form of a program or plug-in, and includes a series of modules, including an acquisition module 210, an initial vector representation module 220, an association relationship determination module 230, and a text vector representation module 240, for implementing the text vector representation method provided in an embodiment of the present invention.

[0142] The modules involved in the embodiments of the present invention may be implemented by software or hardware, wherein the name of a module does not limit the module itself in some cases.

[0143] Based on the same principle as the method shown in the embodiments of the present invention, an electronic device is also provided in the embodiments of the present invention, which may include but is not limited to: a processor and a memory; the memory is used to store a computer program; the processor is used to execute the method shown in any embodiment of the present invention by calling the computer program.

[0144] In an alternative embodiment, an electronic device is provided, such as Figure 3 As shown, Figure 3The electronic device 4000 shown includes: a processor 4001 and a memory 4003. The processor 4001 and the memory 4003 are connected, such as through a bus 4002. Optionally, the electronic device 4000 may also include a transceiver 4004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 4004 is not limited to one, and the structure of the electronic device 4000 does not constitute a limitation on the embodiments of the present invention.

[0145] Processor 4001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of the present invention. Processor 4001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0146] The bus 4002 may include a path to transmit information between the above components. The bus 4002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 4002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0147] The memory 4003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to this.

[0148] The memory 4003 is used to store the application code (computer program) for executing the solution of the present invention, and the execution is controlled by the processor 4001. The processor 4001 is used to execute the application code stored in the memory 4003 to implement the content shown in the above method embodiment.

[0149] The electronic device may also be a terminal device. Figure 3 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0150] An embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer can execute the corresponding contents of the aforementioned method embodiment.

[0151] According to another aspect of the present invention, a computer program product or a computer program is also provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in the implementation modes of the above-mentioned various embodiments.

[0152] Computer program code for performing the operations of the present invention may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0153] It should be understood that the flow charts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the module, the program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0154] The computer-readable storage medium provided by the embodiment of the present invention may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device, or device.

[0155] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0156] The above description is only a preferred embodiment of the present invention and an explanation of the technical principles used. Those skilled in the art should understand that the disclosure scope involved in the present invention is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present invention (but not limited to) by each other.

Claims

1. A text vector representation method, characterized in that: The following steps are involved: Acquire a text to be represented, wherein the text to be represented includes a plurality of text segments; Determine a static embedding vector and a position vector of each of the text segments, wherein each of the static embedding vectors represents a text vector of the corresponding text segment itself, and each of the position vectors represents a position of the corresponding text segment in the text to be represented; Determine the association relationship between each of the text segments according to the static embedding vectors and position vectors of all the text segments; The text vector of the text to be represented is determined according to the static embedding vectors, position vectors and association relationships of all text segments.

2. The method according to claim 1, characterized in that Determining the position vector of each of the text segments includes: For each of the text segments, based on the position of the text segment in the text to be represented, a position vector of the text segment is determined by adopting a sine position coding method or a cosine position coding method.

3. The method according to claim 2, characterized in that For each of the text segments, if the position of the text segment in the text to be represented is an even position, then determining the position vector of the text segment based on the position of the text segment in the text to be represented by using a sine position encoding method or a cosine position encoding method includes: Based on the position of the text segment in the text to be represented, a position vector of the text segment is determined by using a sinusoidal position encoding method; For each of the text segments, if the position of the text segment in the text to be represented is an odd position, then based on the position of the text segment in the text to be represented, the position vector of the text segment is determined by using a sine position encoding method or a cosine position encoding method, including: Based on the position of the text segment in the text to be represented, a position vector of the text segment is determined by using a cosine position encoding method.

4. The method according to any one of claims 1 to 3, characterized in that The determining the association relationship between the text segments according to the static embedding vectors and position vectors of all the text segments includes: According to the static embedding vectors and position vectors of all text fragments, a multi-head self-attention mechanism is used to determine the association relationship between each of the text fragments.

5. The method according to claim 4, characterized in that The method of determining the association between the text segments using a multi-head self-attention mechanism based on the static embedding vectors and position vectors of all the text segments includes: For each of the text fragments, a multi-head self-attention mechanism is used to determine the query vector and key vector corresponding to the text fragment according to the static embedding vectors and position vectors of all text fragments, wherein the query vector represents the text features of the text fragment, and the key vector represents the association relationship between the text fragment and each text fragment in the text to be represented.

6. The method according to claim 5, characterized in that The step of determining the text vector of the text to be represented according to the static embedding vectors, position vectors and association relationships of all text segments includes: For each of the text fragments, according to the query vector and the key vector corresponding to the text fragment, a multi-head self-attention mechanism is used to determine the value vector corresponding to the text fragment, wherein the value vector represents the text vector combined with the context information corresponding to the text fragment; The text vector of the text to be represented is determined according to the query vectors, key vectors and value vectors corresponding to all the text segments.

7. A text vector representation device, characterized in that: include: An acquisition module, used for acquiring a text to be represented, wherein the text to be represented includes a plurality of text segments; An initial vector representation module, used to determine a static embedding vector and a position vector of each of the text segments, each of the static embedding vectors represents a text vector of the corresponding text segment itself, and each of the position vectors represents a position of the corresponding text segment in the text to be represented; An association relationship determination module, used to determine the association relationship between each of the text segments according to the static embedding vectors and position vectors of all the text segments; The text vector representation module is used to determine the text vector of the text to be represented based on the static embedding vectors, position vectors and association relationships of all text fragments.

8. The device according to claim 7, characterized in that When determining the position vector of each text segment, the initial vector representation module is specifically used to: For each of the text segments, based on the position of the text segment in the text to be represented, a position vector of the text segment is determined by adopting a sine position coding method or a cosine position coding method.

9. An electronic device, characterized in that: The method comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 6 when executing the computer program.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images