Text semantic determination methods, devices, electronic devices and storage media
By processing text sentence encoding vectors through a cross-attention neural network layer and combining it with a linear classifier to determine sentence pair relationships, the problem of insufficient capture of semantic relationships between sentence pairs in existing models is solved, thereby improving the generalization performance and accuracy of text semantic models.
Patent Information
- Application Number
- CN202311423674.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-27
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2043-10-27
AI Technical Summary
Existing text semantic models are insufficient in generalization performance and accuracy, and cannot effectively capture the complex semantic relationships between sentence pairs.
A cross-attention neural network layer is used to encode sentences in the text. By calculating the difference of mutual attention vectors, dot product and element-wise product between sentences, and combining it with a linear classifier, the relationship between sentences is determined, and thus the semantics of the text is determined.
It improves the ability to grasp subtle differences in sentences during text semantic recognition and enhances the model's generalization ability.
Smart Images

Figure CN117540015B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, electronic device and storage medium for determining text semantics. Background Technology
[0002] Text semantic models are an important research topic in the field of natural language processing, with wide applications in text retrieval, text clustering, and automatic summarization. Existing text semantic models are mainly based on single-sentence level modeling, such as cosine semantics based on the bag-of-words model and semantics based on large-scale pre-trained models.
[0003] In related technologies, text semantic models are typically based on single-sentence levels, such as cosine semantics based on the bag-of-words model and semantics based on large-scale pre-trained models. These models measure the semantics between two sentences by calculating the semantics of a single sentence.
[0004] However, these models in related technologies suffer from poor generalization performance and insufficient grasp of subtle differences within sentences. This is because these models consider the semantics between words and the overall meaning within a single sentence, rather than the complex semantic relationships between sentence pairs. Summary of the Invention
[0005] In view of this, embodiments of this application provide a text semantic determination method, apparatus, electronic device, and storage medium to solve the problems of poor generalization ability and low accuracy of text semantic determination models in the prior art.
[0006] A first aspect of this application provides a text semantic determination method, including:
[0007] Extract the first and second statements from the text;
[0008] Encode the first statement and the second statement respectively to obtain the encoding vector of the first statement and the encoding vector of the second statement.
[0009] The first statement encoding vector and the second statement encoding vector are processed using a cross-attention neural network layer to obtain the first statement mutual attention vector and the second statement mutual attention vector.
[0010] The average vectors of the first and second statements are calculated by averaging the mutual attention vectors of the first and second statements respectively.
[0011] The difference between the average vector of the first statement and the average vector of the second statement is taken as the first intermediate vector, the dot product of the average vector of the first statement and the average vector of the second statement is taken as the second intermediate vector, and the element-wise product of the average vector of the first statement and the average vector of the second statement is taken as the third intermediate vector.
[0012] The first, second, and third intermediate vectors are concatenated in parallel to obtain the classification vector;
[0013] A linear classifier is used to classify the classification vectors to obtain the relationship between the first and second statements;
[0014] The semantics of the text are determined based on the relationship between the first and second statements.
[0015] A second aspect of this application provides a text semantic determination apparatus, comprising:
[0016] The acquisition module is configured to acquire the first and second statements in the text.
[0017] The encoding module is configured to encode the first statement and the second statement respectively, to obtain the encoding vector of the first statement and the encoding vector of the second statement;
[0018] The cross-attention module is configured to use a cross-attention neural network layer to process the first statement encoding vector and the second statement encoding vector to obtain the first statement mutual attention vector and the second statement mutual attention vector.
[0019] The calculation module is configured to perform average calculations on the mutual attention vectors of the first statement and the mutual attention vectors of the second statement respectively, to obtain the average vector of the first statement and the average vector of the second statement.
[0020] The calculation module is also configured to use the difference between the average vector of the first statement and the average vector of the second statement as the first intermediate vector, the dot product of the average vector of the first statement and the average vector of the second statement as the second intermediate vector, and the element-wise product of the average vector of the first statement and the average vector of the second statement as the third intermediate vector.
[0021] The calculation module is also configured to concatenate the first intermediate vector, the second intermediate vector, and the third intermediate vector in parallel to obtain a classification vector;
[0022] The classification module is configured to classify the classification vectors using the existing classifier to obtain the relationship between the first statement and the second statement;
[0023] The determination module is configured to determine the semantics of the text based on the relationship between the first and second statements.
[0024] A third aspect of this application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described method.
[0025] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.
[0026] The beneficial effects of this application embodiment compared with the prior art are as follows: This application embodiment performs cross-attention processing on the sentences in the text and uses the processed mutual attention vector to calculate the relationship between the sentences, and then determines the semantics of the text based on the relationship between the sentences, thereby improving the ability to grasp the subtle differences in the sentences during text semantic recognition and improving the generalization ability. Attached Figure Description
[0027] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0028] Figure 1 This is a schematic diagram illustrating the application scenario of this application embodiment.
[0029] Figure 2 This is a flowchart illustrating a text semantic determination method provided in an embodiment of this application.
[0030] Figure 3 This is a flowchart illustrating a method provided in this application for encoding a first statement and a second statement to obtain encoding vectors for the first statement and the second statement, respectively.
[0031] Figure 4 This is a flowchart illustrating the method provided in this application for processing the first statement encoding vector and the second statement encoding vector using a cross-attention neural network layer to obtain the first statement mutual attention vector and the second statement mutual attention vector.
[0032] Figure 5 This is a flowchart illustrating a method provided in this application for classifying classification vectors using a linear classifier to obtain the relationship between the first statement and the second statement.
[0033] Figure 6 This is a flowchart illustrating a method for obtaining a learning loss function by comparing and learning a first interaction representation vector and a second interaction representation vector, as provided in an embodiment of this application.
[0034] Figure 7 This is a flowchart illustrating a method for determining the semantics of text based on the relationship between a first statement and a second statement, as provided in an embodiment of this application.
[0035] Figure 8 This is a schematic diagram of a text semantic determination device provided in an embodiment of this application.
[0036] Figure 9 This is a schematic diagram of the electronic device provided in the embodiments of this application. Detailed Implementation
[0037] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.
[0038] A text semantic determination method and apparatus according to embodiments of this application will now be described in detail with reference to the accompanying drawings.
[0039] Figure 1 This is a schematic diagram illustrating an application scenario according to an embodiment of this application. The application scenario may include terminal devices 1, 2, and 3, server 4, and network 5.
[0040] Terminal devices 1, 2, and 3 can be hardware or software. When terminal devices 1, 2, and 3 are hardware, they can be various electronic devices with displays that support communication with server 4, including but not limited to smartphones, tablets, laptops, and desktop computers. When terminal devices 1, 2, and 3 are software, they can be installed in the electronic devices described above. Terminal devices 1, 2, and 3 can be implemented as multiple software programs or software modules, or as a single software program or software module; this application embodiment does not impose any limitations on this. Furthermore, various applications can be installed on terminal devices 1, 2, and 3, such as data processing applications, instant messaging tools, social platform software, search applications, shopping applications, etc.
[0041] Server 4 can be a server that provides various services, such as a backend server that receives requests sent by terminal devices with which it has established communication connections. This backend server can receive and analyze the requests sent by the terminal devices and generate processing results. Server 4 can be a single server, a server cluster consisting of several servers, or a cloud computing service center; this embodiment of the application does not impose any limitations on these aspects.
[0042] It should be noted that server 4 can be either hardware or software. When server 4 is hardware, it can be various electronic devices that provide various services to terminal devices 1, 2, and 3. When server 4 is software, it can be multiple software programs or software modules that provide various services to terminal devices 1, 2, and 3, or it can be a single software program or software module that provides various services to terminal devices 1, 2, and 3. This application embodiment does not impose any limitations on this.
[0043] Network 5 can be a wired network using coaxial cable, twisted pair, and fiber optic connection, or it can be a wireless network that enables interconnection of various communication devices without wiring, such as Bluetooth, Near Field Communication (NFC), and Infrared. This application embodiment does not limit this.
[0044] Users can establish a communication connection with server 4 via network 5 through terminal devices 1, 2, and 3 to receive or send information. Specifically, server 4 can receive text from users on terminal devices 1, 2, and 3, and parse the text to determine its semantics. Server 4 can also send the determined semantics to terminal devices 1, 2, and 3 to display to the user.
[0045] It should be noted that the specific types, quantities, and combinations of terminal devices 1, 2, and 3, server 4, and network 5 can be adjusted according to the actual needs of the application scenario, and this application embodiment does not impose any restrictions on this.
[0046] As mentioned above, related technologies typically rely on sentence-level text semantic models, such as cosine semantics based on the bag-of-words model and semantics based on large-scale pre-trained models. These models measure the semantics between two sentences by calculating the semantics of individual sentences. However, these models suffer from poor generalization performance and insufficient grasp of subtle differences within sentences. This is because these models consider the semantics between words and the overall meaning within a single sentence, rather than the complex semantic relationships between sentence pairs.
[0047] In view of this, embodiments of this application provide a text semantic determination method, which performs cross-attention processing on the sentences in the text and uses the processed mutual attention vectors to calculate the relationship between the sentences, and then determines the semantics of the text based on the relationship between the sentences, thereby improving the ability to grasp the subtle differences in the sentences during text semantic recognition and improving the generalization ability.
[0048] Figure 2 This is a flowchart illustrating a text semantic determination method provided in an embodiment of this application. Figure 2 Text semantic determination methods can be derived from Figure 1The server, or a terminal device with a certain computing power, executes the command. For example... Figure 2 As shown, the text semantic determination method includes the following steps:
[0049] In step S201, the first and second statements in the text are obtained.
[0050] In step S202, the first statement and the second statement are encoded respectively to obtain the first statement encoding vector and the second statement encoding vector.
[0051] In step S203, the first statement encoding vector and the second statement encoding vector are processed using a cross-attention neural network layer to obtain the first statement mutual attention vector and the second statement mutual attention vector.
[0052] In step S204, the mutual attention vectors of the first statement and the mutual attention vectors of the second statement are averaged to obtain the average vectors of the first statement and the second statement.
[0053] In step S205, the difference between the average vector of the first statement and the average vector of the second statement is taken as the first intermediate vector, the dot product of the average vector of the first statement and the average vector of the second statement is taken as the second intermediate vector, and the element-wise product of the average vector of the first statement and the average vector of the second statement is taken as the third intermediate vector.
[0054] In step S206, the first intermediate vector, the second intermediate vector, and the third intermediate vector are connected in parallel to obtain the classification vector.
[0055] In step S207, a linear classifier is used to classify the classification vectors to obtain the relationship between the first statement and the second statement.
[0056] In step S208, the semantics of the text are determined based on the relationship between the first statement and the second statement.
[0057] In this embodiment of the application, the text semantic determination method can be executed by a server or a terminal device with a certain computing power. For ease of explanation and description, the following description uses the method being executed by a server as an example.
[0058] In this embodiment, the first and second sentences in the text are first obtained, and then encoded to obtain first sentence encoding vectors and second sentence encoding vectors respectively. Next, a cross-attention neural network layer is used to process the first and second sentence encoding vectors to obtain first sentence mutual attention vectors and second sentence mutual attention vectors. Then, the first and second sentence mutual attention vectors are averaged to obtain first sentence average vectors and second sentence average vectors. Then, the difference between the first and second sentence average vectors is used as the first intermediate vector, the dot product of the first and second sentence average vectors is used as the second intermediate vector, and the element-wise product of the first and second sentence average vectors is used as the third intermediate vector. The first, second, and third intermediate vectors are concatenated in parallel to obtain a classification vector. A linear classifier is used to classify the classification vector to obtain the relationship between the first and second sentences. Finally, the semantics of the text are determined based on the relationship between the first and second sentences. The relationship between the first and second sentences can be an implication relationship, a neutral relationship, or a contradictory relationship.
[0059] In one example, the first statement can be statement A, and the second statement can be statement B. The calculated average vector of the first statement can be VA, and the average vector of the second statement can be VB. The difference between the first statement's average vector VA and the second statement's average vector VB, VA-VB, is calculated and used as the first intermediate vector. The inner product of the first statement's average vector VA and the second statement's average vector VB, VA*VB, is calculated and used as the second intermediate vector. The element-wise product of the first statement's average vector VA and the second statement's average vector VB, VA×VB, is calculated and used as the third intermediate vector. These three intermediate vectors are concatenated in parallel to obtain the classification vector VAB = Concatenate(VA-VB, VA*VB, VA×VB), where Concatenate is the join function.
[0060] Furthermore, by inputting the classification vector into a trained linear classifier, the relationship between the first and second sentences can be determined, thereby determining the semantics of the text. The linear classifier can be trained using supervised training methods, the specific training methods of which will be described in detail later and will not be repeated here.
[0061] According to the technical solution provided in the embodiments of this application, by performing cross-attention processing on the sentences in the text and using the processed mutual attention vector to calculate the relationship between the sentences, the semantics of the text is determined based on the relationship between the sentences, thereby improving the ability to grasp the subtle differences in the sentences during text semantic recognition and improving the generalization ability.
[0062] Figure 3 This is a flowchart illustrating a method provided in this application for encoding a first statement and a second statement respectively to obtain encoding vectors for the first statement and the second statement. Figure 3 As shown, the method includes the following steps:
[0063] In step S301, the number of characters k1 of the first statement and the number of characters k2 of the second statement are obtained.
[0064] In step S302, the first and second sentences are encoded using a pre-trained language representation model to obtain the first sentence encoding vector, which includes k1 N-dimensional vectors, and the second sentence encoding vector, which includes k2 N-dimensional vectors.
[0065] Where N is a positive integer.
[0066] In this embodiment of the application, when encoding the first statement and the second statement respectively, the number of words k1 of the first statement and the number of words k2 of the second statement can be obtained first. Then, a pre-trained language representation model is used to encode the first statement and the second statement respectively to obtain the first statement encoding vector including k1 N-dimensional vectors and the second statement encoding vector including k2 N-dimensional vectors.
[0067] In one example, if the first sentence contains 5 characters (k1 = 5) and the second sentence contains 6 characters (k2 = 6), and further, if N is 768, then a pre-trained language representation model, such as BERT (Bidirectional Encoder Representation from Transformers), can be used to encode the first sentence into 5 first sentence encoding vectors of 768 dimensions, and the second sentence into 6 second sentence encoding vectors of 768 dimensions.
[0068] Figure 4 This is a flowchart illustrating a method provided in this application for processing the first statement encoding vector and the second statement encoding vector using a cross-attention neural network layer to obtain the first statement mutual attention vector and the second statement mutual attention vector. Figure 4 As shown, the method includes the following steps:
[0069] In step S401, the first statement encoding vector and the second statement encoding vector are concatenated to obtain k1+k2 N-dimensional vectors.
[0070] In step S402, a cross-attention neural network layer is used to process k1+k2 N-dimensional vectors to obtain k1+k2 N-dimensional vectors after interactive attention processing.
[0071] In step S403, k1 N-dimensional first statement mutual attention vectors and k2 N-dimensional second statement mutual attention vectors are determined based on k1+k2 N-dimensional vectors after interactive attention processing.
[0072] In this embodiment, the first statement encoding vector and the second statement encoding vector can be concatenated to obtain k1+k2 N-dimensional vectors. Then, a cross-attention neural network layer is used to process the k1+k2 N-dimensional vectors to obtain k1+k2 N-dimensional vectors after interactive attention processing. Finally, based on the k1+k2 N-dimensional vectors after interactive attention processing, k1 N-dimensional first statement mutual attention vectors and k2 N-dimensional second statement mutual attention vectors are determined.
[0073] Continuing with the example of the first statement encoding vector being 5 x 768-dimensional vectors and the second statement encoding vector being 6 x 768-dimensional vectors, a total of 11 vectors can be arranged in a row and fed into a cross-attention neural network layer for processing, resulting in 11 results after interactive attention processing. The mutual attention vectors for the first statement are also 5 x 768-dimensional vectors, and the mutual attention vectors for the second statement are also 6 x 768-dimensional vectors. Averaging the 5 x 768-dimensional mutual attention vectors of the first statement yields the average vector VA, and averaging the 6 x 768-dimensional mutual attention vectors yields the average vector VB.
[0074] Figure 5 This is a flowchart illustrating a method provided in this application for classifying classification vectors using a linear classifier to obtain the relationship between the first and second statements. Figure 5 As shown, the method includes the following steps:
[0075] In step S501, the classification loss function of the linear classifier is obtained.
[0076] In step S502, the mutual attention vectors of the first statement and the mutual attention vectors of the second statement are averaged to obtain the first interaction representation vector between the first statement and the second statement.
[0077] In step S503, a second interaction representation vector between the first statement and at least one third statement is determined.
[0078] In step S504, the first interaction representation vector and the second interaction representation vector are compared and learned to obtain the learning loss function.
[0079] In step S505, the classification loss function and the learning loss function are weighted and summed to obtain the target loss function.
[0080] In step S506, in response to the target loss function being less than or equal to a preset threshold, the classification result of the linear classifier is used as the relationship between the first statement and the second statement.
[0081] In this embodiment, the linear classifier can be a pre-trained classifier, and its training method may include: firstly, obtaining the classification loss function L1 of the linear classifier; then, averaging the mutual attention vectors of the first and second statements to obtain the first interaction representation vector (VAB) between the first and second statements. Taking the example that the first statement mutual attention vector consists of 5 768-dimensional vectors and the second statement encoding vector consists of 6 768-dimensional vectors, the average of these 11 mutual attention vectors can be calculated to obtain the first interaction representation vector (VAB) between the first and second statements.
[0082] Next, the second interaction representation vector between the first statement and at least one third statement can be determined. Simultaneously, using the same method as calculating the first interaction representation vector between the first and second statements, the second interaction representation vector between the first and third statements is calculated. The third statement can include, for example, statement C, statement D, etc., and the calculated second interaction representation vector can be VAC, VAD, etc. By comparing and learning the first and second interaction representation vectors, the learning loss function L2 can be obtained.
[0083] Furthermore, the classification loss function and the learning loss function can be weighted and summed to obtain the target loss function. Specifically, the target loss function can be calculated using the formula L = L1 + αL2, where L is the target loss function and α is an adjustable hyperparameter, which is greater than 0 and less than 1.
[0084] Finally, the target loss function can be compared with a preset threshold. If the target loss function is greater than the preset threshold, the parameters of the linear classifier are updated, and L1 and L2 are recalculated until the target loss function is less than or equal to the preset threshold. At this point, the classification result of the linear classifier can be used as the relationship between the first and second statements.
[0085] The technical solution adopted in this application uses a contrastive learning-based method to model the similarity of text at the sentence-pair level, thereby improving the ability to grasp the details in sentences; and uses a sentence-pair level similarity model to improve the generalization performance of the text similarity model.
[0086] Figure 6 This is a flowchart illustrating a method for comparing and learning a first interaction representation vector and a second interaction representation vector to obtain a learning loss function, as provided in an embodiment of this application. Figure 6 As shown, the method includes the following steps:
[0087] In step S601, the first interaction representation vector and the second interaction representation vector are compared and learned using triplet loss.
[0088] In step S602, in response to the fact that the relationship between the first statement and the second statement is the same as the relationship between the first statement and the third statement, the distance between the first interaction representation vector and the second interaction representation vector is reduced.
[0089] In step S603, in response to the opposite relationship between the first statement and the second statement and the first statement and the third statement, the distance between the first interaction representation vector and the second interaction representation vector is increased.
[0090] In this embodiment, when comparing and learning the first interaction representation vector and the second interaction representation vector to obtain the learning loss function, a triplet loss can be used for the comparative learning of the first and second interaction representation vectors. Specifically, when the relationship between the first and second statements is the same as the relationship between the first and third statements, the distance between the first and second interaction representation vectors can be reduced. When the relationship between the first and second statements is opposite to the relationship between the first and third statements, the distance between the first and second interaction representation vectors can be increased.
[0091] Taking the first interaction representation vector as VAB and the second interaction representation vectors as VAC and VAD as an example, further, if the first statement A and the second statement B are contradictory, the first statement A and the third statement C are also contradictory, and the first statement A and the third statement D are implied, then when comparing and learning the first interaction representation vector and the second interaction representation vector, the distance between the first interaction representation vector VAB and the second interaction representation vector VAC can be reduced, and the distance between the first interaction representation vector VAB and the second interaction representation vector VAD can be increased, thus obtaining the learning loss function L2.
[0092] Figure 7 This is a flowchart illustrating a method for determining the semantics of text based on the relationship between a first statement and a second statement, as provided in an embodiment of this application. Figure 7 As shown, the method includes the following steps:
[0093] In step S701, the text is traversed to determine the relationship between any two sentences in the text.
[0094] In step S702, the semantics of the text are determined using a three-classification model based on the relationship between any two sentences in the text.
[0095] In this embodiment, the text can be traversed to determine the relationship between any two sentences. Then, based on the relationship between any two sentences, a three-classification model is used to determine the semantics of the text.
[0096] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.
[0097] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.
[0098] Figure 8 This is a schematic diagram of a text semantic determination device provided in an embodiment of this application. Figure 8 As shown, the text semantic determination device includes:
[0099] The acquisition module 801 is configured to acquire the first and second statements in the text.
[0100] The encoding module 802 is configured to encode the first statement and the second statement respectively to obtain the first statement encoding vector and the second statement encoding vector.
[0101] The cross-attention module 803 is configured to use a cross-attention neural network layer to process the first statement encoding vector and the second statement encoding vector to obtain the first statement mutual attention vector and the second statement mutual attention vector.
[0102] The calculation module 804 is configured to perform average calculations on the mutual attention vectors of the first statement and the mutual attention vectors of the second statement, respectively, to obtain the average vector of the first statement and the average vector of the second statement.
[0103] The calculation module 804 is also configured to use the difference between the average vector of the first statement and the average vector of the second statement as the first intermediate vector, the dot product of the average vector of the first statement and the average vector of the second statement as the second intermediate vector, and the element-wise product of the average vector of the first statement and the average vector of the second statement as the third intermediate vector.
[0104] The calculation module 804 is also configured to connect the first intermediate vector, the second intermediate vector, and the third intermediate vector in parallel to obtain a classification vector.
[0105] The classification module 805 is configured to classify the classification vectors using the existing classifier to obtain the relationship between the first statement and the second statement.
[0106] The determination module 806 is configured to determine the semantics of the text based on the relationship between the first statement and the second statement.
[0107] According to the technical solution provided in the embodiments of this application, by performing cross-attention processing on the sentences in the text and using the processed mutual attention vector to calculate the relationship between the sentences, the semantics of the text is determined based on the relationship between the sentences, thereby improving the ability to grasp the subtle differences in the sentences during text semantic recognition and improving the generalization ability.
[0108] In this embodiment of the application, the first statement and the second statement are encoded respectively to obtain the first statement encoding vector and the second statement encoding vector, including: obtaining the number of words k1 of the first statement and the number of words k2 of the second statement; using a pre-trained language representation model to encode the first statement and the second statement respectively to obtain the first statement encoding vector including k1 N-dimensional vectors and the second statement encoding vector including k2 N-dimensional vectors; where N is a positive integer.
[0109] In this embodiment of the application, a cross-attention neural network layer is used to process the first statement encoding vector and the second statement encoding vector to obtain the first statement mutual attention vector and the second statement mutual attention vector. This includes: concatenating the first statement encoding vector and the second statement encoding vector to obtain k1+k2 N-dimensional vectors; processing the k1+k2 N-dimensional vectors using the cross-attention neural network layer to obtain k1+k2 N-dimensional vectors after interactive attention processing; and determining k1 N-dimensional first statement mutual attention vectors and k2 N-dimensional second statement mutual attention vectors based on the k1+k2 N-dimensional vectors after interactive attention processing.
[0110] In this embodiment, a linear classifier is used to classify the classification vectors to obtain the relationship between the first statement and the second statement. This includes: obtaining the classification loss function of the linear classifier; averaging the mutual attention vectors of the first and second statements to obtain a first interaction representation vector between the first and second statements; determining a second interaction representation vector between the first statement and at least one third statement; performing comparative learning on the first and second interaction representation vectors to obtain a learning loss function; weighted summing the classification loss function and the learning loss function to obtain a target loss function; and, in response to the target loss function being less than or equal to a preset threshold, using the classification result of the linear classifier as the relationship between the first and second statements.
[0111] In this embodiment of the application, a learning loss function is obtained by comparing and learning the first interaction representation vector and the second interaction representation vector, including: using triplet loss to compare and learn the first interaction representation vector and the second interaction representation vector; reducing the distance between the first interaction representation vector and the second interaction representation vector in response to the same relationship between the first statement and the second statement and the same relationship between the first statement and the third statement; and increasing the distance between the first interaction representation vector and the second interaction representation vector in response to the opposite relationship between the first statement and the second statement and the opposite relationship between the first statement and the third statement.
[0112] In this embodiment of the application, determining the semantics of the text based on the relationship between the first statement and the second statement includes: traversing the text to determine the relationship between any two sentences in the text; and using a three-classification model to determine the semantics of the text based on the relationship between any two sentences in the text.
[0113] In this embodiment of the application, the relationship between the first statement and the second statement includes: the first statement and the second statement have an implication relationship; or the first statement and the second statement have a neutral relationship; or the first statement and the second statement have a contradictory relationship.
[0114] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0115] Figure 9 This is a schematic diagram of an electronic device provided in an embodiment of this application. Figure 9 As shown, the electronic device 9 of this embodiment includes a processor 901, a memory 902, and a computer program 903 stored in the memory 902 and executable on the processor 901. When the processor 901 executes the computer program 903, it implements the steps in the various method embodiments described above. Alternatively, when the processor 901 executes the computer program 903, it implements the functions of each module / unit in the various device embodiments described above.
[0116] Electronic device 9 can be a desktop computer, laptop, handheld computer, cloud server, or other electronic device. Electronic device 9 may include, but is not limited to, processor 901 and memory 902. Those skilled in the art will understand that... Figure 9 This is merely an example of electronic device 9 and does not constitute a limitation on electronic device 9. It may include more or fewer components than shown, or different components.
[0117] The processor 901 can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
[0118] The memory 902 can be an internal storage unit of the electronic device, such as a hard disk or RAM of the electronic device 9. The memory 902 can also be an external storage device of the electronic device 9, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the electronic device 9. The memory 902 can also include both internal and external storage units of the electronic device 9. The memory 902 is used to store computer programs and other programs and data required by the electronic device.
[0119] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0120] If integrated modules / units are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program may include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. Computer-readable media may include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.
[0121] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.
Claims
1. A method of text semantic determination, characterized by, include: Extract the first and second statements from the text; Encode the first statement and the second statement respectively to obtain the first statement encoding vector and the second statement encoding vector; The first statement encoding vector and the second statement encoding vector are processed using a cross-attention neural network layer to obtain the first statement mutual attention vector and the second statement mutual attention vector. The average of the mutual attention vectors of the first statement and the second statement is calculated to obtain the average vector of the first statement and the average vector of the second statement. The difference between the average vector of the first statement and the average vector of the second statement is used as the first intermediate vector, the dot product of the average vector of the first statement and the average vector of the second statement is used as the second intermediate vector, and the element-wise product of the average vector of the first statement and the average vector of the second statement is used as the third intermediate vector. The first intermediate vector, the second intermediate vector, and the third intermediate vector are connected in parallel to obtain the classification vector; The classification vector is classified using a linear classifier to obtain the relationship between the first statement and the second statement; The semantics of the text are determined based on the relationship between the first statement and the second statement; The step of classifying the classification vectors using a linear classifier to obtain the relationship between the first statement and the second statement includes: obtaining the classification loss function of the linear classifier; averaging the mutual attention vectors of the first and second statements to obtain a first interaction representation vector between the first and second statements; determining a second interaction representation vector between the first statement and at least one third statement; performing comparative learning on the first and second interaction representation vectors to obtain a learning loss function; weighted summing the classification loss function and the learning loss function to obtain a target loss function; and, in response to the target loss function being less than or equal to a preset threshold, using the classification result of the linear classifier as the relationship between the first and second statements. The step of comparing and learning the first interaction representation vector and the second interaction representation vector to obtain a learning loss function includes: using triplet loss to compare and learn the first interaction representation vector and the second interaction representation vector; reducing the distance between the first interaction representation vector and the second interaction representation vector in response to the same relationship between the first statement and the second statement and the same relationship between the first statement and the third statement; and increasing the distance between the first interaction representation vector and the second interaction representation vector in response to the opposite relationship between the first statement and the second statement and the same relationship between the first statement and the third statement.
2. The method of claim 1, wherein, The process of encoding the first statement and the second statement respectively to obtain the first statement encoding vector and the second statement encoding vector includes: Get the number of characters k1 in the first statement and the number of characters k2 in the second statement; The first and second sentences are encoded using a pre-trained language representation model, resulting in a first sentence encoding vector consisting of k1 N-dimensional vectors and a second sentence encoding vector consisting of k2 N-dimensional vectors. Where N is a positive integer.
3. The method of claim 2, wherein, The process of using a cross-attention neural network layer to process the first statement encoding vector and the second statement encoding vector to obtain the first statement mutual attention vector and the second statement mutual attention vector includes: Concatenate the first statement encoding vector and the second statement encoding vector to obtain k1+k2 N-dimensional vectors; The k1+k2 N-dimensional vectors are processed using a cross-attention neural network layer to obtain k1+k2 N-dimensional vectors after interactive attention processing. Based on the k1+k2 N-dimensional vectors processed by interactive attention, determine k1 N-dimensional first statement mutual attention vectors and k2 N-dimensional second statement mutual attention vectors.
4. The method of claim 1, wherein, Determining the semantics of the text based on the relationship between the first statement and the second statement includes: Traverse the text to determine the relationship between any two sentences in the text; Based on the relationship between any two sentences in the text, a three-classification model is used to determine the semantics of the text.
5. The method according to any one of claims 1 to 4, characterized in that, The relationship between the first statement and the second statement includes: The first and second statements have an implication relationship; or The first and second statements are in a neutral relationship; or The first statement and the second statement are contradictory.
6. A text semantic determination apparatus characterized by comprising: include: The acquisition module is configured to acquire the first and second statements in the text. The encoding module is configured to encode the first statement and the second statement respectively to obtain the first statement encoding vector and the second statement encoding vector; The cross-attention module is configured to use a cross-attention neural network layer to process the first statement encoding vector and the second statement encoding vector to obtain the first statement mutual attention vector and the second statement mutual attention vector. The calculation module is configured to perform average calculations on the first statement mutual attention vector and the second statement mutual attention vector respectively to obtain the first statement average vector and the second statement average vector. The calculation module is further configured to use the difference between the average vector of the first statement and the average vector of the second statement as the first intermediate vector, the dot product of the average vector of the first statement and the average vector of the second statement as the second intermediate vector, and the element-wise product of the average vector of the first statement and the average vector of the second statement as the third intermediate vector. The calculation module is also configured to connect the first intermediate vector, the second intermediate vector, and the third intermediate vector in parallel to obtain a classification vector; The classification module is configured to use a linear classifier to classify the classification vector to obtain the relationship between the first statement and the second statement; The determination module is configured to determine the semantics of the text based on the relationship between the first statement and the second statement; The step of classifying the classification vectors using a linear classifier to obtain the relationship between the first statement and the second statement includes: obtaining the classification loss function of the linear classifier; averaging the mutual attention vectors of the first and second statements to obtain a first interaction representation vector between the first and second statements; determining a second interaction representation vector between the first statement and at least one third statement; performing comparative learning on the first and second interaction representation vectors to obtain a learning loss function; weighted summing the classification loss function and the learning loss function to obtain a target loss function; and, in response to the target loss function being less than or equal to a preset threshold, using the classification result of the linear classifier as the relationship between the first and second statements. The step of comparing and learning the first interaction representation vector and the second interaction representation vector to obtain a learning loss function includes: using triplet loss to compare and learn the first interaction representation vector and the second interaction representation vector; reducing the distance between the first interaction representation vector and the second interaction representation vector in response to the same relationship between the first statement and the second statement and the same relationship between the first statement and the third statement; and increasing the distance between the first interaction representation vector and the second interaction representation vector in response to the opposite relationship between the first statement and the second statement and the same relationship between the first statement and the third statement.
7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 5.
8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 5.
Citation Information
Patent Citations
Text matching method fusing multi-head attention alignment
CN115408494A
Natural language reasoning model training method, reasoning method, terminal and medium
CN115659950A
