Robust spoken language knowledge extraction method and system based on contrastive learning and attention mechanism

Abstract

The application provides a robust spoken language knowledge extraction method and system based on contrastive learning and an attention mechanism, comprising the following steps: S1: extracting joint sentence-level representations of writing dialogue history text and external knowledge by using BERT; S2: extracting joint sentence-level representations of spoken dialogue history text output by a speech recognition module corresponding to the writing dialogue history text and external knowledge by using BERT; S3: fusing multiple joint sentence-level representations of spoken dialogue history text and external knowledge into one representation by using an attention mechanism; and S4: selecting correct external knowledge based on the fused joint representation of spoken dialogue history text and external knowledge and the joint sentence-level representations of writing dialogue history text and external knowledge by using contrastive learning.

Description

Technical Field

[0001] This invention relates to the field of task-oriented spoken dialogue systems, specifically to a robust spoken knowledge extraction method and system based on contrastive learning and attention mechanisms, and more specifically to a method and system for extracting external knowledge in spoken dialogue scenarios based on contrastive learning and attention mechanisms. Background Technology

[0002] Task-oriented dialogue systems aim to help users complete specific tasks through conversation. Voice-based task-oriented dialogue systems are the most popular because the dialogue contains natural and rich information. Unlike written text-based dialogue systems, these systems rely heavily on the performance of automatic speech recognition systems and external knowledge bases. Therefore, designing a system that can achieve high accuracy in acquiring external knowledge in human-computer spoken dialogue environments is an important and challenging task.

[0003] Patent document CN1 15392263A (application number CN202211023548.9) discloses a knowledge-selection-based dialogue model. The model includes: an encoding module comprising a text encoder and a knowledge encoder. The text encoder encodes the questioner's statement to obtain an encoding vector, and the knowledge encoder encodes knowledge in a knowledge base to obtain a knowledge encoding vector. A knowledge selection module selects knowledge from the knowledge base based on the encoding vector and the knowledge encoding vector obtained by the encoding module. This knowledge selection module is trained using reinforcement learning, with the dialogue text and its context as input and the selected knowledge as output. A decoding module decodes the knowledge selected by the knowledge selection module and the encoding vector of the questioner's statement to generate a response statement to the questioner's statement. This invention proposes a knowledge-selection dialogue model based on the encoding module, knowledge selection module, and decoding module. However, it does not consider the influence of spoken text on the system, which may lead to the system's inability to select the correct external knowledge in spoken dialogue scenarios. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the purpose of this invention is to provide a robust spoken language knowledge extraction method and system based on contrastive learning and attention mechanisms.

[0005] A robust spoken knowledge extraction method based on contrastive learning and attention mechanisms, provided by the present invention, includes:

[0006] Step S1: Use BERT to extract a joint sentence-level representation of the writing dialogue history text and external knowledge;

[0007] Step S2: Use BERT to extract the joint sentence-level representation of the spoken dialogue history text output by the speech recognition module corresponding to the written dialogue history text and external knowledge.

[0008] Step S3: Use attention mechanisms to fuse the joint sentence-level representations of multiple spoken dialogue history texts and external knowledge into a single representation;

[0009] Step S4: Based on the fused spoken dialogue history text-external knowledge joint representation and the written dialogue history text and external knowledge joint sentence-level representation, select the correct external knowledge using contrastive learning.

[0010] Preferably, step S1 employs:

[0011] Step S1.1: Concatenate the writing dialogue history text and C external knowledge points used for training the knowledge selection model; represented as: {[W t [k1], ..., [W] t k C ]}; where W t Represents the writing dialogue history text; {k1, ..., k C} represents C pieces of external knowledge;

[0012] Step S1.2: Utilize BERT to extract joint sentence-level representations from the concatenation of writing dialogue history text and external knowledge; represented as: Among them, h w，i With [W] t k i One-to-one correspondence, i = 1, ..., C; d represents each representation vector h w，i A vector of dimension d*1.

[0013] Preferably, step S2 employs:

[0014] Step S2.1: Extract multiple outputs from the speech recognition module to obtain multiple segments of spoken dialogue history text; represented as: Where N represents the number of outputs;

[0015] Step S2.2: Concatenate the extracted multiple segments of spoken dialogue history text with C external knowledge points used for training the knowledge selection model to obtain the concatenation of spoken dialogue history text and external knowledge; represented as:

[0016] Step S2.3: Use BERT to extract representations from the cascade of spoken dialogue history text and external knowledge; represented as: in, and One-to-one correspondence, j = 1, ..., C.

[0017] Preferably, step S3 employs the following methods:

[0018] For a given external knowledge j, we obtain a joint sentence-level representation of N spoken dialogue historical texts and external knowledge. The attention mechanism is used to fuse the current N representations; represented as:

[0019]

[0020] in, It is a completely one vector; These are the learnable parameters of the attention mechanism module; the joint sentence-level representation of the C spoken dialogue history texts and external knowledge after attention mechanism fusion is as follows:

[0021] Preferably, step S4 employs the following methods:

[0022] Step S4.1: Construct positive and negative sample pairs corresponding to the writing dialogue history text-external knowledge representation;

[0023] Step S4.2: Construct positive and negative sample pairs corresponding to the fused spoken dialogue history text and external knowledge representation;

[0024] Step S4.3: Use contrastive learning to guide the selection of correct external knowledge in the written dialogue history text-external knowledge representation.

[0025] Preferably, step S4.1 employs the condition that: if and only if indices i and j point to the same correct external knowledge. This constitutes a pair of positive samples;

[0026] The set of negative samples belonging to the writing text domain is as follows: The set of negative samples belonging to the spoken text domain is as follows: The final set of all negative samples can be written as:

[0027] Step S4.2 adopts the following:

[0028] If and only if the indices i and j point to the same correct external knowledge. This constitutes a pair of positive samples;

[0029] The set of negative samples belonging to the spoken text domain is as follows: The set of negative samples belonging to the writing text domain is: The final set of all negative samples can be written as:

[0030] Preferably, step S4.3 employs: using contrastive learning to guide the selection of correct external knowledge by the written dialogue history text-external knowledge representation;

[0031] The total loss used in the entire knowledge selection model is: Training was conducted to guide the selection of correct external knowledge from the fused spoken dialogue history text-external knowledge representation.

[0032] The loss function for comparative learning using the constructed positive and negative sample pairs is shown in the following formula:

[0033]

[0034] Characterize H w and Passing through linear layers respectively The classification probabilities of obtaining the joint representation of the written dialogue history text and external knowledge, and the joint representation of the spoken dialogue history text and external knowledge, are denoted as p, respectively. w and p s :

[0035]

[0036] The training of the knowledge selection model is supervised by a loss function, and the unique correct knowledge is selected from C external candidate knowledge.

[0037]

[0038] Where, p w，g p represents the probability of selecting the correct knowledge using the joint representation of written text, dialogue history, and external knowledge; s，g This represents the probability of selecting the correct knowledge using the fused spoken text dialogue history-external knowledge joint representation.

[0039] A robust spoken language knowledge extraction system based on contrastive learning and attention mechanisms, provided by the present invention, includes:

[0040] Module M1: Utilizes BERT to extract joint sentence-level representations of writing dialogue history text and external knowledge;

[0041] Module M2: Utilizes BERT to extract the joint sentence-level representation of the spoken dialogue history text output by the speech recognition module corresponding to the written dialogue history text and external knowledge;

[0042] Module M3: Utilizes an attention mechanism to fuse joint sentence-level representations of multiple spoken dialogue history texts and external knowledge into a single representation;

[0043] Module M4: Based on the fused spoken dialogue history text-external knowledge joint representation and the written dialogue history text and external knowledge joint sentence-level representation, the correct external knowledge is selected using contrastive learning.

[0044] Preferably, module M1 adopts:

[0045] Module M1.1: Concatenates the writing dialogue history text and C external knowledge sources used for training the knowledge selection model; represented as: {[W t [k1], ..., [W] t k C ]}; where W t Represents the writing dialogue history text; {k1, ..., k C} represents C pieces of external knowledge;

[0046] Module M1.2: Utilizes BERT to extract joint sentence-level representations from the concatenation of writing dialogue history text and external knowledge; represented as: Among them, h w，i With [W] t k i One-to-one correspondence, i = 1, ..., C; d represents each representation vector h w，i A vector of size d*1 with a dimension of d.

[0047] The module M2 adopts:

[0048] Module M2.1: Extracts multiple outputs from the speech recognition module to obtain multiple segments of spoken dialogue history text; represented as: Where N represents the number of outputs;

[0049] Module M2.2: Concatenates the extracted spoken dialogue history texts with C external knowledge sources used for training the knowledge selection model to obtain the spoken dialogue history text-external knowledge concatenation; represented as:

[0050] Module M2.3: Utilizes BERT to extract representations from the cascade of spoken dialogue history text and external knowledge; represented as: in, and One-to-one correspondence, j = 1, ..., C;

[0051] The module M3 adopts:

[0052] For a given external knowledge j, we obtain a joint sentence-level representation of N spoken dialogue historical texts and external knowledge. The attention mechanism is used to fuse the current N representations; represented as:

[0053]

[0054] in, It is a completely one vector; These are the learnable parameters of the attention mechanism module; the joint sentence-level representation of the C spoken dialogue history texts and external knowledge after attention mechanism fusion is as follows:

[0055] Preferably, the module M4 adopts:

[0056] Module M4.1: Construct positive and negative sample pairs corresponding to the writing dialogue history text-external knowledge representation;

[0057] Module M4.2: Construct positive and negative sample pairs corresponding to the fused spoken dialogue history text and external knowledge representation;

[0058] Module M4.3: Using contrastive learning to guide the selection of correct external knowledge from written dialogue history text-external knowledge representation;

[0059] Module M4.1 adopts the following condition: If and only if indices i and j point to the same correct external knowledge. This constitutes a pair of positive samples;

[0060] The set of negative samples belonging to the writing text domain is as follows: The set of negative samples belonging to the spoken text domain is as follows: The final set of all negative samples can be written as:

[0061] The module M4.2 adopts:

[0062] If and only if the indices i and j point to the same correct external knowledge. This constitutes a pair of positive samples;

[0063] The set of negative samples belonging to the spoken text domain is as follows: The set of negative samples belonging to the writing text domain is: The final set of all negative samples can be written as:

[0064] Module M4.3 employs the following approach: using contrastive learning to guide the selection of correct external knowledge from the written dialogue history text-external knowledge representation.

[0065] The total loss used in the entire knowledge selection model is: Training was conducted to guide the selection of correct external knowledge from the fused spoken dialogue history text-external knowledge representation.

[0066] The loss function for comparative learning using the constructed positive and negative sample pairs is shown in the following formula:

[0067]

[0068] Characterize H w and Passing through linear layers respectively The classification probabilities of obtaining the joint representation of the written dialogue history text and external knowledge, and the joint representation of the spoken dialogue history text and external knowledge, are denoted as p, respectively. w and p s :

[0069]

[0070] The training of the knowledge selection model is supervised by a loss function, and the unique correct knowledge is selected from C external candidate knowledge.

[0071]

[0072] Where, p w，g p represents the probability of selecting the correct knowledge using the joint representation of written text, dialogue history, and external knowledge; s，g This represents the probability of selecting the correct knowledge using the fused spoken text dialogue history-external knowledge joint representation.

[0073] Compared with the prior art, the present invention has the following beneficial effects:

[0074] 1. This invention provides a method for extracting external knowledge in spoken dialogue scenarios based on contrastive learning and attention mechanisms, which effectively enhances the accuracy of dialogue systems in selecting external knowledge in spoken dialogue scenarios;

[0075] 2. This invention utilizes an attention mechanism to fuse multiple outputs of the automatic speech recognition module into one, aggregating information from multiple outputs;

[0076] 3. This invention utilizes comparative learning, using clean written text dialogue history to guide the selection of correct external knowledge from corresponding spoken text dialogue history;

[0077] 4. This invention utilizes an attention mechanism to fuse multidimensional information output by the speech recognition module, and uses the written text dialogue history-external knowledge representation to guide the fused spoken dialogue history-external knowledge representation to select the correct external knowledge. Compared with other methods, it can select external knowledge more accurately. Attached Figure Description

[0078] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0079] Figure 1 A flowchart of a preferred embodiment of the present invention, showing a method and system for extracting external knowledge in a spoken dialogue scenario based on contrastive learning and attention mechanisms;

[0080] Figure 2 A schematic diagram of the structure of an external knowledge extraction method and system based on contrastive learning and attention mechanisms in a spoken dialogue scenario, provided by a preferred embodiment of the present invention;

[0081] Figure 3 The schematic diagram of a preferred embodiment of the present invention provides a method and system for extracting external knowledge in spoken dialogue scenarios based on contrastive learning and attention mechanisms. Detailed Implementation

[0082] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0083] Example 1

[0084] According to the present invention, a method for extracting external knowledge in spoken dialogue scenarios based on contrastive learning and attention mechanisms is provided, such as... Figure 1 As shown, it includes:

[0085] BERT is used to extract joint sentence-level representations of writing dialogue history text and external knowledge;

[0086] BERT is used to extract the spoken dialogue history text output by the speech recognition module corresponding to the written dialogue history text and the sentence-level representation of external knowledge.

[0087] We use an attention mechanism to fuse the joint sentence-level representations of multiple spoken dialogue history texts and external knowledge into a single representation;

[0088] By using contrastive learning and writing dialogue historical texts-external knowledge representations, the fused representations are guided to select the correct external knowledge.

[0089] Specifically, the speech recognition module is abbreviated as ASR. ASR takes speech as input and the recognized content as output. Since ASR generates an estimate of the letter corresponding to the current pronunciation based on information such as phonemes in the speech signal, this estimate is not 100% accurate, but has a confidence score. ASR will arrange these estimates in order from high to low, thus producing multiple outputs.

[0090] Specifically, the writing dialogue history text and C external knowledge points used for model training are first concatenated, and the formula can be written as: {[W t [k1], ..., [W] t k C ]}. Among them, W t Represents the writing dialogue history text {k1, ..., k} C} represents C pieces of knowledge.

[0091] The joint sentence-level representation extracted using BERT from the concatenation of writing dialogue history text and external knowledge can be represented as follows: Where h w，i With [W] t k i There is a one-to-one correspondence, i = 1, ..., C.

[0092] Next, multiple outputs from the speech recognition module are extracted to obtain multiple segments of spoken dialogue history text, which can be represented by the following formula: Where N represents the number of outputs. Multiple spoken dialogue historical texts are concatenated with C external knowledge sources used for model training, resulting in C×N input samples, which can be written as:

[0093] By leveraging BERT to extract representations from the concatenation of spoken dialogue history text and external knowledge, the joint sentence-level representation of spoken dialogue history text and external knowledge extracted by BERT can be used for writing.

[0094] For a given external knowledge j, we obtain a joint sentence-level representation of N spoken dialogue historical texts and external knowledge. Then, an attention mechanism is used to fuse these N representations. The fusion process can be written as a formula:

[0095]

[0096] in It is a completely one vector. These are the learnable parameters of the attention mechanism module. The joint sentence-level representation of C spoken dialogue history texts and external knowledge after fusion through the attention mechanism can be written as:

[0097] Characterize H w and Passing through linear layers respectively The classification probabilities of obtaining the joint representation of the written dialogue history text and external knowledge, and the joint representation of the spoken dialogue history text and external knowledge, are denoted as p, respectively. w and p s :

[0098]

[0099] The training of the network can be supervised using the following loss function to select the unique correct knowledge from C external candidate knowledge.

[0100]

[0101] Finally, the writing dialogue history text - external knowledge representation h w，g Constructing a positive sample pair: if and only if the indices i and j point to the same correct external knowledge. This constitutes a pair of positive samples; for writing dialogue history text - external knowledge representation h w，g Constructing negative sample pairs: The set of negative samples belonging to the same writing text domain is as follows: The set of negative samples belonging to the spoken text domain is as follows: The final set of all negative samples can be written as:

[0102] Representation of spoken dialogue history texts as external knowledge Constructing a positive sample pair: if and only if the indices i and j point to the same correct external knowledge. This constitutes a pair of positive samples. It represents the spoken dialogue history text as an external knowledge representation. Constructing negative sample pairs: The set of negative samples belonging to the spoken text domain is as follows: The set of negative samples belonging to the writing text domain is: The final set of all negative samples can be written as:

[0103] Finally, contrastive learning is used to guide the selection of correct external knowledge from the written dialogue history text - external knowledge representation. The loss function for contrastive learning is designed using the constructed positive and negative sample pairs, as shown in the following formula:

[0104]

[0105] The total loss can be used for the entire network: The network is trained to guide the selection of correct external knowledge from the fused spoken dialogue history text-external knowledge representation.

[0106] Example 2

[0107] Example 2 is a preferred example of Example 1.

[0108] According to the present invention, an external knowledge extraction system for spoken dialogue scenarios based on contrastive learning and attention mechanisms is provided, such as... Figure 2-3 As shown, it includes:

[0109] BERT is used to extract joint sentence-level representations of writing dialogue history text and external knowledge;

[0110] BERT is used to extract the spoken dialogue history text output by the speech recognition module corresponding to the written dialogue history text and the sentence-level representation of external knowledge.

[0111] We use an attention mechanism to fuse the joint sentence-level representations of multiple spoken dialogue history texts and external knowledge into a single representation;

[0112] By using contrastive learning and writing dialogue historical texts-external knowledge representations, the fused representations are guided to select the correct external knowledge.

[0113] Specifically, the speech recognition module is abbreviated as ASR. ASR takes speech as input and the recognized content as output. Since ASR generates an estimate of the letter corresponding to the current pronunciation based on information such as phonemes in the speech signal, this estimate is not 100% accurate, but has a confidence score. ASR will arrange these estimates in order from high to low, thus producing multiple outputs.

[0114] Specifically, the writing dialogue history text and C external knowledge points used for model training are first concatenated, and the formula can be written as: {[W t [k1], ..., [W] t k C ]}. Among them, W t Represents the writing dialogue history text {k1, ..., k} C} represents C pieces of knowledge.

[0115] The joint sentence-level representation extracted using BERT from the concatenation of writing dialogue history text and external knowledge can be represented as follows: Where h w，i With [W] t k i There is a one-to-one correspondence, i = 1, ..., C.

[0116] Next, multiple outputs from the speech recognition module are extracted to obtain multiple segments of spoken dialogue history text, which can be represented by the following formula: Where N represents the number of outputs. Multiple spoken dialogue historical texts are concatenated with C external knowledge sources used for model training, resulting in C×N input samples, which can be written as:

[0117] By leveraging BERT to extract representations from the concatenation of spoken dialogue history text and external knowledge, the joint sentence-level representation of spoken dialogue history text and external knowledge extracted by BERT can be used for writing.

[0118] For a given external knowledge j, we obtain a joint sentence-level representation of N spoken dialogue historical texts and external knowledge. Then, an attention mechanism is used to fuse these N representations. The fusion process can be written as a formula:

[0119]

[0120] in It is a completely one vector. These are the learnable parameters of the attention mechanism module. The joint sentence-level representation of C spoken dialogue history texts and external knowledge after fusion through the attention mechanism can be written as:

[0121] Characterize H w and Passing through linear layers respectively The classification probabilities of obtaining the joint representation of the written dialogue history text and external knowledge, and the joint representation of the spoken dialogue history text and external knowledge, are denoted as p, respectively. w and p s :

[0122]

[0123] The training of the network can be supervised using the following loss function to select the unique correct knowledge from C external candidate knowledge.

[0124]

[0125] Finally, the writing dialogue history text - external knowledge representation h w，g Constructing a positive sample pair: if and only if the indices i and j point to the same correct external knowledge. This constitutes a pair of positive samples; for writing dialogue history text - external knowledge representation h w，g Constructing negative sample pairs: The set of negative samples belonging to the same writing text domain is as follows: The set of negative samples belonging to the spoken text domain is as follows: The final set of all negative samples can be written as:

[0126] Representation of spoken dialogue history texts as external knowledge Constructing a positive sample pair: if and only if the indices i and j point to the same correct external knowledge. This constitutes a pair of positive samples. It represents the spoken dialogue history text as an external knowledge representation. Constructing negative sample pairs: The set of negative samples belonging to the spoken text domain is as follows: The set of negative samples belonging to the writing text domain is: The final set of all negative samples can be written as:

[0127] Finally, contrastive learning is used to guide the selection of correct external knowledge from the written dialogue history text-external knowledge representation. The loss function for contrastive learning, designed using constructed positive and negative sample pairs, is shown in the following formula:

[0128]

[0129] The total loss can be used for the entire network: The network is trained to guide the selection of correct external knowledge from the fused spoken dialogue history text-external knowledge representation.

[0130] Those skilled in the art will understand that, in addition to implementing the system, apparatus, and their modules provided by this invention in purely computer-readable program code, the same program can be implemented in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers by logically programming the method steps. Therefore, the system, apparatus, and their modules provided by this invention can be considered a hardware component, and the modules included therein for implementing various programs can also be considered structures within the hardware component; alternatively, modules for implementing various functions can be considered both software programs implementing the method and structures within the hardware component.

[0131] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A robust spoken language knowledge extraction method based on contrastive learning and attention mechanism, characterized in that, Comprising: Step S1: extracting joint sentence-level representations of the written dialogue history text and external knowledge using BERT; Step S2: extracting joint sentence-level representations of the spoken dialogue history text corresponding to the written dialogue history text and output by the speech recognition module and external knowledge using BERT; Step S3: fusing multiple joint sentence-level representations of the spoken dialogue history text and external knowledge into one representation using an attention mechanism; Step S4: selecting the correct external knowledge based on the fused joint representation of the spoken dialogue history text and external knowledge and the joint sentence-level representations of the written dialogue history text and external knowledge using contrastive learning; The step S1 adopts: Step S1.1: writing a dialogue history text and an external knowledge cascade for knowledge selection model training; denoted as: ; wherein, denotes a writing dialogue history text; represents an external knowledge; Step S1.2: Extract joint sentence-level representation with BERT for the concatenation of the writing dialogue history text and external knowledge; denoted as: ; wherein, corresponds to one-to-one, each representation vector ; The step S2 adopts: Step S2.1: Extracting multiple outputs of the speech recognition module to obtain multiple segments of spoken dialogue history text; denoted as: , wherein denotes the number of outputs. Step S2.2: concatenate the extracted multi-turn spoken dialogue history text with the C external knowledge obtained from the knowledge selection model training to obtain the concatenated spoken dialogue history text-external knowledge; denoted as: ; Step S2.3: representation extraction of the concatenation of the spoken dialogue history text-external knowledge with BERT; denoted as: ; wherein, corresponds to one-to-one, ; The step S3 adopts: For a certain external knowledge Organized A joint sentence-level representation of spoken dialogue history text and external knowledge Use attention mechanisms for the current The representations are fused; represented as: wherein, is an all-one vector; is a learnable parameter of the attention mechanism module; and is represented as: ; The step S4 adopts: Step S4.1: constructing positive sample pairs and negative sample pairs corresponding to the written dialogue history text-external knowledge representation; Step S4.2: constructing positive sample pairs and negative sample pairs corresponding to the fused joint representation of the spoken dialogue history text and external knowledge; Step S4.3: using contrastive learning to enable the written dialogue history text-external knowledge representation to guide the spoken dialogue history text-external knowledge representation to select the correct external knowledge; S4.1 employs: if and only if the subscript and point to the same correct external knowledge constitute a pair of positive samples; The negative sample set belonging to the writing text domain is: The negative sample set belonging to the oral text domain is: The final all-negative sample set can be written as: ; The step S4.2 adopts: iff the subscripts and point to the same correct external knowledge form a pair of positive samples; The negative sample set belonging to the oral text domain is: The negative sample set belonging to the writing text domain is: The final all-negative sample set can be written as: ; The step S4.3 adopts: using contrastive learning to enable the written dialogue history text-external knowledge representation to guide the spoken dialogue history text-external knowledge representation to select the correct external knowledge; The whole knowledge selection model uses total loss: Training is conducted to guide the selection of correct external knowledge by the fused spoken dialogue history text-external knowledge representation. Wherein, the loss function of the contrastive learning designed by the constructed positive and negative sample pairs is shown in the following formula: characterize and pass through linear layers obtained writing dialogue history text-external knowledge joint representation and spoken dialogue history text-external knowledge joint representation classification probability respectively recorded as and : ； The training of the knowledge selection model is supervised by a loss function, from one of the external alternative knowledge to the only correct knowledge; wherein, The probability of selecting the correct knowledge using the written text dialogue history-external knowledge joint representation; The probability of selecting the correct knowledge using the fused spoken text dialogue history-external knowledge joint representation.

2. A robust spoken language knowledge extraction system based on contrastive learning and attention mechanism, characterized in that, Comprising: Module M1: extracting joint sentence-level representations of the written dialogue history text and external knowledge using BERT; Module M2: extracting joint sentence-level representations of the spoken dialogue history text corresponding to the written dialogue history text and output by the speech recognition module and external knowledge using BERT; Module M3: fusing multiple joint sentence-level representations of the spoken dialogue history text and external knowledge into one representation using an attention mechanism; Module M4: selecting the correct external knowledge based on the fused joint representation of the spoken dialogue history text and external knowledge and the joint sentence-level representations of the written dialogue history text and external knowledge using contrastive learning; The module M1 adopts: Module M1.1: writing a dialogue history text and an external knowledge cascade for knowledge selection model training; denoted as: ; wherein, denotes a writing dialogue history text; represents an external knowledge; Module M1.2: jointly extract sentence-level representations using BERT on the concatenation of the writing dialogue history text and external knowledge; denoted as: ; wherein, corresponds to one-to-one, each representation vector ; The module M2 adopts: Module M2.1: extracting a plurality of outputs of the speech recognition module to obtain a plurality of pieces of spoken dialogue history text; denoted as: , wherein denotes the number of outputs. Module M2.2: concatenate the extracted multi-turn spoken dialogue history text with C external knowledge respectively to obtain the concatenation of the multi-turn spoken dialogue history text and the external knowledge, denoted as: ; Module M2.3: representation extraction from the concatenation of the spoken dialogue history text-external knowledge with BERT; denoted as: ; wherein, corresponds to one-to-one, ; The module M3 adopts: for a certain external knowledge obtained by sorting joint sentence-level representation of the oral dialogue history text and the external knowledge ; the current representation is fused using an attention mechanism; represented as: wherein, is an all-one vector; is a learnable parameter of the attention mechanism module; and the fused text representation of the oral dialogue history and the external knowledge at the sentence level is ;​ The module M4 adopts: Module M4.1: constructing positive sample pairs and negative sample pairs corresponding to the written dialogue history text-external knowledge representation; Module M4.2: constructing positive sample pairs and negative sample pairs corresponding to the fused joint representation of the spoken dialogue history text and external knowledge; Module M4.3: using contrastive learning to enable the written dialogue history text-external knowledge representation to guide the spoken dialogue history text-external knowledge representation to select the correct external knowledge; The module M4.1 employs: if and only if the subscript and point to the same correct external knowledge constitute a pair of positive samples; The negative sample set belonging to the writing text domain is: The negative sample set belonging to the oral text domain is: The final all-negative sample set can be written as: ; The module M4.2 adopts: iff the subscripts and point to the same correct external knowledge form a pair of positive samples; The negative sample set belonging to the oral text domain is: The negative sample set belonging to the writing text domain is: The final all-negative sample set can be written as: ; The module M4.3 adopts: using contrastive learning to enable the written dialogue history text-external knowledge representation to guide the spoken dialogue history text-external knowledge representation to select the correct external knowledge; The whole knowledge selection model uses total loss: Training is conducted to guide the selection of correct external knowledge by the fused spoken dialogue history text-external knowledge representation. Wherein, the loss function of the contrastive learning designed by the constructed positive and negative sample pairs is shown in the following formula: characterize and pass through linear layers obtained writing dialogue history text-external knowledge joint representation and spoken dialogue history text-external knowledge joint representation classification probability respectively recorded as and : ； The training of the knowledge selection model is supervised by a loss function, from one of the external alternative knowledge to the only correct knowledge; wherein, The probability of selecting the correct knowledge using the written text dialogue history-external knowledge joint representation; The probability of selecting the correct knowledge using the fused spoken text dialogue history-external knowledge joint representation.

Images