Question and answer interaction method, device and equipment in financial field

Abstract

The invention discloses a financial field knowledge question answering method, device and equipment. The method comprises the steps of firstly obtaining a to-be-answered target question text in the financial field; and generating a target question feature vector corresponding to the target question text. Then performing matching calculation on the target problem feature vector and feature vectors of candidate nodes stored in a pre-constructed graph neural network structure, and judging whether feature vectors of successfully matched candidate nodes exist or not according to a calculation result; then, the feature vectors of the successfully matched candidate nodes are input into a full connection layer for reasoning, reply content of the target question text is generated, and the feature vectors of the candidate nodes comprise emotion vectors obtained after emotion analysis is conducted on the sample question text; therefore, the high-quality feature information can be extracted from the financial data, and deep understanding is performed on the target question text in combination with sentiment analysis, so that the answering efficiency and accuracy of the target question are improved, and the question and answer experience of the user is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and particularly relates to a financial field question and answer interaction method, device and equipment. BACKGROUND

[0002] With the rapid development of social economy, banks as the center of the financial field have brought great convenience to people's life. Moreover, with the development of artificial intelligence technology, banks have successively launched intelligent voice interaction systems, such as intelligent customer service, voice navigation, etc., so as to facilitate some interactions and answers about the knowledge in the financial field with users.

[0003] However, the existing bank intelligent voice interaction system is mainly based on the mandarin speech recognition technology, and the support for dialect users is very limited. According to statistics, more than 30% of the population uses dialect in daily communication, especially in the elderly group and rural areas, and the proportion is higher. For users who use dialect to interact and answer, the deficiency of the existing bank intelligent voice interaction system in dialect recognition will lead to low reply efficiency and accuracy rate for these users, and thus reduces the question and answer interaction experience of these users. SUMMARY

[0004] The main purpose of the embodiment of the present application is to provide a financial field question and answer interaction method, device and equipment, which can improve the reply efficiency and accuracy rate of the financial field question proposed by the user who uses dialect to interact and answer, and thus improve the question and answer interaction experience of these users.

[0005] The embodiment of the present application provides a financial field question and answer interaction method, comprising:

[0006] Obtaining target dialect voice to be replied in the financial field;

[0007] Extracting acoustic features, first-order differences and second-order differences of the target dialect voice;

[0008] Inputting the target dialect voice and the acoustic features, the first-order differences and the second-order differences into a pre-constructed dialect recognition model to predict a standard phonetic symbol corresponding to the target dialect voice; and converting the target dialect voice into target standard voice in the financial field by using the standard phonetic symbol;

[0009] Generating mandarin reply content of the target standard voice; and converting the mandarin reply content into dialect reply voice corresponding to the target dialect voice.

[0010] In an optional implementation, the dialect recognition model comprises a time domain feature extraction network, a frequency domain feature extraction network, a feature fusion layer, a dialect classification head, and a phoneme recognition head; the target dialect speech and its acoustic features and first-order and second-order differences are input into the pre-constructed dialect recognition model, and a standard phonetic symbol corresponding to the target dialect speech is predicted, comprising:

[0011] The target dialect speech is input into the time domain feature extraction network to obtain time domain features of pronunciation boundaries;

[0012] The acoustic features, first-order differences, and second-order differences of the target dialect speech are input into the time domain feature extraction network to obtain frequency domain features of tonal information;

[0013] The time domain features of pronunciation boundaries and the frequency domain features of tonal information are input into the feature fusion layer to obtain fused features;

[0014] The fused features are input into the dialect classification head to identify a dialect category to which the target dialect speech belongs;

[0015] The fused features are input into the phoneme recognition head to generate a standard phonetic symbol corresponding to the target dialect speech in combination with the dialect category to which the target dialect speech belongs.

[0016] In an optional implementation, the time domain feature extraction network is a network architecture stacked with an interpretable convolutional neural network, a depth separable convolution, and a Transformer model; the frequency domain feature extraction network is a network architecture stacked with a convolutional neural network (CNN) and a bidirectional long short-term memory network (BiLSTM); and the feature fusion layer is a network architecture of gated attention fusion.

[0017] In an optional implementation, the dialect recognition model is constructed in the following manner:

[0018] A sample dialect speech is obtained;

[0019] The sample dialect speech and a gradient inversion method are used to train an initial dialect recognition model to obtain the dialect recognition model;

[0020] The initial dialect recognition model comprises an adversarial learning head, which is used for adversarial training to eliminate a voiceprint feature irrelevant to the sample dialect and implement gradient inversion training of the model.

[0021] In an optional implementation, the method further comprises:

[0022] A verification dialect speech is obtained;

[0023] extracting acoustic features and first-order and second-order differences of the verification dialect speech;

[0024] inputting the verification dialect speech and the acoustic features and the first-order and second-order differences thereof into the dialect recognition model to obtain a standard phonetic symbol prediction result corresponding to the verification dialect speech;

[0025] when the standard phonetic symbol prediction result of the verification dialect speech is inconsistent with a phoneme standard result corresponding to the verification dialect speech, reusing the verification dialect speech as the sample dialect speech to update the dialect recognition model.

[0026] Corresponding to the above-mentioned financial field question and answer interaction method, the present application proposes a financial field question and answer interaction device, comprising:

[0027] a first acquisition unit configured to acquire a target dialect speech to be replied in the financial field;

[0028] a first extraction unit configured to extract acoustic features and first-order and second-order differences of the target dialect speech;

[0029] an input unit configured to input the target dialect speech and the acoustic features and the first-order and second-order differences thereof into a pre-constructed dialect recognition model to predict a standard phonetic symbol corresponding to the target dialect speech, and convert the target dialect speech into a target standard speech in the financial field by using the standard phonetic symbol;

[0030] a generation unit configured to generate a Mandarin reply content of the target standard speech, and convert the Mandarin reply content into a dialect reply speech corresponding to the target dialect speech.

[0031] In an alternative implementation manner, the dialect recognition model comprises a time domain feature extraction network, a frequency domain feature extraction network, a feature fusion layer, a dialect classification head, and a phoneme recognition head; and the input unit comprises:

[0032] a first input subunit configured to input the target dialect speech into the time domain feature extraction network to obtain time domain features of pronunciation boundaries;

[0033] a second input subunit configured to input acoustic features and first-order and second-order differences of the target dialect speech into the time domain feature extraction network to obtain frequency domain features of tonal information;

[0034] a third input subunit configured to input the time domain features of pronunciation boundaries and the frequency domain features of tonal information into the feature fusion layer to obtain fused features;

[0035] The fourth input subunit is configured to input the fused features into the dialect classification head to identify a dialect category to which the target dialect speech belongs.

[0036] The fifth input subunit is configured to input the fused features into the phoneme recognition head to generate a standard phonetic symbol corresponding to the target dialect speech in combination with the dialect category to which the target dialect speech belongs.

[0037] In an optional implementation, the time-domain feature extraction network is a network architecture in which an interpretable convolutional neural network and a depthwise separable convolution and a Transformer model are stacked; the frequency-domain feature extraction network is a network architecture in which a convolutional neural network CNN and a bidirectional long short-term memory network BiLSTM are stacked; and the feature fusion layer is a network architecture in which a gated attention fusion is used.

[0038] In an optional implementation, the apparatus further includes:

[0039] The second acquisition unit is configured to acquire sample dialect speech.

[0040] The training unit is configured to train an initial dialect recognition model by using the sample dialect speech and a gradient inversion manner to obtain the dialect recognition model.

[0041] The initial dialect recognition model includes an adversarial learning head, which is configured to eliminate a voiceprint feature irrelevant to the sample dialect through adversarial training to implement gradient inversion training of the model.

[0042] In an optional implementation, the apparatus further includes:

[0043] The third acquisition unit is configured to acquire verification dialect speech.

[0044] The second extraction unit is configured to extract acoustic features, first-order differences and second-order differences of the verification dialect speech.

[0045] The obtaining unit is configured to input the verification dialect speech, the acoustic features, the first-order differences and the second-order differences into the dialect recognition model to obtain a standard phonetic symbol prediction result corresponding to the verification dialect speech.

[0046] The updating unit is configured to, when the standard phonetic symbol prediction result of the verification dialect speech is inconsistent with a phoneme standard result corresponding to the verification dialect speech, re-use the verification dialect speech as the sample dialect speech to update the dialect recognition model.

[0047] Embodiments of the present application also provide a financial field question and answer interaction device, which includes a processor, a memory and a system bus.

[0048] The processor and the memory are connected through the system bus;

[0049] The memory is configured to store one or more programs including instructions, which, when executed by the processor, cause the processor to perform any one of the above-mentioned implementation manners of the financial field question and answer interaction method.

[0050] The present application also provides a computer readable storage medium, which stores instructions, and when the instructions run on a terminal device, the terminal device performs any one of the above-mentioned implementation manners of the financial field question and answer interaction method.

[0051] Therefore, the present application has the following beneficial effects:

[0052] The present application provides a financial field question and answer interaction method, device and equipment. First, a target dialect voice to be replied in the financial field is acquired. Then, acoustic features and first-order and second-order differences of the target dialect voice are extracted. Next, the target dialect voice and the acoustic features and first-order and second-order differences are input into a pre-constructed dialect recognition model to predict a standard phonetic symbol corresponding to the target dialect voice. The standard phonetic symbol is used to convert the target dialect voice into a target standard voice in the financial field. Then, a Mandarin reply content of the target standard voice can be generated. The Mandarin reply content is converted into a dialect reply voice corresponding to the target dialect voice. In this way, the reply efficiency and accuracy of a question in the financial field proposed by a user using a dialect for question and answer interaction can be improved, and the question and answer interaction experience of the user is improved. BRIEF DESCRIPTION OF DRAWINGS

[0053] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0054] Figure 1 A flow chart of a financial field question and answer interaction method provided by the present application;

[0055] Figure 2 A structural schematic diagram of a dialect recognition model provided by the present application;

[0056] Figure 3 A process schematic diagram of dynamic self-adaptive learning provided by the present application;

[0057] Figure 4A composition schematic diagram of a financial field question and answer interaction device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0058] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in a clear and complete manner with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0059] At present, dialect speech recognition faces multiple technical challenges. For example, data scarcity: there are few publicly available dialect speech datasets, and the annotation cost is high; phonological complexity: the phonological system of each local dialect is quite different from that of Mandarin, and some dialects have phonemes that do not exist in Mandarin; regional differences: there are significant variations of the same dialect in different regions; context dependence: a large number of words and expressions in dialects are highly dependent on context; mixed use: users often mix dialects and Mandarin in the same sentence.

[0060] In the service of banks and other financial institutions, many important businesses (such as account opening, money transfer, and loan) often require accurate voice interaction. When users use dialects, the existing bank intelligent voice interaction system often has a high error rate in recognition, resulting in poor service experience and low efficiency, and even may cause business errors. Especially in emergency situations, such as when an elderly person needs to quickly handle a business, insufficient dialect recognition capability can cause more serious consequences.

[0061] Based on this, the present application proposes a financial field question and answer interaction method, device and equipment, aiming to solve the problem of insufficient dialect recognition capability of the existing bank intelligent voice interaction system, and better serve the dialect user group that is not proficient in Mandarin by fusing a pre-constructed dialect recognition model and context understanding technology, etc., to improve the inclusiveness and user experience of bank services.

[0062] The financial field question and answer interaction method provided by the embodiments of the present application will be described in detail below with reference to the drawings. Referring to FIG. 1, a flowchart of an embodiment of a financial field question and answer interaction method provided by the present application is shown, which can include the following steps: Figure 1

[0063] S101: Obtain a target dialect speech to be replied in the financial field.

[0064] ​In the embodiment, the user inputted dialect speech of any need to be replied to the intelligent voice interaction system of the financial field institution (such as a bank) is defined as the target dialect speech to be replied to, and the embodiment does not limit the dialect type of the target dialect speech, for example, the target dialect speech can be Cantonese, Minnan, Wu, Hakka and the like. The embodiment also does not limit the length of the target dialect speech, for example, the target dialect speech can be a sentence or a paragraph and the like.

[0065] S102: Extracting the acoustic features and the first-order difference and the second-order difference of the target dialect speech.

[0066] In the embodiment, after the target dialect speech to be replied to in the financial field is acquired by step S101, the acoustic features and the first-order difference and the second-order difference of the target dialect speech can be further extracted by using a feature extraction method, so as to perform subsequent step 103. Specifically, when the acoustic features of the target dialect speech are extracted, the target dialect speech needs to be first frame processed to obtain a corresponding speech frame sequence, and then the pre-emphasis is performed on the frame processed speech frame sequence; and then the acoustic features of each speech frame are extracted in sequence, wherein the acoustic features refer to feature data used to represent the acoustic information of the corresponding speech frame, for example, can be Mel-scale Frequency Cepstral Coefficients (MFCC) features or Perceptual Linear Predictive (PLP) features and the like.

[0067] It should be noted that the embodiment of the present application does not limit the extraction method of the acoustic features of the target dialect speech, and does not limit the specific extraction process, and the appropriate extraction method can be selected according to the actual situation, and the corresponding feature extraction operation is performed, and the acoustic features MFCC will be taken as an example for introduction in the subsequent.

[0068] The first-order difference can reflect the instantaneous change speed of the speech signal, effectively capture the transition features of the consonant / vowel, and the second-order difference reflects the acceleration of the speech signal, can enhance the capture of the suprasegmental features such as intonation, stress and the like, and improve the boundary detection ability of the continuous speech.

[0069] S103: The target dialect speech and the acoustic features and the first-order difference and the second-order difference are input into the pre-constructed dialect recognition model, the standard phonetic symbols corresponding to the target dialect speech are predicted, and the target dialect speech is converted into the target standard speech in the financial field by using the standard phonetic symbols.

[0070] In this embodiment, after the acoustic features (such as MFCC) and the first-order difference and the second-order difference of the target dialect speech are extracted by step S102, the target dialect speech and the acoustic features (such as MFCC) and the first-order difference (Δ) and the second-order difference (ΔΔ) can be further input into the pre-constructed dialect recognition model, so as to predict the standard phonetic symbols corresponding to the target dialect speech; and the standard phonetic symbols are used to convert the target dialect speech into the target standard speech in the financial field, for performing the subsequent step S104.

[0071] Specifically, as shown in Figure 2 The dialect recognition model includes a time domain feature extraction network, a frequency domain feature extraction network, a feature fusion layer, a dialect classification head, and a phoneme recognition head. On this basis, the process of inputting the target dialect speech and the acoustic features and the first-order difference and the second-order difference into the pre-constructed dialect recognition model to predict the standard phonetic symbols corresponding to the target dialect speech in step S103 can specifically include:

[0072] (1) First, the target dialect speech (which can also be the target dialect speech after preprocessing) can be input into the time domain feature extraction network to obtain the time domain features of the pronunciation boundary. Before that, the original waveform samples of the target dialect speech can be directly used as input features through the Raw Waveform channel, and the optimal feature representation can be automatically learned through a deep learning model, and then input into the time domain feature extraction network to obtain the time domain features of the pronunciation boundary.

[0073] The time domain feature extraction network is a network architecture stacked with an interpretable convolutional neural network (SincNet) and a depth separable convolution and a Transformer model. SincNet is a parameterized bandpass filter simulating the hearing characteristics of human ears, the depth separable convolution network is used to reduce the parameter amount while maintaining the receptive field, and the local Transformer is used to process long-time dependence. The time domain feature extraction network is used to capture dialect micro features, extract transient features in the waveform, and identify the rhythm patterns specific to the dialect in continuous speech.

[0074] (2) Secondly, the acoustic features (such as MFCC) and the first-order difference (Δ) and the second-order difference (ΔΔ) of the target dialect speech can be input into the time domain feature extraction network to obtain the frequency domain features of the tone information.

[0075] The frequency domain feature extraction network is a network architecture stacked with a convolutional neural network (CNN) and a bidirectional long short-term memory network (BiLSTM), which is used to analyze the spectral characteristics of the dialect, analyze the fundamental frequency trajectory of the tone language, and detect the formant distribution.

[0076] (3) Then, the time domain features of the pronunciation boundary and the frequency domain features of the tone information can be input into a feature fusion layer to obtain fused features.

[0077] The feature fusion layer is a network architecture of gated attention fusion, which is used for dynamic interaction learning between features and controls the information flow strength. This layer is used to combine the time domain features of the pronunciation boundary and the frequency domain features of the tone information for comprehensive judgment, and to increase the weight of important features such as consonants specific to dialects.

[0078] (4) Further, the fused features can be input into a dialect classification head to identify the dialect category to which the target dialect speech belongs.

[0079] The dialect classification head is an auxiliary task that automatically identifies the dialect category of the input language and provides dialect context information for subsequent phoneme recognition. Since most users live in relatively stable regions, the information of the network site can be increased to have more weight in the dialect classification head, so as to reduce resource consumption in the training process.

[0080] (5) Finally, the fused features can be input into a phoneme recognition head to generate the standard phonetic symbols corresponding to the target dialect speech in combination with the dialect category to which the target dialect speech belongs.

[0081] The phoneme recognition head is used to generate a standard phoneme sequence, convert the dialect speech into standard phonetic symbols, and support end-to-end business term conversion (dialect to bank standard term).

[0082] S104: Generate the Mandarin reply content of the target standard speech; and convert the Mandarin reply content into the dialect reply speech corresponding to the target dialect speech.

[0083] In this embodiment, after the target dialect speech is converted into the target standard speech in the financial field through step S103, the bank business intent can be further extracted through semantic analysis, and the dialect vocabulary can be mapped to the standard bank business term through term mapping. Finally, the Mandarin text that meets the business specification is output, and the corresponding Mandarin reply content is generated as the Mandarin reply content for the target standard speech, and the Mandarin reply content is converted into natural dialect expression, and finally the dialect speech is generated by the dialect speech synthesis (Text To Speech, TTS) model to output to the user using the target dialect speech, thereby realizing the conversion of the Mandarin reply content into the dialect reply speech corresponding to the target dialect speech.

[0084] Next, this embodiment will introduce the construction process of the dialect recognition model. One optional implementation method is that the construction process of the dialect recognition model may specifically include: firstly, obtaining sample dialect speech, and then training the initial dialect recognition model with the sample dialect speech and gradient inversion method to obtain the dialect recognition model.

[0085] Among them, such as Figure 2 As shown, the initial dialect recognition model includes an adversarial learning head; and this adversarial learning head is used for adversarial training to eliminate voiceprint features that are irrelevant to the dialect of the sample, thereby achieving gradient inversion training of the model and improving the model's focus on dialect features.

[0086] Furthermore, since the terminology used by banks and other financial institutions is relatively fixed, the voice information that users need to convey when conducting business is controllable. Therefore, dialect voices can be collected specifically when constructing training data. Firstly, the corpus collection can cover geographical and demographic ranges. For example, according to the division of dialect regions such as Cantonese and Min, voice samples from at least five representative cities should be collected for each dialect region (e.g., the Cantonese region includes Guangzhou, Shenzhen, Hong Kong, and Macau). Within these regions, voice samples should be collected according to different age gradients for teenagers, young adults, middle-aged people, and the elderly, or according to different occupations for financial workers, farmers, etc., as sample dialect voices. Secondly, sample processing can be performed. Samples are collected using certain recording standards, and then labeled sequentially according to the following steps: "sample dialect voice → phoneme-level annotation (→ dialect-specific pronunciation annotation → banking business semantic annotation → sentiment annotation)". During the annotation process, special handling should be paid to points where Mandarin and dialect switch, and colloquial expressions. Finally, data augmentation processing can be performed. To improve the model's generalization ability and robustness, a portion of the data was processed using a ±20% speed adjustment, and another portion was subjected to added environmental noise (such as ATM operation sounds and background noise in the business hall).

[0087] Specifically, during model training, a sample dialect speech can be extracted from the training data as the model input and the corresponding phoneme-level annotation result as the output. Multiple rounds of model training are performed, and the phoneme-level annotation result obtained in each round of training is compared with the corresponding manual annotation result. The model parameters are updated based on the difference between the two until the preset conditions are met, such as the loss constraint function being very small and basically unchanged. Then, the update of the model parameters is stopped, the training of the dialect recognition model is completed, and a trained dialect recognition model is generated.

[0088] Based on this, after training and generating a dialect recognition model from sample dialect speech, the generated dialect recognition model can be further validated using verification dialect speech. The specific validation process may include the following steps (1)-(3):

[0089] Step (1): Obtain verification dialect voice.

[0090] In this embodiment, in order to verify the dialect recognition model, first, the verification dialect voice needs to be obtained. For example, 1000 pieces of dialect voice issued by different users to the bank intelligent interaction system can be collected as verification dialect voice under the condition that the user allows. The verification dialect voice refers to voice information that can be used to verify the dialect recognition model. After obtaining the verification dialect voice and the phoneme-level label corresponding to each verification dialect voice, the subsequent step (2) can be continued.

[0091] Step (2): Extract the acoustic features, first-order difference and second-order difference of the verification dialect voice.

[0092] Step (3): Input the verification dialect voice and the acoustic features, first-order difference and second-order difference thereof into the dialect recognition model to obtain the standard phonetic symbol prediction result corresponding to the verification dialect voice.

[0093] Step (4): When the standard phonetic symbol prediction result of the verification dialect voice is inconsistent with the phoneme standard result corresponding to the verification dialect voice, the verification dialect voice is re-used as a sample dialect voice, and the dialect recognition model is updated.

[0094] Through the above embodiment, the dialect recognition model can be effectively verified by using the verification dialect voice, which can further improve the prediction accuracy and accuracy of the audio quality inspection model.

[0095] In addition, due to the large amount of data collection and labeling work, it is difficult to completely cover all dialects in all contexts. In order to further improve the recognition accuracy of the model, the application proposes to continuously optimize the dialect recognition model in the interaction process through dynamic adaptive learning. In this way, the data amount required for collection / labeling can be reduced. The process diagram of dynamic adaptive learning is shown in Figure 3 In this way, by interacting with the user, new dialect samples can be obtained, and the confidence and threshold of the sample are compared. If the confidence is greater than the threshold, the sample is subjected to online incremental learning and a model patch is generated. After security verification, it can be hot deployed to the corresponding regional node.

[0096] In addition, in addition to voice recognition, touch screen input, camera vision analysis and other functions can also be added. When the confidence of voice recognition is low in some cases, manual input, visual analysis and other methods can be used for compensation. The inputs of the three are fused and analyzed by a fusion engine to obtain the real interaction intention of the user, and feedback is given to the user through the speaker, screen, light and the like to improve the user's question and answer interaction experience.

[0097] In summary, the financial field question and answer interaction method provided in this embodiment first acquires target dialect speech to be replied in the financial field; then extracts acoustic features and first-order differences and second-order differences of the target dialect speech; then inputs the target dialect speech and the acoustic features and the first-order differences and the second-order differences into a dialect recognition model constructed in advance, to predict a standard phonetic symbol corresponding to the target dialect speech; and converts the target dialect speech into target standard speech in the financial field by using the standard phonetic symbol; further, Mandarin reply content of the target standard speech can be generated; and the Mandarin reply content is converted into dialect reply speech corresponding to the target dialect speech. In this way, the reply efficiency and accuracy of a question and answer interaction user who uses dialect to propose a question in the financial field can be improved, and the question and answer interaction experience of the user is improved.

[0098] Referring to Figure 4 As shown in the figure, the present application also provides a financial field question and answer interaction device embodiment, which can include:

[0099] The first acquisition unit 401 is configured to acquire target dialect speech to be replied in the financial field;

[0100] The first extraction unit 402 is configured to extract acoustic features and first-order differences and second-order differences of the target dialect speech;

[0101] The input unit 403 is configured to input the target dialect speech and the acoustic features and the first-order differences and the second-order differences into a dialect recognition model constructed in advance, to predict a standard phonetic symbol corresponding to the target dialect speech; and convert the target dialect speech into target standard speech in the financial field by using the standard phonetic symbol;

[0102] The generation unit 404 is configured to generate Mandarin reply content of the target standard speech; and convert the Mandarin reply content into dialect reply speech corresponding to the target dialect speech.

[0103] In some possible implementation manners of the present application, the dialect recognition model includes a time domain feature extraction network, a frequency domain feature extraction network, a feature fusion layer, a dialect classification head, and a phoneme recognition head; and the input unit 403 includes:

[0104] The first input subunit is configured to input the target dialect speech into the time domain feature extraction network, to obtain time domain features of pronunciation boundaries;

[0105] The second input subunit is configured to input acoustic features and first-order differences and second-order differences of the target dialect speech into the time domain feature extraction network, to obtain frequency domain features of tonal information;

[0106] The third input subunit is configured to input the time domain features of the pronunciation boundary and the frequency domain features of the tonal information into the feature fusion layer to obtain fused features.

[0107] The fourth input subunit is configured to input the fused features into the dialect classification head to identify a dialect category to which the target dialect speech belongs.

[0108] The fifth input subunit is configured to input the fused features into the phoneme recognition head to generate a standard phonetic symbol corresponding to the target dialect speech in combination with the dialect category to which the target dialect speech belongs.

[0109] In some possible implementation manners of the present application, the apparatus further includes:

[0110] The second acquisition unit is configured to acquire the sample question and answer data; the sample question and answer data includes a sample question text and a reply content of the sample question text.

[0111] The training unit is configured to train an initial graph neural network structure and an initial sentiment analysis model by using the sample question and answer data and a cross-entropy loss function to obtain the trained graph neural network structure and sentiment analysis model.

[0112] In some possible implementation manners of the present application, the time domain feature extraction network is a network architecture stacked by an interpretable convolutional neural network, a depth separable convolution and a Transformer model; the frequency domain feature extraction network is a network architecture stacked by a convolutional neural network CNN and a bidirectional long short-term memory network BiLSTM; and the feature fusion layer is a network architecture of gated attention fusion.

[0113] In some possible implementation manners of the present application, the apparatus further includes:

[0114] The second acquisition unit is configured to acquire sample dialect speech.

[0115] The training unit is configured to train an initial dialect recognition model by using the sample dialect speech and a gradient inversion method to obtain the dialect recognition model.

[0116] The initial dialect recognition model includes an adversarial learning head; and the adversarial learning head is configured to perform adversarial training to eliminate a voiceprint feature irrelevant to the sample dialect and implement gradient inversion training of the model.

[0117] In some possible implementation manners of the present application, the apparatus further includes:

[0118] The third acquisition unit is configured to acquire verification dialect speech.

[0119] a second extraction unit configured to extract acoustic features and first-order and second-order differences of the verification dialect speech;

[0120] an obtaining unit configured to input the verification dialect speech and the acoustic features and the first-order and second-order differences into the dialect recognition model to obtain a standard phonetic symbol prediction result corresponding to the verification dialect speech;

[0121] an updating unit configured to, when the standard phonetic symbol prediction result of the verification dialect speech is inconsistent with a phoneme standard result corresponding to the verification dialect speech, update the dialect recognition model by taking the verification dialect speech as the sample dialect speech again.

[0122] As can be seen from the above embodiments, the financial field question and answer interaction device provided by the embodiments of the present application first acquires target dialect speech to be replied in the financial field; then extracts acoustic features and first-order and second-order differences of the target dialect speech; then inputs the target dialect speech and the acoustic features and the first-order and second-order differences into a pre-constructed dialect recognition model to predict a standard phonetic symbol corresponding to the target dialect speech; and converts the target dialect speech into target standard speech in the financial field by using the standard phonetic symbol; further generates a Mandarin reply content of the target standard speech; and converts the Mandarin reply content into dialect reply speech corresponding to the target dialect speech. Thus, the reply efficiency and accuracy of a question and answer interaction user's question in the financial field can be improved, and the question and answer interaction experience of the user is improved.

[0123] Further, the embodiments of the present application further provide a financial field question and answer interaction device, which comprises a processor, a memory and a system bus;

[0124] The processor and the memory are connected through the system bus;

[0125] The memory is configured to store one or more programs, the one or more programs comprising instructions which, when executed by the processor, cause the processor to perform any of the implementation methods of the above financial field question and answer interaction method.

[0126] Further, the embodiments of the present application further provide a computer readable storage medium, which stores instructions, when the instructions run on a terminal device, cause the terminal device to perform any of the implementation methods of the above financial field question and answer interaction method.

[0127] Those skilled in the art can clearly understand that all or part of the steps of the above-mentioned method can be implemented by means of software and necessary universal hardware platforms through the description of the above embodiments. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product in essence or in the form of a software product that contributes to the prior art. The computer software product can be stored in a storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the method described in each embodiment or some parts of the embodiments of the present application.

[0128] It should be noted that the various embodiments in the specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0129] It should also be noted that in this document, relationship terms such as first and second are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0130] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method of financial domain question answering interaction, characterized by, The method comprises the following steps: acquiring a target dialect voice to be replied in the financial field; extracting acoustic features and first-order and second-order differences of the target dialect voice; inputting the target dialect voice and the acoustic features and first-order and second-order differences thereof into a pre-constructed dialect recognition model to predict a standard phonetic symbol corresponding to the target dialect voice; and converting the target dialect voice into a target standard voice in the financial field by using the standard phonetic symbol; generating a Mandarin reply content of the target standard voice; and converting the Mandarin reply content into a dialect reply voice corresponding to the target dialect voice.

2. The method of claim 1, wherein, The dialect recognition model comprises a time-domain feature extraction network, a frequency-domain feature extraction network, a feature fusion layer, a dialect classification head and a phoneme recognition head; the step of inputting the target dialect voice and the acoustic features and first-order and second-order differences thereof into the pre-constructed dialect recognition model to predict a standard phonetic symbol corresponding to the target dialect voice comprises the following steps: inputting the target dialect voice into the time-domain feature extraction network to obtain time-domain features of pronunciation boundaries; inputting acoustic features and first-order and second-order differences of the target dialect voice into the time-domain feature extraction network to obtain frequency-domain features of tonal information; inputting the time-domain features of pronunciation boundaries and the frequency-domain features of tonal information into the feature fusion layer to obtain fused features; inputting the fused features into the dialect classification head to identify a dialect category to which the target dialect voice belongs; inputting the fused features into the phoneme recognition head to generate a standard phonetic symbol corresponding to the target dialect voice in combination with the dialect category to which the target dialect voice belongs.

3. The method of claim 2, wherein, The time-domain feature extraction network is a network architecture stacked with an interpretable convolutional neural network, a depthwise separable convolution and a Transformer model; the frequency-domain feature extraction network is a network architecture stacked with a convolutional neural network (CNN) and a bidirectional long short-term memory network (BiLSTM); and the feature fusion layer is a network architecture of gated attention fusion.

4. The method of claim 1, wherein, The dialect recognition model is constructed in the following manner: acquiring sample dialect voices; training an initial dialect recognition model by using the sample dialect voices and a gradient inversion method to obtain the dialect recognition model; wherein the initial dialect recognition model comprises an adversarial learning head for adversarial training to eliminate voiceprint features irrelevant to the sample dialect and realize gradient inversion training of the model.

5. The method of claim 4, wherein, The method further comprises the following steps: acquiring a verification dialect voice; extracting acoustic features and first-order and second-order differences of the verification dialect voice; inputting the verification dialect voice and the acoustic features and first-order and second-order differences thereof into the dialect recognition model to obtain a standard phonetic symbol prediction result corresponding to the verification dialect voice; when the standard phonetic symbol prediction result of the verification dialect voice is inconsistent with a phoneme standard result corresponding to the verification dialect voice, reusing the verification dialect voice as the sample dialect voice to update the dialect recognition model.

6. A financial field question and answer interactive apparatus characterized by comprising: The method comprises the following steps: a first acquiring unit configured to acquire a target dialect voice to be replied in the financial field; The first extraction unit is configured to extract the acoustic features and the first-order difference and the second-order difference of the target dialect speech. The input unit is configured to input the target dialect speech and the acoustic features and the first-order difference and the second-order difference into a pre-constructed dialect recognition model to predict a standard phonetic symbol corresponding to the target dialect speech, and convert the target dialect speech into a target standard speech in the financial field by using the standard phonetic symbol. The generation unit is configured to generate a Mandarin reply content of the target standard speech, and convert the Mandarin reply content into a dialect reply speech corresponding to the target dialect speech.

7. The apparatus of claim 6, wherein, The dialect recognition model comprises a time-domain feature extraction network, a frequency-domain feature extraction network, a feature fusion layer, a dialect classification head and a phoneme recognition head. The input unit comprises: The first input subunit is configured to input the target dialect speech into the time-domain feature extraction network to obtain time-domain features of pronunciation boundaries. The second input subunit is configured to input the acoustic features and the first-order difference and the second-order difference of the target dialect speech into the time-domain feature extraction network to obtain frequency-domain features of tonal information. The third input subunit is configured to input the time-domain features of pronunciation boundaries and the frequency-domain features of tonal information into the feature fusion layer to obtain fused features. The fourth input subunit is configured to input the fused features into the dialect classification head to identify a dialect category to which the target dialect speech belongs.

8. The apparatus of claim 7, wherein, The fifth input subunit is configured to input the fused features into the phoneme recognition head to generate a standard phonetic symbol corresponding to the target dialect speech in combination with the dialect category to which the target dialect speech belongs.

9. A financial domain question answering interactive device, characterized by, The time-domain feature extraction network is a network architecture stacked with an interpretable convolutional neural network, a depth separable convolution and a Transformer model. The frequency-domain feature extraction network is a network architecture stacked with a convolutional neural network (CNN) and a bidirectional long short-term memory network (BiLSTM). The feature fusion layer is a network architecture of gated attention fusion. The system bus is configured to connect the processor and the memory.

10. A computer-readable storage medium, characterized in that, The memory is configured to store one or more programs comprising instructions, which, when executed by the processor, cause the processor to execute the method of any one of claims 1-5. The computer readable storage medium stores instructions, which, when executed on a terminal device, cause the terminal device to execute the method of any one of claims 1-5. The computer readable storage medium stores instructions, which, when executed on a terminal device, cause the terminal device to execute the method of any one of claims 1-5.

Images