Voice interaction method and device, electronic equipment and storage medium

Abstract

The invention provides a voice interaction method, a voice interaction device, electronic equipment and a computer storage medium, and relates to the technical field of intelligent interaction. The method comprises the following steps: receiving a voice signal input by a target user, and performing feature extraction on the voice signal to obtain a first feature sequence of the voice signal; searching an initial large-class entry matched with the first feature sequence from an entry database, and determining a target large-class entry based on a first similarity between the first feature sequence and the initial large-class entry; extracting initial sub-class entries from the initial large-class entries, and determining target sub-class entries based on a second similarity between the first feature sequence and the initial sub-class entries; and adding the target large-class vocabulary entries and the target sub-class vocabulary entries into a historical dialogue text of the target user to obtain a first input sequence, so as to obtain a target output text based on the first input sequence. The vocabulary entry recognition method and device can adapt to a wide vocabulary entry recognition range and are high in recognition accuracy.

Description

Background Art

[0002] With the advancement of voice interaction and natural language processing technology, intelligent interactive applications such as smart voice assistants and voice customer service can more accurately understand user needs and provide more intelligent services. However, in some specific technical fields, such as place names and station names with local characteristics, it is very easy to make mistakes in translating the entries from voice to text, which leads to poor performance in subsequent answers.

[0003] At present, the existing methods mainly access large language models (LLM) to obtain stronger context understanding capabilities. For entries such as restaurants and stations that are easily misidentified during speech-to-text conversion, a small hot word capture pre-training model is used to enhance the extracted hot word features. Among them, the hot word capture pre-training model is specially used to identify and enhance the entry features of specific fields (such as place names, proper nouns, etc.), thereby improving the translation accuracy of hot words. However, the above-mentioned method of enhancing hot word features depends on the quality and coverage of the hot word list, and has limited effect on non-hot word recognition, especially for the understanding ability of easy-to-error and important entries cannot be improved, and the accuracy of rare entries and new entries cannot be guaranteed.

[0004] Therefore, there is an urgent need for a voice interaction method that has a wide range of recognition terms and high accuracy.

[0005] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0006] The purpose of the present disclosure is to provide a voice interaction method, a voice interaction device, an electronic device and a computer storage medium, thereby at least to a certain extent solving the problems of limited recognition scope and low recognition accuracy caused by the limitations and defects of related technologies.

[0007] According to a first aspect of the present disclosure, a voice interaction method is provided, the method comprising: receiving a voice signal input by a target user, and performing feature extraction on the voice signal to obtain a first feature sequence of the voice signal; searching an initial major category term that matches the first feature sequence from a term database, and determining a target major category term based on a first similarity between the first feature sequence and the initial major category term; extracting an initial subcategory term from the initial major category term, and determining a target subcategory term based on a second similarity between the first feature sequence and the initial subcategory term; adding the target major category term and the target subcategory term to the historical conversation text of the target user to obtain a first input sequence, so as to obtain a target output text based on the first input sequence.

[0008] According to a second aspect of the present disclosure, a voice interaction device is provided, which includes: a feature extraction module, which is used to receive a voice signal input by a target user, and perform feature extraction on the voice signal to obtain a first feature sequence of the voice signal; a first determination module, which is used to search an initial major category term that matches the first feature sequence from a term database, and determine a target major category term based on a first similarity between the first feature sequence and the initial major category term; a second determination module, which is used to extract an initial subcategory term from the initial major category term, and determine a target subcategory term based on a second similarity between the first feature sequence and the initial subcategory term; and a term adding module, which is used to add the target major category term and the target subcategory term to the historical conversation text of the target user to obtain a first input sequence, so as to obtain a target output text based on the first input sequence.

[0009] According to a third aspect of the present disclosure, a computer storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, any one of the above-mentioned voice interaction methods is implemented.

[0010] According to a fourth aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned voice interaction methods by executing the executable instructions.

[0011] According to a fifth aspect of the present disclosure, an embodiment of the present disclosure provides a computer program product, including a computer program, which is executed by a processor to implement the above voice interaction method.

[0012] In the technical solutions provided by some embodiments of the present disclosure, a first feature sequence of the voice signal is obtained by receiving a voice signal input by a target user and extracting features of the voice signal; an initial major category term matching the first feature sequence is searched from a term database, and a target major category term is determined based on a first similarity between the first feature sequence and the initial major category term; an initial subcategory term is extracted from the initial major category term, and a target subcategory term is determined based on a second similarity between the first feature sequence and the initial subcategory term; the target major category term and the target subcategory term are added to the historical conversation text of the target user to obtain a first input sequence, so as to obtain a target output text based on the first input sequence. The method converts the problem of text generation based on voice data into a similarity matching problem by matching the feature sequence of the voice signal from the term database layer by layer with high similarity, thereby eliminating the need to generate candidate terms and reducing the difficulty of the task. On the other hand, the accuracy of term matching can be improved based on similarity, and is not limited by the quality and coverage of the hot word list, and is not only adapted to hot word recognition, but also to non-hot word recognition, further improving the adaptation range while meeting the high accuracy of voice recognition.

[0013] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present disclosure, and together with the specification, are used to explain the principles of the present disclosure. Obviously, the drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0015] Figure 1 An architectural diagram showing a voice interaction system according to an exemplary embodiment of the present disclosure;

[0016] Figure 2 A flowchart showing a voice interaction method according to an exemplary embodiment of the present disclosure is provided;

[0017] Figure 3 A flowchart showing another voice interaction method according to an exemplary embodiment of the present disclosure;

[0018] Figure 4 A flowchart showing a text output method according to an exemplary embodiment of the present disclosure;

[0019] Figure 5 A schematic diagram showing a complete voice interaction process according to an exemplary embodiment of the present disclosure is shown;

[0020] Figure 6 A schematic diagram showing a voice interaction device according to an exemplary embodiment of the present disclosure is shown;

[0021] Figure 7 A block diagram of an electronic device according to an exemplary embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0022] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present disclosure will be more comprehensive and complete, and the concepts of the example embodiments are fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0023] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0024] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the steps. For example, some steps may be decomposed, while some steps may be combined or partially combined, so the actual execution order may change according to the actual situation.

[0025] In order to help those skilled in the art better understand the technical solution of the present application, the relevant contents involved in the technical solution of the present application are introduced below.

[0026] 1) Large Language Model (LLM): It is a large-scale natural language processing model. It is based on deep learning technology, especially the Transformer architecture, and learns language rules by training massive amounts of text data, and generates natural language text that conforms to grammatical and semantic rules. Among them, LLM is widely used in various natural language processing (NLP) tasks, such as text generation, machine translation, question-answering systems, text summarization, etc.

[0027] 2) Speech Transformer model: used to convert audio signals into high-dimensional feature representations so that subsequent models (such as CTC models or language models) can better understand and process speech input.

[0028] 3) CTC model: used to handle the alignment problem between speech input and target text, especially between non-fixed-length input and output sequences. The speech is shortened by CTC-based whitespace filtering, converted by the adapter, and then connected with the text embedding before being input into the T5 encoder. Using the output of the adapter as a query, the retriever finds the top k nearest terms (texts) and adds the terms to the front of the text input.

[0029] 4) T5 model: in the embodiments of the present application, it is used to uniformly process multimodal input (such as voice and text input) and generate text output.

[0030] 5) Speech2Text Adapter: The speech features output from the speech encoder are converted and adapted to match the input requirements of the language model. Specifically, the feature sequence output by the speech encoder is further processed by the Adapter to meet the needs of downstream tasks such as dialogue state tracking and text generation.

[0031] In the relevant technical background, with the continuous advancement of voice interaction and natural language processing technology, intelligent interactive applications can more accurately understand user needs and provide more intelligent services. However, in some specific technical fields, such as place names and station names with territorial characteristics, it is very easy to make mistakes in the process of translating the entries from voice to text, which leads to poor performance of subsequent intelligent answers. Among them, the above-mentioned intelligent applications can be, for example, intelligent voice assistants, voice customer service, intelligent networked car voice navigation, etc., which track the dialogue status and understand complex dialogue context system applications.

[0032] At present, the current intelligent interactive applications mainly obtain stronger context understanding capabilities by accessing LLM. In the face of terms such as restaurants and stations that are easily misidentified during the speech-to-text conversion process, a small hot word capture pre-training model is used to enhance the extracted features. The specific process is as follows: Steps 1 to 4:

[0033] Step 1: Get a training dataset containing hot words.

[0034] Step 2: Use a large pre-trained model (such as BERT, GPT, etc.) to extract common audio features from each speech data in the training dataset.

[0035] Step 3: Use a small hot word capture pre-trained model to enhance the extracted general audio features to achieve recognition of specific hot words;

[0036] Step 4: Perform weighted connection on the output features of the large pre-trained model and the small hot word capture pre-trained model, and input the weighted connected features into the large pre-trained model to generate the final text translation result through decoding fusion.

[0037] However, the above-mentioned related technical solutions optimize the hot words in a specific technical field and improve the accuracy of hot word recognition. However, the above-mentioned hot word capture pre-training model depends on the quality and coverage of the hot word list, and has limited effect on non-hot word recognition, especially for the understanding ability of easy-to-error and important terms. The accuracy of rare terms and new terms cannot be guaranteed.

[0038] Therefore, the inventors believe that the existing technical solutions cannot provide high accuracy for both hot word and non-hot word recognition, and there is an urgent need for a voice interaction method with a wide recognition range and high accuracy.

[0039] In view of the above problems, the exemplary embodiment of the present disclosure proposes a voice interaction method, which receives a voice signal input by a target user, extracts features of the voice signal, and obtains a first feature sequence of the voice signal; searches for an initial major category term matching the first feature sequence from a term database, and determines a target major category term based on a first similarity between the first feature sequence and the initial major category term; extracts an initial subcategory term from the initial major category term, and determines a target subcategory term based on a second similarity between the first feature sequence and the initial subcategory term; adds the target major category term and the target subcategory term to the historical conversation text of the target user to obtain a first input sequence, so as to obtain a target output text based on the first input sequence. The method converts the problem of text generation based on voice data into a similarity matching problem by matching the feature sequence of the voice signal to terms with high similarity from the term database layer by layer, thereby reducing the difficulty of the task without generating candidate terms. On the other hand, similarity can improve the accuracy of term matching and is not limited by the quality and coverage of the hot word list. It is not only suitable for hot word recognition, but also for non-hot word recognition, further improving the adaptation range while meeting the high accuracy of speech recognition.

[0040] In this regard, we will combine Figure 1 An exemplary description is given of a system architecture in which a voice interaction method and device proposed in the present disclosure are applied to an exemplary application environment.

[0041] Figure 1 The system architecture diagram of the voice interaction method provided in the embodiment of the present application is as follows: Figure 1As shown, the system 100 includes a terminal device 101 and a server 102. The terminal device 101 and the server 102 can be directly or indirectly connected to each other through one or more networks 103. The network 103 can be a wired network or a wireless network. For example, the wireless network can be a mobile cellular network or a wireless fidelity (Wireless-Fidelity, WIFI) network. Of course, it can also be other possible networks, which are not limited in the embodiments of the present disclosure. It should be noted that Figure 1 What is shown is just an example. In fact, the number of terminal devices, servers and networks is not limited and can be adjusted according to actual needs. It is not specifically limited in the embodiments of the present disclosure.

[0042] Among them, the terminal device 101 can be, for example, a mobile phone, a tablet computer (Portable Android Device, PAD for short), a laptop computer, a desktop computer, a smart TV, a smart car device, a smart wearable device, a smart TV, and an aircraft, and any other device involving voice interaction. The terminal device 101 can be installed with a target application, for example, the target application can be any application with intelligent customer service and intelligent voice assistant. It should be noted that the above-mentioned target application involved in the embodiment of the present application can be a software client, or it can be a client such as a web page or a small program. The server 102 is a server corresponding to the software or a web page, a small program, etc., and does not limit the specific type of the client.

[0043] The server 102 may be a background server of the target application, used to provide corresponding background services for it, such as voice interaction and translation into text services, etc. It may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, i.e., content delivery networks (CDNs), as well as big data and artificial intelligence platforms, but is not limited thereto.

[0044] It should be noted that the voice interaction method in the embodiment of the present application can be executed by the terminal device 101 or the server 102 alone, or can be executed by the server 102 and the terminal device 101 together, and the embodiment of the present disclosure does not impose any special restrictions on this. Among them, the server 102 and the terminal device 101 can both include one or more processors, memories, and interactive I / O interfaces, etc. The memories of the server 102 and the terminal device 101 can also store the program instructions required to be executed in the voice interaction method provided in the embodiment of the present disclosure, and these program instructions can be used to implement the voice interaction process provided in the embodiment of the present disclosure when executed by the processor.

[0045] It should be noted that when the voice interaction method provided in the embodiment of the present disclosure is executed by the server 102 or the terminal device 101 alone, the above-mentioned application scenario may also only include a single device of the server 102 or the terminal device 101, or the server 102 and the terminal device 101 may also be considered to be the same device. Of course, in actual application, when the voice interaction method provided in the embodiment of the present disclosure is jointly executed by the server 102 and the terminal device 101, the server 102 and the terminal device 101 may also be the same device, that is, the server 102 and the terminal device 101 may be different functional modules of the same device, or virtual devices virtualized from the same physical device.

[0046] It should be noted that the above application scenarios are only shown to facilitate understanding of the spirit and principles of the present disclosure, and the implementation methods of the present disclosure are not limited in this regard.

[0047] After understanding the system architecture of the present disclosure, combined with the application scenarios described above, refer to the attached Figure 2 The following method can be executed by the above-mentioned terminal device or server, or by the terminal device and the server together, and is specifically illustrated here by taking the execution of the terminal device or the server as an example.

[0048] Figure 2 The flowchart of a voice interaction method provided by the embodiment of the present disclosure is as follows. The embodiment of the present disclosure provides a voice interaction method, which can be executed by any device for executing the voice interaction method, and the device can be implemented by software and / or hardware. In this embodiment, the device is integrated in Figure 1 The server 102 shown in FIG. Figure 2 As shown, the voice interaction method provided by the embodiment of the present disclosure may include the following steps S201 to S204:

[0049] S201. Receive a speech signal input by a target user, and perform feature extraction on the speech signal to obtain a first feature sequence of the speech signal.

[0050] S202: Searching for an initial general category term that matches the first feature sequence from a term database, and determining a target general category term based on a first similarity between the first feature sequence and the initial general category term.

[0051] S203: extracting initial sub-category terms from the initial major category terms, and determining target sub-category terms based on a second similarity between the first feature sequence and the initial sub-category terms.

[0052] S204: Add the target major category terms and the target subcategory terms to the historical conversation text of the target user to obtain a first input sequence, so as to obtain a target output text based on the first input sequence.

[0053] Based on the above Figure 2 In the embodiment shown, a speech signal input by a target user is received and a feature extraction is performed on the speech signal to obtain a first feature sequence of the speech signal; an initial major category entry matching the first feature sequence is searched from the entry database, and a target major category entry is determined based on a first similarity between the first feature sequence and the initial major category entry; an initial subcategory entry is extracted from the initial major category entry, and a target subcategory entry is determined based on a second similarity between the first feature sequence and the initial subcategory entry; the target major category entry and the target subcategory entry are added to the historical conversation text of the target user to obtain a first input sequence, so as to obtain a target output text based on the first input sequence. The method converts the problem of text generation based on speech data into a similarity matching problem by matching the feature sequence of the speech signal from the entry database layer by layer with high similarity, thereby eliminating the need to generate candidate entries and reducing the difficulty of the task. On the other hand, the accuracy of entry matching can be improved based on similarity, and is not limited by the quality and coverage of the hot word list, and is not only adapted to hot word recognition, but also to non-hot word recognition, further improving the adaptation range while meeting the high accuracy of speech recognition.

[0054] The following will be combined with specific embodiments to Figure 2 The specific implementation methods of each step in the embodiment shown are described in detail:

[0055] In S201, a speech signal input by a target user is received, and features of the speech signal are extracted to obtain a first feature sequence of the speech signal.

[0056] The target user is a user who logs into the voice interaction application and uses the voice interaction system. Taking the voice interaction application as an intelligent connected car navigation application as an example, the target user is a user who logs into the car navigation application. The voice signal can contain hot words such as landmarks, station names, hotel names, and restaurant names, and the voice signal can also contain non-hot words.

[0057] Exemplarily, the feature extraction of the speech signal can be performed based on a large pre-trained model to obtain a first feature sequence of the speech signal, wherein the large pre-trained model can be, for example, a Speech Transformer encoding model, a BERT model, a GPT model, etc.

[0058] In some application scenarios, the voice signal input by the target user usually carries a lot of noise, especially in outdoor environments. In order to improve the quality of the voice signal, according to some embodiments of the present disclosure, when the voice signal input by the target user is received, the voice signal is denoised.

[0059] In an optional embodiment of the present disclosure, when executing step S201 to extract features of a speech signal to obtain a first feature sequence of the speech signal, features are extracted from the speech signal to obtain an initial feature sequence of the speech signal; redundant frame sequences in the initial feature sequence are deleted based on a connectionist temporal classification (CTC) model to obtain a first feature sequence.

[0060] The redundant frame sequence may be a blank frame sequence, a repeated frame sequence, etc. in the initial feature sequence. The blank frame sequence is a frame sequence in which no speech exists in the speech signal.

[0061] Exemplarily, after extracting features of a speech signal based on a large pre-trained model and obtaining an initial feature sequence, the CTC model can be used to delete redundant frame sequences in the initial feature sequence, thereby facilitating the reduction of the feature sequence corresponding to the speech signal and retaining only the frame sequence containing important information to provide efficient and accurate speech features for subsequent processing.

[0062] In S202, an initial general category term matching the first feature sequence is searched from the term database, and a target general category term is determined based on a first similarity between the first feature sequence and the initial general category term.

[0063] Among them, the major category terms refer to terms with the intent major category attribute. For example, in various fields, the intent major category is usually identified for each term in the term database to obtain the intent major category to which each term in the term database belongs. The intent major category can be understood as the field involved in the term, such as automobiles, education, shopping, finance, landmarks, hotels, etc.

[0064] Exemplarily, the term database contains a plurality of general category terms with intended general category attributes. Through the term database, an initial general category term matching the first feature sequence can be searched, and a first similarity between the first feature sequence and the initial general category term can be determined, so as to filter out the target general category term from the initial general category term based on the first similarity.

[0065] When executing the step of searching the term database for initial major category terms that match the first feature sequence in S202 above, in some example embodiments of the present disclosure, text encoding is performed on each term contained in the term database based on an encoder to obtain an original text sequence; and initial major category terms that match the first feature sequence are searched from the term database based on the original text sequence.

[0066] Exemplarily, the process consists of a dual encoder, wherein the first encoder is used to implement text encoding of each entry in the entry database to obtain an original text sequence; and the second encoder is used to encode the audio input of the target user.

[0067] Since the sequence length of the encoder output will become longer, it is necessary to map the sequence length of the encoder output to a vector of fixed length.

[0068] In order to achieve the above technical effects, in some example embodiments of the present disclosure, after performing text encoding on each term contained in the term database based on the first encoder to obtain the original text sequence, average pooling processing is performed on the original text sequence.

[0069] Average pooling can be performed based on the following formula (1). The average pooling result h avg for:

[0070]

[0071] In formula (1), h avg is the result of average pooling, T is the sequence length of the original text sequence, h t is a vector representation of each time step t, where t∈{1,2,...,T}.

[0072] It should be explained that the above time step is used to represent the unit of time relationship in sequence data. In the fields of speech recognition, natural language processing, bioinformatics, etc., the time step of deep learning is usually used to describe the time sequence of different events or behaviors.

[0073] When executing the above step S202 of determining the target major category terms based on the first similarity between the first feature sequence and the initial major category terms, it can be implemented based on the following embodiments:

[0074] In an optional embodiment of the present disclosure, a first similarity between the first feature sequence and the initial major category terms is determined based on cosine similarity; and the initial major category terms whose first similarity is greater than a first preset similarity are determined as target major category terms.

[0075] Exemplarily, after obtaining the first feature sequence of the speech signal and obtaining the initial major category terms, the first similarity between the first feature sequence and the initial major category terms can be determined based on the cosine similarity.

[0076] Assume that the first feature sequence of the speech signal is Q and the initial category term is C, then the cosine similarity shown in the following formula (2) is used to calculate the similarity between Q and C:

[0077]

[0078] In formula (2), consine_similarity(Q,C i ) is the first similarity between the first feature sequence and the i-th initial category term, Q is the first feature sequence of the speech signal, C i is the i-th initial category term, and N is the total number of retrieved initial category terms.

[0079] After determining the first similarity, the initial category terms whose first similarity is greater than the first preset similarity can be determined as the final target category terms. For example, there are 5 initial category terms, and their corresponding first similarities are 95%, 72%, 50%, 85% and 65%, respectively, and the first preset similarity is 70%, then the initial category terms of 95%, 85% and 72% are determined as the final target category terms.

[0080] In another optional embodiment of the present disclosure, a first similarity between the first feature sequence and the initial major category terms is determined based on cosine similarity; the terms are sorted from large to small according to the first similarity, and the first first preset number of initial major category terms in the sorting result are determined as target major category terms.

[0081] Exemplarily, after determining the first similarity between the first feature sequence and the initial major category terms based on the above formula (2), the multiple initial major category terms can be sorted according to the first similarity, so as to select the first K (i.e., the first preset number) initial major category terms with the highest first similarity as the target major category terms. For example, there are 5 initial major category terms, and their corresponding first similarities are 95%, 72%, 50%, 85% and 65%, respectively, and the first preset number is 3, then the first similarities are sorted from large to small, and the major category terms corresponding to 95%, 85%, and 72% are selected as the target major category terms.

[0082] It should be understood that the items may also be sorted from small to large according to the first similarity, so that the first preset number of initial major category items in the sorting result are determined as target major category items.

[0083] In S203, initial sub-category terms are extracted from the initial major category terms, and target sub-category terms are determined based on the second similarity between the first feature sequence and the initial sub-category terms.

[0084] The initial sub-category terms are sub-category terms in the initial major category terms. For example, if the initial major category term is car navigation, the corresponding sub-category terms include place names, hotel names, station names, public transportation, and the like.

[0085] For example, from the retrieved initial category terms C i Extract the initial sub-category term E j . And use the second encoder to encode the initial sub-category terms E j Encode to calculate the first feature sequence Q and the initial sub-category term E j The second similarity between .

[0086] In an optional embodiment of the present disclosure, a second similarity between the first feature sequence and the initial sub-category terms is determined based on cosine similarity; and the initial sub-category terms whose second similarity is greater than a second preset similarity are determined as target sub-category terms.

[0087] The second similarity calculation method between the first feature sequence and the initial sub-category term is shown in the following formula (3):

[0088]

[0089] In formula (3), consine_similarity(Q,E j ) is the first feature sequence Q and the jth initial subclass term E j The second similarity between them; M is the total number of initial subcategory entries.

[0090] After determining the second similarity, the initial sub-category terms whose second similarity is greater than the second preset similarity can be determined as the final target sub-category terms. For example, if there are five initial sub-category terms, and their corresponding second similarities are 95%, 72%, and 85%, respectively, and the second preset similarity is 80%, then the initial sub-category terms of 95% and 85% are determined as the final target sub-category terms.

[0091] In an optional embodiment of the present disclosure, a second similarity between the first feature sequence and the initial sub-category terms is determined based on cosine similarity; the terms are sorted from large to small according to the second similarity, and the first second preset number of initial sub-category terms in the sorting result are determined as target sub-category terms.

[0092] Exemplarily, after determining the second similarity, the items are sorted from large to small according to the second similarity, so that the first second preset number of initial sub-category items in the sorting result are determined as target sub-category items. For example, the initial sub-category items include sub-category items with similarities of 95%, 85%, and 72%, and the second preset number is 2, then the target sub-category items are sub-category items with similarities of 95% and 85%.

[0093] Through the above embodiment, the task of generating text from speech is converted into a sorting problem, so that the model only needs to sort the retrieved target category terms and target subcategory terms without generating terms, which reduces the difficulty of the task. In addition, through multi-layer retrieval of target category terms and target subcategory terms that meet the similarity requirements, the key terms with the highest similarity to the speech signal input by the target user can be matched, thereby improving the accuracy of voice interaction.

[0094] In S204, the target major category terms and the target subcategory terms are added to the historical conversation text of the target user to obtain a first input sequence, so as to obtain a target output text based on the first input sequence.

[0095] Exemplarily, after obtaining the target major category terms and target subcategory terms in the above steps, Bodian can integrate them, that is, add the target major category terms and target subcategory terms as prefixes to the historical conversation text between the target user and the voice interaction system to form an enhanced first input sequence.

[0096] For example, suppose the target category terms to be retrieved are C1, C2, ..., C k , the target sub-category entries are E1, E2, ..., E k , the historical conversation text of the target user is H, then the enhanced first input sequence obtained after integration is: I = [C1, C2, ..., C k ,E1,E2,...,E k ,H].

[0097] When executing step S204 of obtaining the target output text based on the first input sequence, the first input sequence can be used as the input of the multimodal input model to output the target output text. The multimodal input model can be a T5 model.

[0098] Further, based on any of the above embodiments, Figure 3 Another voice interaction method is exemplified. Figure 3 A flowchart of another voice interaction method according to an exemplary embodiment of the present disclosure is shown, wherein the method executes the following steps S201 and S301 to S302:

[0099] Step S201: receiving a speech signal input by a target user, and performing feature extraction on the speech signal to obtain a first feature sequence of the speech signal.

[0100] Step S301: input the first feature sequence into the Speech2Text adaptation network for correction to obtain a second feature sequence.

[0101] Step S302: concatenate the second feature sequence and the historical conversation text, and determine the concatenated target feature sequence as the second input sequence.

[0102] Exemplarily, the first feature sequence of the speech signal is converted and adapted through the Speech2Text adaptation network so that it can match the input requirements of the model, thereby achieving task requirements such as intelligent dialogue state tracking or text generation.

[0103] In this embodiment, a dual encoder is provided, so that the second feature sequence (speech mode) after conversion of the speech signal and the historical conversation text (text mode) are respectively obtained based on the dual encoder, so that the second feature sequence and the text sequence of the historical conversation text are spliced ​​to obtain a second input sequence.

[0104] After the second input sequence is obtained, the second input sequence can be used as the input of the multimodal input model to output the target output text. The multimodal input model can be a T5 model.

[0105] Furthermore, in order to improve the accuracy of text generation based on speech signals, the first input sequence and the second input sequence obtained above can also be used as inputs of the multimodal input model. Figure 4 An exemplary description is given. Figure 4 A flowchart of a text output method according to an exemplary embodiment of the present disclosure is shown, and the method performs the following steps S401 to S402:

[0106] Step S401: input the first input sequence and the second input sequence into a multimodal input model to obtain an initial output text.

[0107] Step S402: input the initial output text into a natural language processing model to obtain a target output text that complies with grammatical and semantic rules.

[0108] The multimodal input model can be a T5 model, through which multimodal input (such as voice mode, text mode, etc.) can be uniformly processed and text can be generated for output. The natural language processing model LLM is used to take the initial output text as input to generate natural language text that conforms to grammatical and semantic rules.

[0109] It should be explained that the LLM model can be a GPT series model, a BERT series model, etc., and the embodiments of the present disclosure do not impose any special restrictions on this.

[0110] The following will take the navigation application scenario of intelligent connected cars as an example. Figure 5 The entire voice interaction process of the voice interaction method according to the exemplary embodiment of the present disclosure is described in detail.

[0111] First, the target user logs in to the navigation application of the smart connected car and enters voice data. For example, navigation to location A, A can be the name of a station, hotel, etc.

[0112] The speech data input by the target user is processed by the Speech Transformer encoder to realize feature extraction of the speech signal, and the CTC model is used to remove redundant frame sequences in the speech feature sequence, so as to use the CTC model for feature reduction to remove redundant frames and retain only important information in the speech data, providing an efficient and accurate speech feature sequence for the subsequent text generation process.

[0113] On the one hand, the speech feature sequence is input into the hierarchical retrieval module, and the general category terms related to the speech feature sequence of the speech data are searched in combination with the term database, and based on the first similarity between the speech feature sequence and each related general category term, the first K general category terms with the highest first similarity are selected, and subcategory terms are extracted from the retrieved general category terms, and the second similarity between the speech feature sequence and each subcategory term is calculated based on the encoder to select the first N general category terms with the highest second similarity (the above process corresponds to Figure 5 Top-K terms shown).

[0114] The first K major category terms and the first N major category terms are added as prefixes to the historical conversation text to obtain the first input sequence as the input of the T5 model.

[0115] On the other hand, based on the Speech2Text adaptation network, the above speech feature sequence is converted and embedded through audio to obtain the converted speech feature sequence so as to adapt to the input requirements of the T5 model.

[0116] In addition, the text input of the navigation voice assistant can be obtained through another encoder, and then the converted voice feature sequence and the historical dialogue text sequence are spliced, and the spliced ​​second input sequence is input into the T5 model for processing. This process combines the voice input and text input of the target user, so that the voice interaction system can fully understand the user's intention and improve the accuracy and adaptability of the user's voice interaction.

[0117] Finally, the first input sequence and the second input sequence are used as inputs of the T5 model to generate an initial output text. In order to make the generated text more in line with actual needs, the initial output text is input into the LLM model to obtain a natural language text that conforms to grammatical and semantic rules.

[0118] Through the above embodiments, the accuracy of speech recognition and understanding can be improved, the term retrieval and matching capabilities can be enhanced, efficient multimodal dialogue support can be provided, and the model training and reasoning processes can be simplified, thereby significantly improving the performance and user experience of the voice navigation system. The above method not only performs well in processing complex dialogue tasks, but also has good scalability and adaptability, and is suitable for various voice interaction scenarios.

[0119] It should be noted that although the steps of the method in the present disclosure are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0120] Furthermore, this example implementation also provides a voice interaction device.

[0121] Figure 6 Schematically shows a block diagram of a voice interaction device according to an exemplary embodiment of the present disclosure. Figure 6 According to an exemplary embodiment of the present disclosure, a voice interaction device 600 may include a feature extraction module 601, a first determination module 602, a second determination module 603, and a term adding module 604, wherein:

[0122] The feature extraction module 601 is used to receive a speech signal input by a target user and perform feature extraction on the speech signal to obtain a first feature sequence of the speech signal;

[0123] A first determination module 602 is used to search the term database for an initial category term that matches the first feature sequence, and determine a target category term based on a first similarity between the first feature sequence and the initial category term;

[0124] A second determination module 603 is used to extract initial sub-category terms from the initial major category terms, and determine target sub-category terms based on a second similarity between the first feature sequence and the initial sub-category terms;

[0125] The term adding module 604 is used to add target major category terms and target subcategory terms to the historical conversation text of the target user to obtain a first input sequence, so as to obtain a target output text based on the first input sequence.

[0126] In some example embodiments of the present disclosure, the feature extraction module 601 is specifically used to extract features from a speech signal to obtain an initial feature sequence of the speech signal; and to delete redundant frame sequences in the initial feature sequence based on a connection temporal classification CTC model to obtain a first feature sequence.

[0127] In some example embodiments of the present disclosure, the first determination module 602 is used to perform text encoding on each term contained in the term database based on an encoder to obtain an original text sequence; and search the term database for initial major category terms that match the first feature sequence based on the original text sequence.

[0128] In some example embodiments of the present disclosure, the apparatus may further include a processing module, and the processing module is configured to perform average pooling processing on the original text sequence.

[0129] In some example embodiments of the present disclosure, a first determination module 602 is used to determine a first similarity between a first feature sequence and an initial major category term based on cosine similarity; determine the initial major category terms whose first similarity is greater than a first preset similarity as target major category terms, or, sort the terms from large to small according to the first similarity, and determine the first first preset number of initial major category terms in the sorting result as target major category terms; a second determination module 603 is used to determine a second similarity between a first feature sequence and an initial subcategory term based on cosine similarity; determine the initial subcategory terms whose second similarity is greater than a second preset similarity as target subcategory terms, or, sort the terms from large to small according to the second similarity, and determine the first second preset number of initial subcategory terms in the sorting result as target subcategory terms.

[0130] In some example embodiments of the present disclosure, the processing module can also be used to input the first feature sequence into the Speech2Text adaptation network for modification to obtain a second feature sequence; splice the second feature sequence and the text sequence of the historical conversation text, and determine the spliced ​​target feature sequence as the second input sequence.

[0131] In some example embodiments of the present disclosure, the device may also include an input module, which can be used to input the first input sequence and the second input sequence into a multimodal input model to obtain an initial output text; and input the initial output text into a natural language processing model to obtain a target output text that conforms to grammatical and semantic rules.

[0132] The voice interaction device 600 provided in the embodiment of the present disclosure can execute the technical solution of the voice interaction method in any of the above-mentioned embodiments. Its implementation principle and beneficial effects are similar to the implementation principle and beneficial effects of the voice interaction method. Please refer to the implementation principle and beneficial effects of the voice interaction method, and no further details will be given here.

[0133] Since the functional modules of the program running performance analysis device according to the embodiment of the present invention are the same as those in the above-mentioned method invention embodiment, they will not be described in detail here.

[0134] In an exemplary embodiment of the present disclosure, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of the present specification is stored. In some possible implementations, various aspects of the present invention may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present invention described in the above "Exemplary Method" section of the present specification.

[0135] The program product for implementing the above method according to an embodiment of the present invention may adopt a portable compact disc read-only memory (CD-ROM) and include program code, and may be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, apparatus, or device.

[0136] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0137] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0138] The program code contained in the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, radio frequency (RF), etc., or any suitable combination of the above.

[0139] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, and the like, as well as conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the user computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0140] In an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided.

[0141] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as a system, method or program product. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as a "circuit", "module" or "system".

[0142] Refer to the following Figure 7 An electronic device 700 according to this embodiment of the present invention will be described. Figure 7 The electronic device 700 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0143] like Figure 7 As shown, the electronic device 700 is in the form of a general computing device. The components of the electronic device 700 may include, but are not limited to: the at least one processing unit 710, the at least one storage unit 720, a bus 730 connecting different system components (including the storage unit 720 and the processing unit 710), and a display unit 740.

[0144] The storage unit stores program codes, which can be executed by the processing unit 710, so that the processing unit 710 performs the steps of various exemplary embodiments of the present invention described in the above “Exemplary Method” section of this specification. For example, the processing unit 710 can perform the following steps: Figure 2 Steps S201 to S204 shown in FIG.

[0145] The storage unit 720 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 7201 and / or a cache storage unit 7202 , and may further include a read-only storage unit (ROM) 7203 .

[0146] The storage unit 720 may also include a program / utility 7204 having a set (at least one) of program modules 7205, such program modules 7205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0147] Bus 730 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0148] The electronic device 700 may also communicate with one or more external devices 1000 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 700, and / or communicate with any device that enables the electronic device 700 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 750. Furthermore, the electronic device 700 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 760. As shown, the network adapter 760 communicates with other modules of the electronic device 700 via a bus 730. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 700, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems, etc.

[0149] Through the description of the above implementation, it is easy for those skilled in the art to understand that the example implementation described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the implementation of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, including several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation of the present disclosure.

[0150] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0151] It should be noted that, although several modules or units of the device for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0152] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0153] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A voice interaction method, characterized in that: include: Receiving a speech signal input by a target user, and performing feature extraction on the speech signal to obtain a first feature sequence of the speech signal; Searching for an initial category term matching the first feature sequence from a term database, and determining a target category term based on a first similarity between the first feature sequence and the initial category term; Extracting initial sub-category terms from the initial major category terms, and determining target sub-category terms based on a second similarity between the first feature sequence and the initial sub-category terms; The target major category terms and the target subcategory terms are added to the historical conversation text of the target user to obtain a first input sequence, so as to obtain a target output text based on the first input sequence.

2. The method according to claim 1, characterized in that The step of extracting features from the speech signal to obtain a first feature sequence of the speech signal comprises: Extracting features from the speech signal to obtain an initial feature sequence of the speech signal; Redundant frame sequences in the initial feature sequence are deleted based on a connection temporal classification (CTC) model to obtain the first feature sequence.

3. The method according to claim 1, characterized in that The step of searching the entry database for an initial major category entry that matches the first feature sequence comprises: Performing text encoding on each entry contained in the entry database based on an encoder to obtain an original text sequence; Based on the original text sequence, an initial category term matching the first feature sequence is searched from the term database.

4. The method according to claim 3, characterized in that After the step of performing text encoding on each entry contained in the entry database based on the encoder to obtain an original text sequence, the method further includes: Perform average pooling processing on the original text sequence.

5. The method according to claim 1, characterized in that The step of determining the target major category term based on the first similarity between the first feature sequence and the initial major category term, or the step of determining the target subcategory term based on the second similarity between the first feature sequence and the initial subcategory term, comprises: Determine a first similarity between the first feature sequence and the initial major category terms based on cosine similarity; Determine the initial major category terms whose first similarity is greater than the first preset similarity as the target major category terms, or sort the initial major category terms from large to small according to the first similarity, and determine the first first preset number of initial major category terms in the sorting result as the target major category terms; Alternatively, determining a second similarity between the first feature sequence and the initial sub-category term based on cosine similarity; The initial subcategory terms whose second similarity is greater than a second preset similarity are determined as the target subcategory terms, or the initial subcategory terms are sorted from large to small according to the second similarity, and the first second preset number of initial subcategory terms in the sorting result are determined as the target subcategory terms.

6. The method according to claim 1, characterized in that The method further comprises: Inputting the first feature sequence into the Speech2Text adaptation network for correction to obtain a second feature sequence; The second feature sequence and the text sequence of the historical conversation text are concatenated, and the concatenated target feature sequence is determined as the second input sequence.

7. The method according to claim 6, characterized in that The step of obtaining a target output text based on the first input sequence comprises: Inputting the first input sequence and the second input sequence into a multimodal input model to obtain an initial output text; The initial output text is input into a natural language processing model to obtain the target output text that complies with grammatical and semantic rules.

8. A voice interaction device, characterized in that: include: A feature extraction module, used for receiving a speech signal input by a target user, and performing feature extraction on the speech signal to obtain a first feature sequence of the speech signal; A first determination module, configured to search an initial category term matching the first feature sequence from a term database, and determine a target category term based on a first similarity between the first feature sequence and the initial category term; A second determination module, configured to extract initial sub-category terms from the initial major category terms, and determine target sub-category terms based on a second similarity between the first feature sequence and the initial sub-category terms; The entry adding module is used to add the target major category entry and the target subcategory entry to the historical conversation text of the target user to obtain a first input sequence, so as to obtain a target output text based on the first input sequence.

9. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the voice interaction method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; Wherein, the processor is configured to execute the voice interaction method described in any one of claims 1 to 7 by executing the executable instructions.

Images