Acoustic semantic large model, server, voice interaction method and storage medium

Abstract

The invention discloses an acoustic semantic large model, a server, a voice interaction method and a computer readable storage medium. The acoustic semantic large model comprises an acoustic coding module, a character transcription module, a knowledge retrieval module and a large language model module. The acoustic coding module is configured to generate an acoustic feature vector of a voice request according to the input voice request. The character transliteration module is configured to transliterate the voice request into a corresponding character sequence, and the character sequence comprises characters corresponding to characters in the voice request. The knowledge retrieval module is configured to obtain supplementary information from an external knowledge base according to the character sequence. The large language model module is configured to determine a natural language processing result according to the acoustic feature vector and the supplementary information. Thus, through the end-to-end acoustic semantic large model, serial processing of a plurality of modules is reduced, the time delay of processing the voice request is reduced, the model response speed is improved, and thus the user experience is enhanced.

Description

Technical Field

[0001] The present application relates to the field of voice interaction, and more specifically, to an acoustic semantic large model, a server, a voice interaction method, and a computer-readable storage medium. Background Art

[0002] In the related art, to facilitate users to drive a vehicle, an in-vehicle intelligent assistant usually performs serial processing of voice requests through multiple modules, including performing acoustic rejection processing on the voice request through an acoustic rejection recognition module, performing voice recognition processing on the voice request through a voice recognition module, performing retrieval processing on the voice request based on a knowledge retrieval module, and performing natural language processing on the voice request using a natural language processing module. However, in this way, it is necessary to wait for each serial module to complete the processing of the voice request before a natural language understanding result that can meet the user's needs can be generated, resulting in a relatively long overall time consumption and a poor user experience. Summary of the Invention

[0003] The present application provides an acoustic semantic large model, a server, a voice interaction method, and a computer-readable storage medium.

[0004] An embodiment of the present application provides an acoustic semantic large model, which includes an acoustic encoding module, a character transcription module, a knowledge retrieval module, and a large language model module;

[0005] The acoustic encoding module is configured to generate an acoustic feature vector of the voice request according to the input voice request;

[0006] The character transcription module is configured to transcribe the voice request into a corresponding character sequence, and the character sequence includes characters corresponding to each word in the voice request;

[0007] The knowledge retrieval module is configured to obtain supplementary information from an external knowledge base according to the character sequence;

[0008] The large language model module is configured to determine a natural language processing result according to the acoustic feature vector and the supplementary information.

[0009] Thus, the acoustic semantic large model includes an acoustic encoding module, a character transcription module, a knowledge retrieval module, and a large language model module. Among them, the acoustic encoding module can generate an acoustic feature vector of the voice request according to the input voice request. The character transcription module can transcribe the voice request into a corresponding character sequence, and the character sequence includes the characters corresponding to each word in the voice request. The knowledge retrieval module can obtain supplementary information from an external knowledge base according to the character sequence. The large language model module can determine the natural language processing result according to the acoustic feature vector and the supplementary information. In this way, through the end-to-end acoustic semantic large model, the serial processing of multiple modules is reduced, the latency of processing voice requests is reduced, the model response speed is improved, and thus the user experience is enhanced. Moreover, by matching Chinese pinyin or English letters for knowledge retrieval, the difficulty of Chinese character decoding is avoided, the recognition accuracy of voice requests is improved, and thus the semantic understanding ability and robustness of the acoustic semantic large model are enhanced.

[0010] In some embodiments, the acoustic encoding module is configured to:

[0011] Perform feature extraction processing on the voice request to generate semantic acoustic features;

[0012] Perform encoding processing on the semantic acoustic features to generate the acoustic feature vector.

[0013] Thus, the acoustic encoding module can perform feature extraction processing on the voice request to generate semantic acoustic features, and can perform encoding processing on the semantic acoustic features to generate an acoustic feature vector. In this way, by extracting semantic acoustic features, the semantic information of the voice signal can be accurately captured, and the influence of factors such as noise on speech recognition can be reduced, improving the semantic understanding ability and robustness of the acoustic semantic large model.

[0014] In some embodiments, the character transcription module is configured to:

[0015] Based on a preset algorithm, perform mapping processing on the acoustic feature vector to transcribe the voice request into the character sequence.

[0016] Thus, the character transcription module can perform mapping processing on the acoustic feature vector based on a preset algorithm to transcribe the voice request into a character sequence. In this way, by performing mapping processing on the acoustic feature vector through the preset algorithm, the voice signal can be accurately converted into a character sequence, providing a basis for subsequent knowledge retrieval.

[0017] In some embodiments, the external knowledge base is constructed based on key information and value information, where the key information is constructed based on Chinese pinyin and English letters, and the value information is constructed based on Chinese characters;

[0018] The external knowledge base includes a basic knowledge base, and iteratively updates the basic knowledge base at a predetermined period.

[0019] In this way, the external knowledge base is constructed based on key information and value information, where the key information is constructed based on Chinese pinyin and English letters, and the value information is constructed based on Chinese characters. In addition, the external knowledge base includes a basic knowledge base, and iteratively updates the basic knowledge base at a predetermined period. In this way, since the indexing method of pinyin and letters is simpler and faster than directly using Chinese characters, by using Chinese pinyin and English letters as key information, indexing and retrieval can be performed quickly. And, by iteratively updating the basic knowledge base at a predetermined period, the timeliness of the content in the external knowledge base can be ensured.

[0020] In some embodiments, the characters include Chinese pinyin and / or English letters, and the knowledge retrieval module is configured to:

[0021] Determine candidate key information corresponding to the character sequence from the external knowledge base according to the character sequence;

[0022] Determine candidate value information corresponding to the candidate key information as the candidate information according to the candidate key information;

[0023] Determine the supplementary information from the candidate information according to the sorting result of the citation times of the candidate information.

[0024] In this way, the knowledge retrieval module can determine candidate key information corresponding to the character sequence from the external knowledge base according to the character sequence. Then, the knowledge retrieval module can determine candidate value information corresponding to the candidate key information as candidate information according to the candidate key information. Finally, the knowledge retrieval module can determine supplementary information from the candidate information according to the sorting result of the citation times of the candidate information. In this way, by matching the candidate key information and the character sequence, it can be ensured that the retrieved supplementary information matches the character sequence input by the user, thereby improving the accuracy of knowledge retrieval. And, by sorting the citation times of the candidate information, information with the highest citation times can be preferentially selected, thereby reasonably handling the situation of polysemous characters.

[0025] In some embodiments, the large language model module is configured to:

[0026] Perform fusion processing on the acoustic feature vector and the supplementary information to generate fusion information;

[0027] Determine the natural language processing result according to the fusion information.

[0028] In this way, the large language model module can fuse and process the acoustic feature vectors and supplementary information to generate fused information. Then, the large language model module can determine the natural language processing result based on the fused information. Thus, by fusing the acoustic feature vectors and supplementary information, it can help the large language model accurately understand the user's intention, thereby improving the recognition accuracy and enhancing the user experience.

[0029] In some embodiments, the large language model module is configured to:

[0030] Perform slot recognition on the fused information to obtain a slot recognition result;

[0031] Perform prediction of the application programming interface on the fused information to obtain a predicted application interface;

[0032] Based on the slot recognition result and the predicted application interface, select the predicted application interface to perform application programming interface parameter filling to obtain the natural language processing result.

[0033] In this way, the large language model module can perform slot recognition on the fused information to obtain a slot recognition result. Then, the large language model module can perform prediction of the application programming interface on the fused information to obtain a predicted application interface. Finally, the large language model module can select the predicted application interface to perform application programming interface parameter filling based on the slot recognition result and the predicted application interface to obtain the natural language processing result. Thus, through slot recognition, the large language model can accurately understand the meaning of each entity in the user's instruction, thereby enhancing the user experience. Through application programming interface prediction and parameter filling, the large language model can select a suitable interface according to the user's instruction and fill in the corresponding parameters to achieve flexible instruction processing, thereby enhancing the user experience.

[0034] An embodiment of the present application provides a server, and the above acoustic semantic large model is deployed on the server.

[0035] An embodiment of the present application provides a voice interaction method, and the method includes:

[0036] Obtain the current voice request;

[0037] Based on the acoustic semantic large model, determine a vehicle control instruction corresponding to the current voice request according to the current voice request, and perform the voice interaction.

[0038] In this way, obtain the current voice request. Then, based on the acoustic semantic large model, determine a vehicle control instruction corresponding to the current voice request according to the current voice request, and perform the voice interaction. Thus, based on the above acoustic semantic large model, the user's voice request can be processed quickly and accurately, enhancing the user experience.

[0039] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the voice interaction method as described above are implemented.

[0040] Additional aspects and advantages of the embodiments of the present application will be given in part in the following description, become apparent in part from the following description, or be understood through the practice of the embodiments of the present application. Description of the Drawings

[0041] The above and / or additional aspects and advantages of the present application will become apparent and be readily understood from the description of the embodiments in conjunction with the following drawings, where:

[0042] Figure 1 is a schematic structural diagram of the acoustic semantic large model according to an embodiment of the present application;

[0043] Figure 2 is one of the schematic flowcharts of the voice interaction method according to an embodiment of the present application;

[0044] Figure 3 is another schematic flowchart of the voice interaction method according to an embodiment of the present application;

[0045] Figure 4 is yet another schematic flowchart of the voice interaction method according to an embodiment of the present application;

[0046] Figure 5 is still another schematic flowchart of the voice interaction method according to an embodiment of the present application;

[0047] Figure 6 is one of the schematic flowcharts of the voice interaction method according to an embodiment of the present application;

[0048] Figure 7 is another schematic flowchart of the voice interaction method according to an embodiment of the present application;

[0049] Figure 8 is still another schematic flowchart of the voice interaction method according to an embodiment of the present application. Detailed Embodiments

[0050] The embodiments of the present application will be described in detail below. Examples of the embodiments are shown in the drawings, where the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the drawings are exemplary only for explaining the embodiments of the present application and should not be construed as limiting the embodiments of the present application.

[0051] In the related art, in order to facilitate users to interact safely and effectively while driving a vehicle, an in-vehicle intelligent assistant is often used to assist users in operating the vehicle in a timely manner. For the voice requests issued by users, the in-vehicle intelligent assistant generally uses multiple modules to process the voice requests serially:

[0052] First, the acoustic rejection module performs acoustic rejection processing on the user's voice request. The acoustic rejection model analyzes the audio quality of the voice request to determine whether it is noise or damaged audio. If the audio quality of the voice request is poor (such as noisy background or intermittent), it is determined as an invalid voice request and directly rejected, ending the process. If the audio quality of the voice request is good, it is determined as a valid voice request and the voice request is released to the next module.

[0053] Next, the valid voice request is sent to the speech recognition module. The speech recognition module can convert the user's voice request into text, that is, decode the voice signal and convert it into understandable text information. However, if the user's voice request is not expressed standardly, the speech recognition module may not be able to convert the user's voice request into the correct text information. For example, if the user's voice request is "I miss my mom, help me call her", the speech recognition module may convert the voice request into "I miss my horse, help me call her" due to accent problems. In this way, the subsequent steps will be based on the incorrect text "I miss my horse, help me call her", which cannot meet the user's needs.

[0054] Then, the converted text information enters the knowledge retrieval module. The knowledge retrieval module can retrieve relevant information in the knowledge base of the in-vehicle intelligent assistant according to the user's request. If relevant entries are retrieved, they are used as supplementary knowledge. If not retrieved (such as no matching content or similarity lower than the threshold), no additional knowledge is added. For example, if the user asks about nearby restaurants, the knowledge retrieval module will search for the user's current location information and retrieve information about nearby restaurants.

[0055] Finally, the retrieved information enters the natural language processing module. The natural language processing module can understand the meaning of the text information and supplementary information, including but not limited to lexical analysis, syntactic analysis, semantic understanding, etc., and finally generate a natural language understanding result that meets the user's needs.

[0056] However, the above serial processing process needs to wait for each module to complete processing in turn, resulting in a relatively long overall response time and a poor user experience, especially in scenarios that require quick response, such as navigation or emergency calls.

[0057] Based on the above problems, please refer to Figure 1, an embodiment of the present application provides an acoustic semantic large model 1000, which includes an acoustic encoding module 100, a character transcription module 200, a knowledge retrieval module 300, and a large language model module 400.

[0058] Among them, the acoustic encoding module 100 can generate an acoustic feature vector of the voice request according to the input voice request;

[0059] The character transcription module 200 can transcribe the voice request into a corresponding character sequence, and the character sequence includes characters corresponding to each word in the voice request;

[0060] The knowledge retrieval module 300 can obtain supplementary information from an external knowledge base according to the character sequence;

[0061] The large language model module 400 can determine the natural language processing result according to the acoustic feature vector and the supplementary information.

[0062] Specifically, the acoustic semantic large model 1000 is an end-to-end system. By integrating acoustic encoding processing, pinyin transcription processing, knowledge retrieval processing, and natural language processing, it realizes end-to-end processing, avoiding the complexity of separate training and tuning between each module in the traditional solution, thus improving the response speed of the system. In some embodiments, the acoustic semantic large model 1000 is usually constructed using deep learning techniques, such as convolutional neural networks (CNNs), recurrent neural networks (RNNs), or Transformer models, etc., which are not limited herein. The acoustic semantic large model 1000 can effectively capture the complex features and semantic information of the voice signal, thereby improving the accuracy of speech recognition and semantic understanding. Moreover, by transcribing into a character sequence, the acoustic semantic large model 1000 can adapt to different speakers, accents, and noise environments, and has strong robustness. It should be noted that the acoustic semantic large model 1000 can not only be deployed in in-vehicle intelligent assistants, but also has broad application prospects in fields such as smart homes and intelligent customer service.

[0063] The acoustic encoding module 100 is a module in the acoustic semantic large model 1000 responsible for converting the voice signal into an acoustic feature vector. Through signal processing techniques, such as Fourier transform, filter banks, etc., the voice signal is converted into an acoustic feature vector, providing data information support for subsequent tasks. The acoustic encoding module 100 is equivalent to the combination of the acoustic rejection recognition module and the speech recognition module in the traditional solution, and can realize some functions of the acoustic rejection recognition module and the speech recognition module. In this way, the acoustic encoding module 100 integrates acoustic feature extraction and speech recognition, realizing end-to-end processing, thereby improving the response speed of the system.

[0064] A voice request refers to an instruction or question sent by a user to the intelligent cockpit system in a voice manner, including various types of information. For example, "Play song B by singer A", "Navigate to Beijing", "What's the weather like today", "What restaurants are nearby", "Turn on the air conditioner", and "Tell a joke", etc.

[0065] An acoustic feature vector refers to a multi-dimensional vector extracted from a voice request that can characterize the acoustic characteristics of the voice request, including important information such as pitch, intensity, and timbre of the voice request, and is the basis for voice processing tasks such as speech recognition, speech synthesis, and speaker recognition.

[0066] The character transcription module 200 is a module in the acoustic semantic large model 1000 that converts the acoustic feature vector output by the speech recognition model into a pinyin sequence, provides input for the knowledge retrieval module 300, and improves the accuracy of the large language model module 400.

[0067] A character sequence refers to a sequence composed of characters such as Chinese pinyin and / or English letters, which is used to represent the text information in a voice request and can be used to retrieve knowledge related to the voice request.

[0068] The knowledge retrieval module 300 is a module in the acoustic semantic large model 1000 that retrieves relevant knowledge from an external knowledge base according to the character sequence, can obtain supplementary information from the external knowledge base, and apply it to subsequent modules.

[0069] The external knowledge base stores various knowledge information and can provide a knowledge source for the knowledge retrieval module 300.

[0070] Supplementary information refers to knowledge related to the pinyin sequence retrieved from the external knowledge base, provides additional knowledge for the large language model module 400, helps it better understand the input text, and make a more accurate response.

[0071] The large language model module 400 refers to a module in the acoustic semantic large model 1000 that determines the natural language processing result according to the acoustic feature vector and supplementary information, can understand and generate natural language, and perform semantic understanding and intent recognition, and output natural language processing results such as application programming interface API and corresponding filling parameters. In some embodiments, the large language model in the large language model module 400 can be a Transformer-based model such as BERT, which is not limited herein.

[0072] First, the acoustic encoding module 100 receives the input voice request and converts the voice request into an acoustic feature vector. Next, the character transcription module 200 transcribes the voice request into a corresponding character sequence, where the character sequence includes the characters corresponding to each word in the voice request, namely Chinese pinyin and / or English letters. Since the granularity of Chinese pinyin and English letters is coarser, the decoding accuracy is higher. Then, the knowledge retrieval module 300 retrieves additional knowledge related to the user input from an external knowledge base according to the pinyin sequence, such as song information, place name information, etc. Finally, the large language model module 400 combines the acoustic feature vector and the retrieved additional knowledge to understand the user's intention and generate a natural language processing result, such as API and Arguments.

[0073] In summary, the acoustic semantic large model 1000 provided by this application includes an acoustic encoding module 100, a character transcription module 200, a knowledge retrieval module 300, and a large language model module 400. Among them, the acoustic encoding module 100 can generate an acoustic feature vector of the voice request according to the input voice request. The character transcription module 200 can transcribe the voice request into a corresponding character sequence, and the character sequence includes the characters corresponding to each word in the voice request. The knowledge retrieval module 300 can obtain supplementary information from an external knowledge base according to the character sequence. The large language model module 400 can determine the natural language processing result according to the acoustic feature vector and the supplementary information. In this way, through the end-to-end acoustic semantic large model 1000, the serial processing of multiple modules is reduced, the latency of processing the voice request is reduced, the model response speed is improved, and thus the user experience is enhanced. Moreover, by performing knowledge retrieval through the matching of Chinese pinyin or English letters, the difficulty of Chinese character decoding is avoided, the recognition accuracy of the voice request is improved, and thus the semantic understanding ability and robustness of the acoustic semantic large model 1000 are enhanced.

[0074] In some embodiments, the acoustic encoding module 100 can perform feature extraction processing on the voice request to generate semantic acoustic features. And it can perform encoding processing on the semantic acoustic features to generate an acoustic feature vector.

[0075] Specifically, the feature extraction processing refers to the process of extracting useful information related to tasks such as speech recognition and semantic understanding from the original voice request, including noise processing, normalization processing, frame processing, feature calculation processing, and feature normalization processing, etc. The feature extraction processing is similar to the process of human auditory system perceiving sound, that is, extracting useful information from sound, such as the pitch, volume, timbre, etc. of the sound, and understanding the meaning of the sound based on this information. The feature extraction processing converts the speech signal into features that can be understood by a computer by simulating the process of the human auditory system perceiving sound, so as to perform subsequent tasks such as speech recognition and semantic understanding.

[0076] Semantic acoustic features refer to acoustic features related to semantic information extracted from voice request signals, and are the bridge between voice signals and semantic information. Semantic acoustic features include information related to semantics in voice request signals, such as phoneme features, syllable features, and prosodic features.

[0077] Among them, phoneme features refer to the acoustic features related to a single phoneme, including phoneme duration, phoneme energy, and phoneme frequency. Phoneme is the smallest unit used to describe speech in phonetics, and it has the function of distinguishing meaning. For example, the pronunciation difference between "mama" and "ma" in Chinese lies in the different phonemes of "a" and "a", the former is an open sound and the latter is a closed sound. Phoneme duration refers to the duration of a phoneme in speech. Phoneme energy refers to the energy intensity of a phoneme. Phoneme frequency refers to the frequency component of a phoneme.

[0078] Syllable features refer to the acoustic features associated with a single syllable, including syllable duration, syllable energy, syllable duration distribution, and syllable energy distribution. A syllable is the basic unit of speech structure, consisting of one or more phonemes, including a core phoneme. A syllable is the smallest unit that can be pronounced independently in a language. For example, "啊" and "衣" in Chinese are both syllables. Syllable duration refers to the duration of a syllable in speech. Syllable energy refers to the energy intensity of a syllable. Syllable duration distribution refers to the distribution of durations of each phoneme within a syllable. Syllable energy distribution refers to the distribution of energy of each phoneme within a syllable.

[0079] Prosodic features refer to acoustic features related to speech rhythm, including intonation, speaking rate, and pauses. Intonation refers to the rise and fall of speech, such as rising, falling, and flat tones. Speaking rate refers to the speed of speech. Pauses refer to the pause position and duration of speech.

[0080] Taking the voice request "play song B by singer A" as an example, the semantic acoustic features include the following:

[0081] Phoneme features: for example, the duration and energy of phonemes such as "b", "o", "f", "a", and "ng".

[0082] Syllable features: for example, the duration, energy, and duration distribution of syllables such as “bo”, “fang”, “ge”, “shou”, “A”, “ge”, “qu”, and “B”.

[0083] Rhythmic features: such as the rise and fall of intonation, the speed of speech, the position of pauses, etc.

[0084] An acoustic feature vector refers to converting a voice request signal into a mathematical form that a computer can understand, including information related to semantics in the voice signal, which can be used for subsequent natural language processing and knowledge retrieval, etc. The acoustic feature vector usually includes multiple eigenvalues such as Mel-frequency Cepstral Coefficients (MFCC) eigenvalues, phoneme eigenvalues, syllable eigenvalues, and prosody eigenvalues. The Mel-frequency Cepstral Coefficients (MFCC) eigenvalue is a commonly used voice feature parameter, which extracts features by converting the voice signal into the Mel-frequency domain and performing discrete cosine transform on the logarithmic spectrum.

[0085] The acoustic encoding module 100 first extracts features from the input voice request to generate semantic acoustic features. Then, the acoustic encoding module 100 encodes the semantic acoustic features into acoustic feature vectors. These acoustic feature vectors are points in a high-dimensional space and can represent the complex characteristics of the voice signal.

[0086] In this way, the acoustic encoding module 100 can perform feature extraction processing on the voice request to generate semantic acoustic features, and can perform encoding processing on the semantic acoustic features to generate acoustic feature vectors. In this way, by extracting semantic acoustic features, the semantic information of the voice signal can be accurately captured, and the influence of factors such as noise on speech recognition can be reduced, improving the semantic understanding ability and robustness of the acoustic semantic large model 1000.

[0087] In some embodiments, the character transcription module 200 can perform mapping processing on the acoustic feature vector based on a preset algorithm to transcribe the voice request into a character sequence.

[0088] Specifically, the character transcription module 200 is a pre-trained model that can map acoustic feature vectors to character sequences. In some embodiments, the character transcription module 200 can use one of the Recurrent Neural Network, Convolutional Neural Network, and transformer deep neural network models as the base model. Subsequently, the base model is trained according to preset training data, and a corresponding preset algorithm is selected to construct the character transcription module 200. It should be noted that when training the base model according to the preset training data, the main learning is to map acoustic feature vectors to Chinese pinyin and how to map acoustic feature vectors to 26 English letters. Compared with training using Chinese characters, by using pinyin and English letters for training, the character transcription module 200 has a coarser modeling granularity, lower modeling difficulty, and higher decoding accuracy for the trained character transcription module 200. For example, for the user voice request "I miss my mom (due to regional accent, pronounced in the third tone), help me call her", if directly converted to Chinese characters, it might be "I miss my horse, help me call her", which would affect subsequent processing. However, when converted to Chinese pinyin "wo xiang wo ma le, bang wo dadian hua gei ta", it can be discovered and corrected during the subsequent processing by the large language model module 400.

[0089] In addition, when using pinyin and English letters for modeling, knowledge matching can be performed in the external knowledge base according to pinyin and English letters. For example, it is possible to match pinyin sequences such as "ge shou A" and "ge qu B", thereby improving the accuracy of knowledge retrieval. Moreover, when using pinyin and English letters for modeling, new words, hot words, and place names can be better processed. Even if these words are not seen during model training, they can be matched through pinyin. For example, new words such as "ge qu B" can be decoded, thereby improving the robustness of the character transcription module 200.

[0090] The preset algorithm refers to the decoding algorithm in the character transcription module 200, which is used to guide the process of character sequence transcription. In some embodiments, the preset algorithm can be any one of the greedy algorithm, Beam Search algorithm, Prefix Beam Search algorithm, and Modified Beam Search algorithm, and is not limited here. In the example given in the embodiments of the present application, considering the latency requirements of the acoustic semantic large model 1000, the greedy algorithm is used as the preset algorithm. The greedy algorithm is a heuristic search algorithm that tries to find the optimal solution to a problem by choosing the currently seemingly optimal option at each step.

[0091] The pinyin transliteration module receives the acoustic feature vector output by the acoustic encoding module 100. Then, the pinyin transliteration module uses a preset algorithm to map the acoustic feature vector and convert the acoustic feature vector into a character sequence corresponding to the voice request input into the acoustic semantic model 1000.

[0092] In this way, the character transcription module 200 can map the acoustic feature vector based on a preset algorithm to transcribe the voice request into a character sequence. In this way, by mapping the acoustic feature vector through a preset algorithm, the voice signal can be accurately converted into a character sequence, providing a basis for subsequent knowledge retrieval.

[0093] In some embodiments, the external knowledge base is constructed based on key information and value information, wherein the key information is constructed based on Chinese pinyin and English letters, and the value information is constructed based on Chinese characters. In addition, the external knowledge base includes a basic knowledge base, and the basic knowledge base is iteratively updated at a predetermined period.

[0094] Specifically, the external knowledge base is constructed based on key information and value information. Among them, the key information (Key) is constructed by Chinese pinyin and English letters, which means that each entry in the external knowledge base is indexed by pinyin and letters instead of directly using Chinese characters. Compared with directly using Chinese characters for matching, pinyin matching can better cope with changes such as new words, hot words and place names. For example, a user can say "qi che shou ce", and the external knowledge base can find relevant information of "car manual". The value information (Value) is constructed by Chinese characters, that is, the specific knowledge content is described using Chinese characters, and each key information corresponds to one or more value information. In some embodiments, the construction process of the external knowledge base is as follows: First, data collection is performed, and knowledge data such as lyrics, movie lines, news, etc. are collected from sources such as the Internet, books, and encyclopedias. Second, data cleaning is performed, and the collected data is cleaned to remove noise, duplicate information, etc., and segmentation and part-of-speech tagging are performed. Third, the cleaned data is constructed into a key-value pair form, wherein the key information is constructed based on Chinese pinyin and English letters, and the value information is constructed based on Chinese characters. For example, the key information of "car" is set to "qi che" and the value information is set to "car is a means of transportation". Fourth, knowledge base construction, the constructed key-value pairs are stored in the external knowledge base and organized in alphabetical order.

[0095] The external knowledge base takes the basic knowledge base as the main part of the external knowledge base, which includes a large amount of basic knowledge and information. In addition, the basic knowledge base will be updated regularly to ensure the timeliness and accuracy of the knowledge and better meet the needs of users. The update cycle is predetermined and can be daily, weekly, monthly, etc.

[0096] In this way, the external knowledge base is constructed based on key information and value information, where the key information is constructed based on Chinese pinyin and English letters, and the value information is constructed based on Chinese characters. In addition, the external knowledge base includes a basic knowledge base, and the basic knowledge base is iteratively updated at a predetermined period. In this way, since the indexing method of pinyin and letters is simpler and faster than directly using Chinese characters, by using Chinese pinyin and English letters as key information, indexing and retrieval can be performed quickly. Moreover, by iteratively updating the basic knowledge base at a predetermined period, the timeliness of the content in the external knowledge base can be ensured.

[0097] In some embodiments, the knowledge retrieval module 300 is capable of determining candidate key information corresponding to the character sequence from the external knowledge base according to the character sequence.

[0098] Next, the knowledge retrieval module 300 is capable of determining the candidate value information corresponding to the candidate key information as candidate information according to the candidate key information.

[0099] Finally, the knowledge retrieval module 300 is capable of determining supplementary information from the candidate information according to the sorting result of the reference times of the candidate information.

[0100] Specifically, the candidate key information is information that matches the character sequence input by the user and can be used as a clue for retrieving the knowledge base, that is, an "index" or "keyword", which can help us narrow the retrieval scope and quickly find information related to the user input. For example, if the input character sequence is "qi che", the candidate key information may be "qi che an quan", "qi che jiashi qing kuang", and "qi che bao yang qing kuang", etc.

[0101] The candidate value information refers to the information extracted from the external knowledge base corresponding to the candidate key information. The candidate value information is related to the input character sequence and can help the user obtain more comprehensive information, that is, "detailed information", which provides more specific information related to the user input.

[0102] The candidate information refers to all knowledge fragments retrieved from the external knowledge base related to the input character sequence during the knowledge retrieval process.

[0103] The reference count of candidate information refers to the number of times each candidate information is cited in the knowledge base, which can be understood as the importance or popularity of the candidate information. The reference count sorting result refers to sorting these candidate information according to their importance or popularity based on the reference count of the candidate information. For example, when the input character sequence is "ge shou A" for knowledge retrieval, the following three candidate information related to "ge shou A" are retrieved from the knowledge base: Candidate Information A: ge shou A, a famous singer, with representative works such as "ge qu B", "ge qu C", etc. Candidate Information B: ge shou A, a well-known actor, with representative works such as "ying shi D", "ying shi F", etc. Candidate Information C: ge shou A, a famous basketball player, playing for a well-known team. Assume that Candidate Information A has the highest reference count, Candidate Information B has the second-highest reference count, and Candidate Information C has the lowest reference count. Then, according to the reference count sorting result of the candidate information, ranked from high to low, these three candidate information will be sorted as: Candidate Information A, Candidate Information B, and Candidate Information C. In this way, through the sorting of candidate information, the information with the highest reference count can be preferentially selected, thus better handling the situation of polysemous words.

[0104] It should be noted that the knowledge retrieval module 300 adopts the Retrieval Augmented Generation (RAG) technology. The RAG knowledge retrieval technology is a technology that combines information retrieval and natural language generation, aiming to enhance the performance of the model by using an external knowledge base.

[0105] In this way, the knowledge retrieval module 300 can determine the candidate key information corresponding to the character sequence from the external knowledge base according to the character sequence. Then, the knowledge retrieval module 300 can determine the candidate value information corresponding to the candidate key information as the candidate information. Finally, the knowledge retrieval module 300 can determine the supplementary information from the candidate information according to the reference count sorting result of the candidate information. In this way, through the matching of the candidate key information and the character sequence, it can be ensured that the retrieved supplementary information matches the character sequence input by the user, thereby improving the accuracy of knowledge retrieval. And, through the sorting of the reference count of candidate information, the information with the highest reference count can be preferentially selected, thus reasonably handling the situation of polysemous words.

[0106] In some embodiments, the large language model module 400 can perform fusion processing on the acoustic feature vector and the supplementary information to generate fusion information.

[0107] Then, the large language model module 400 can determine the natural language processing result according to the fusion information.

[0108] Specifically, the fusion process refers to combining the acoustic feature vector and supplementary information so that the large language model module 400 can better understand and process the user's voice request. In some embodiments, the fusion methods include feature concatenation and feature weighted concatenation, etc. Feature concatenation means directly concatenating the acoustic feature vector and supplementary information into a longer feature vector. Feature weighted concatenation means weighting the acoustic feature vector and supplementary information and then performing concatenation.

[0109] The acoustic feature vector and supplementary information are input into the large language model module 400, which will perform a fusion process on this information, combine the acoustic feature vector and supplementary information, and generate more comprehensive fusion information for semantic understanding. The acoustic feature vector contains rich acoustic information, and the supplementary information provides additional semantic information. The fusion of the two can help the large language model more accurately understand the user's intention, thereby improving the recognition accuracy.

[0110] Next, the large language model module 400 determines the natural language processing result based on the fusion information, such as identifying the user's instruction, answering the user's question, etc.

[0111] In this way, the large language model module 400 can perform a fusion process on the acoustic feature vector and supplementary information to generate fusion information. Next, the large language model module 400 can determine the natural language processing result based on the fusion information. In this way, by fusing the acoustic feature vector and supplementary information, it can help the large language model accurately understand the user's intention, thereby improving the recognition accuracy and enhancing the user experience.

[0112] In some embodiments, the large language model module 400 can perform slot recognition on the fusion information to obtain the slot recognition result.

[0113] Next, the large language model module 400 can perform prediction of the application program interface on the fusion information to obtain the predicted application interface.

[0114] Finally, the large language model module 400 can select the predicted application interface according to the slot recognition result and the predicted application interface, perform parameter filling of the application program interface, and obtain the natural language processing result.

[0115] Specifically, slot identification refers to extracting specific information fragments from the user's input, which are usually called "slots". Slots are usually the key information necessary to complete a certain task or request, such as time, location, object, etc. Taking the user's voice request "What's the temperature tomorrow?" as an example, the slot information obtained through slot identification includes ["tomorrow" - Date], that is, the slot information includes the slot value and the slot type, where "tomorrow" is the slot value and Date is the slot type. Taking the user's voice request "Navigate to address Q" as an example, the slot information obtained through slot identification is ["address Q" - Place], where "Zhongguancun" is the slot value and Place is the slot type. Slot identification is an important task in natural language processing, and its goal is to identify specific entities and attributes in the text and map them to predefined slots. Slot identification can help the large language model module 400 understand the user's intention and generate more accurate and natural responses.

[0116] The slot identification result refers to the named entities obtained by performing slot identification on the voice request. Such as the above slot information ["tomorrow" - Date] and slot information ["address Q" - Place], etc.

[0117] Application programming interface prediction refers to predicting the operation type corresponding to the input text according to the semantics of the input text and generating the corresponding application programming interface call instruction. Application programming interface prediction can help the large language model module 400 understand the user's intention and perform the corresponding operations.

[0118] Application programming interface parameter filling is a task in natural language processing, and its goal is to specify specific values for the parameters in the application programming interface call instruction according to the semantics of the input text and the application programming interface call instruction.

[0119] The large language model module 400 first performs slot identification on the fused information, decomposing the user's instruction into different slots, such as song name, singer name, and play mode, etc. Then, the large language model module 400 performs application programming interface prediction according to the fused information, predicting the application programming interfaces that the user needs to execute, such as "play song" and "query weather", etc. Finally, the large language model module 400 selects the corresponding predicted application programming interface to perform application programming interface parameter filling according to the slot identification result and the predicted application interface, such as filling the song name and singer name as parameters into the "play song" interface.

[0120] Finally, the large language model module 400 outputs the natural language processing result, such as identifying the user's instruction as "play song" and calling the music player to play the song specified by the user.

[0121] In this way, the large language model module 400 can perform slot recognition on the fused information to obtain the slot recognition result. Then, the large language model module 400 can perform application programming interface prediction on the fused information to obtain the predicted application interface. Finally, the large language model module 400 can select the predicted application interface according to the slot recognition result and the predicted application interface, and execute the filling of the application programming interface parameters to obtain the natural language processing result. In this way, through slot recognition, the large language model can accurately understand the meaning of each entity in the user instruction, thereby improving the user experience. Through application programming interface prediction and parameter filling, the large language model can select the appropriate interface according to the user instruction and fill in the corresponding parameters to achieve flexible instruction processing, thereby improving the user experience.

[0122] An embodiment of the present application provides a server, on which the above-mentioned acoustic semantic large model 1000 is deployed.

[0123] Specifically, the server is a hardware device that runs application programs and provides services, and can provide a running environment for the acoustic semantic large model 1000, including computing resources, storage space, network connections, etc. And it can deploy the trained acoustic semantic large model 1000 to the online environment so that it can provide services externally. It should be noted that when the computing power of the vehicle is sufficient for the acoustic semantic large model 1000 to use, the server can be deployed locally on the vehicle. Otherwise, it is deployed in the cloud.

[0124] The server receives the voice input from the user and returns the processing result to the user.

[0125] In this way, the server integrates the acoustic semantic large model 1000 into a complete system, realizes the end-to-end process from voice input to natural language processing result output, quickly responds to the user's voice instruction, and provides accurate results, thereby improving the user experience.

[0126] Please refer to Figure 2 , an embodiment of the present application provides a voice interaction method, and the method includes:

[0127] 01: Obtain the current voice request;

[0128] 02: Based on the acoustic semantic large model, determine the vehicle control instruction corresponding to the current voice request according to the current voice request, and perform voice interaction.

[0129] Specifically, the system receives the current voice request issued by the user through hardware devices such as microphones. Then, the acoustic semantic large model 1000 understands and analyzes the current voice request, recognizes the user's intention and target, and converts them into vehicle control instructions that the vehicle can understand.

[0130] Please refer to Figure 3, in some embodiments, step 02 (based on the acoustic semantic large model, determine the vehicle control instruction corresponding to the current voice request according to the current voice request) includes:

[0131] 021: Generate an acoustic feature vector of the voice request according to the input voice request;

[0132] 022: Transcribe the voice request into a corresponding character sequence;

[0133] 023: Obtain supplementary information from an external knowledge base according to the character sequence;

[0134] 024: Determine the natural language processing result according to the acoustic feature vector and the supplementary information.

[0135] In this way, through the end-to-end acoustic semantic large model, the serial processing of multiple modules is reduced, the latency of processing voice requests is reduced, the model response speed is improved, and thus the user experience is enhanced. Moreover, by performing knowledge retrieval through the matching of Chinese pinyin or English letters, the difficulty of Chinese character decoding is avoided, the recognition accuracy of voice requests is improved, and thus the semantic understanding ability and robustness of the acoustic semantic large model are enhanced.

[0136] Please refer to Figure 4 , in some embodiments, step 021 (generate an acoustic feature vector of the voice request according to the input voice request) includes:

[0137] 0211: Perform feature extraction processing on the voice request to generate semantic acoustic features;

[0138] 0212: Perform encoding processing on the semantic acoustic features to generate an acoustic feature vector.

[0139] In this way, by extracting semantic acoustic features, the semantic information of the voice signal can be accurately captured, and the influence of factors such as noise on speech recognition can be reduced, improving the semantic understanding ability and robustness of the acoustic semantic large model.

[0140] Please refer to Figure 5 , in some embodiments, step 022 (transcribe the voice request into a corresponding character sequence) includes:

[0141] 0221: Based on a preset algorithm, perform mapping processing on the acoustic feature vector to transcribe the voice request into a character sequence.

[0142] In this way, by performing mapping processing on the acoustic feature vector through the preset algorithm, the voice signal can be accurately converted into a character sequence, providing a basis for subsequent knowledge retrieval.

[0143] In some embodiments, the external knowledge base is constructed based on key information and value information, where the key information is constructed based on Chinese pinyin and English letters, and the value information is constructed based on Chinese characters. Moreover, the external knowledge base includes a basic knowledge base, and the basic knowledge base is iteratively updated at a predetermined period.

[0144] Thus, since the indexing method of pinyin and letters is simpler and faster than directly using Chinese characters, by using Chinese pinyin and English letters as the key information, indexing and retrieval can be performed quickly. Also, by iteratively updating the basic knowledge base at a predetermined period, the timeliness of the content in the external knowledge base can be ensured.

[0145] Please refer to Figure 6 , in some embodiments, the character includes Chinese pinyin and / or English letters, and step 023 (obtaining supplementary information from the external knowledge base according to the character sequence) includes:

[0146] 0231: Determine the candidate key information corresponding to the character sequence from the external knowledge base according to the character sequence;

[0147] 0232: Determine the candidate value information corresponding to the candidate key information as the candidate information according to the candidate key information;

[0148] 0233: Determine the supplementary information from the candidate information according to the sorting result of the reference times of the candidate information.

[0149] Thus, by matching the candidate key information with the character sequence, it can be ensured that the retrieved supplementary information matches the character sequence input by the user, thereby improving the accuracy of knowledge retrieval. Also, by sorting the reference times of the candidate information, the information with the highest reference times can be preferentially selected, thus reasonably handling the situation of polysemous characters.

[0150] Please refer to Figure 7 , in some embodiments, step 024 (determining the natural language processing result according to the acoustic feature vector and the supplementary information) includes:

[0151] 0241: Perform a fusion process on the acoustic feature vector and the supplementary information to generate fusion information;

[0152] 0242: Determine the natural language processing result according to the fusion information.

[0153] Thus, by fusing the acoustic feature vector and the supplementary information, it can help the large language model accurately understand the user's intention, thereby improving the recognition accuracy and enhancing the user experience.

[0154] Please refer to Figure 8 , in some embodiments, step 0242 (determining the natural language processing result according to the fusion information) includes:

[0155] 02421: Perform slot recognition on the fused information to obtain the slot recognition result;

[0156] 02422: Perform application programming interface prediction on the fused information to obtain the predicted application interface;

[0157] 02423: According to the slot recognition result and the predicted application interface, select the predicted application interface to perform application programming interface parameter filling to obtain the natural language processing result.

[0158] In this way, through slot recognition, the large language model can accurately understand the meanings of various entities in the user instruction, thereby improving the user experience. Through application programming interface prediction and parameter filling, the large language model can select the appropriate interface according to the user instruction and fill in the corresponding parameters to achieve flexible instruction processing, thereby improving the user experience.

[0159] It should be noted that the voice interaction method provided in the implementation of this application is implemented based on the aforementioned acoustic semantic large model 1000. Specifically, each step of the voice interaction method depends on each module of the acoustic semantic large model 1000. That is, step 021 and its sub-steps 0211 and 0212 are based on the acoustic encoding module 100, step 022 and its sub-step 0221 are based on the character transcription module 200, step 023 and its sub-steps 0231, 0232, and 0233 are based on the knowledge retrieval module 300, and step 024, sub-steps 0241 and 0242 of step 024, and sub-sub-steps 02421, 02422, and 02423 of sub-step 0242 are based on the large language model module 400. Among them, for the technical terms and explanatory notes involved in the method steps of the voice interaction method, please refer to the detailed description of each module of the acoustic semantic large model, which will not be elaborated here.

[0160] In summary, the voice interaction method provided by the implementation of this application, which is implemented based on the above-mentioned acoustic semantic large model 1000, can quickly and accurately process the user's voice request and enhance the user experience.

[0161] This application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the vehicle control method as described above are implemented.

[0162] It can be understood that a computer program includes computer program code. The computer program code can be in the form of source code, object code, executable file or some intermediate form, etc. The computer-readable storage medium can include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM), random access memory (RAM), and software distribution medium, etc.

[0163] In the description of this specification, the descriptions referring to terms such as "specifically", "furthermore", "specially", "understandably", etc. are intended to mean that the specific features, structures, materials or characteristics described in connection with the embodiments or examples are included in at least one embodiment or example of the present application. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in a suitable manner in any one or more embodiments or examples. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0164] Any process or method description shown in the flowchart or described in other ways herein can be understood as representing a module, segment or part of the code of executable instructions including one or more steps for implementing a specific logical function or process, and the scope of the preferred embodiments of the present application includes additional implementations, where the functions can be executed in a substantially simultaneous manner or in a reverse order according to the involved functions, rather than in the order shown or discussed, which should be understood by those skilled in the technical field of the embodiments of the present application.

[0165] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limiting the present application. Those of ordinary skill in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present application.

Claims

1. A large acoustic semantic model, characterized in that: The acoustic semantic large model includes an acoustic encoding module, a character transcription module, a knowledge retrieval module and a large language model module; The acoustic encoding module is configured to generate an acoustic feature vector of the voice request according to the input voice request; The character transcription module is configured to transcribe the voice request into a corresponding character sequence, wherein the character sequence includes characters corresponding to each word in the voice request; The knowledge retrieval module is configured to obtain supplementary information from an external knowledge base according to the character sequence; The large language model module is configured to determine a natural language processing result based on the acoustic feature vector and the supplementary information.

2. The acoustic semantic big model according to claim 1, characterized in that: The acoustic encoding module is configured to: Performing feature extraction processing on the voice request to generate semantic acoustic features; The semantic acoustic feature is encoded to generate the acoustic feature vector.

3. The acoustic semantic big model according to claim 1, characterized in that: The character transcription module is configured as follows: Based on a preset algorithm, the acoustic feature vector is mapped to transcribe the voice request into the character sequence.

4. The acoustic semantic big model according to claim 1, characterized in that: The external knowledge base is constructed based on key information and value information, wherein the key information is constructed based on Chinese pinyin and English letters, and the value information is constructed based on Chinese characters; The external knowledge base includes a basic knowledge base, and the basic knowledge base is iteratively updated at a predetermined period.

5. The acoustic semantic big model according to claim 4, characterized in that: The characters include Chinese pinyin and / or English letters, and the knowledge retrieval module is configured to: According to the character sequence, determining candidate key information corresponding to the character sequence from the external knowledge base; According to the candidate key information, determining the candidate value information corresponding to the candidate key information as the candidate information; The supplementary information is determined from the candidate information according to the result of sorting the number of citations of the candidate information.

6. The acoustic semantic big model according to claim 1, characterized in that: The large language model module is configured as follows: fusing the acoustic feature vector and the supplementary information to generate fused information; The natural language processing result is determined according to the fusion information.

7. The acoustic semantic big model according to claim 6, characterized in that: The large language model module is configured as follows: Performing slot identification on the fusion information to obtain a slot identification result; Performing application program interface prediction on the fusion information to obtain a predicted application interface; According to the slot identification result and the predicted application interface, the predicted application interface is selected to execute application program interface parameter filling to obtain the natural language processing result.

8. A server, characterized in that: The server includes a processor and a memory, wherein a computer program is stored in the memory. When the computer program is executed by the processor, the acoustic semantic large model described in any one of claims 1 to 7 is implemented.

9. A voice interaction method, characterized in that: The voice interaction method is based on the acoustic semantic big model according to any one of claims 1 to 7, and the method comprises: Get the current voice request; Based on the acoustic semantic big model and according to the current voice request, a vehicle control instruction corresponding to the current voice request is determined to perform the voice interaction.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the steps of the method according to claim 9 are implemented.

Images