Voice recognition methods, devices, cockpits, vehicles, and storage media

Abstract

This application relates to a speech recognition method, device, cockpit, vehicle, and storage medium. The method includes: receiving a voice request from a user inside the cockpit; encoding the voice request to be recognized to obtain a first acoustic feature vector; retrieving a matching second acoustic feature vector from an acoustic knowledge base based on the first acoustic feature vector; the acoustic knowledge base is obtained by speech synthesis based on text information in a text knowledge base, wherein each second acoustic feature vector in the acoustic knowledge base has a mapping relationship with the text feature vector corresponding to the text information; concatenating the text feature vector mapped from the matching second acoustic feature vector with the first acoustic feature vector, and inputting the concatenation into a language model for processing to obtain the corresponding speech recognition result for cockpit response. The solution provided by this application can efficiently retrieve external knowledge, correctly identify new words in voice commands, ensure the accuracy of speech recognition results, and reliably respond to user needs.

Description

Technical Field

[0001] This application relates to the field of speech recognition technology, and in particular to a speech recognition method, device, cockpit, vehicle and storage medium. Background Technology

[0002] An LLM (Large Language Model) is an artificial intelligence model built using deep learning techniques, specifically designed to process and generate natural language text. Training an LLM requires processing massive amounts of data to accurately understand speech content and generate relevant text.

[0003] In today's digital age, the emergence of new and trending online terms is accelerating, and information dissemination is becoming increasingly rapid. Training LLM models is costly and time-consuming. Immediately updating LLM models to capture the semantics and usage of these emerging terms requires not only significant cost and time investment but may also lead to a decline in model stability and performance due to frequent training updates. To address this challenge, Retrieval-Augmented Generation (RAG) technology is employed to inject relevant knowledge into LLM models, enabling them to quickly understand and adapt to the ever-emerging new and trending online terms. Specifically, RAG technology provides additional context and external knowledge to the LLM model by retrieving information related to user questions from a text knowledge base, thereby enhancing the accuracy and relevance of the content generated by the LLM model.

[0004] With the increasing intelligence of vehicles, users are interacting with in-vehicle systems via voice. In in-vehicle dialogue scenarios, user commands are not presented in text form; the LLM model needs to first perform text recognition on the user's voice. However, since voice and text belong to different modalities, RAG technology cannot perform similarity matching within the same semantic space. Consequently, it cannot retrieve accurate relevant information for the LLM model to generate content, thus affecting the in-vehicle system's accurate recognition and response to user voice commands, impacting the user experience. Summary of the Invention

[0005] To address or partially address the problems existing in related technologies, this application provides a speech recognition method, device, cockpit, vehicle, and storage medium, which can efficiently retrieve external knowledge, correctly identify novel words in voice commands, ensure the accuracy of speech recognition results, and reliably respond to user needs.

[0006] The first aspect of this application provides a speech recognition method, which includes: Receive voice requests from users inside the cockpit; The voice request to be recognized is feature-encoded to obtain the first acoustic feature vector; Based on the first acoustic feature vector, a matching second acoustic feature vector is retrieved from a pre-constructed acoustic knowledge base; wherein, the acoustic knowledge base is obtained by speech synthesis based on text information in a pre-constructed text knowledge base, and each second acoustic feature vector in the acoustic knowledge base has a mapping relationship with the text feature vector corresponding to the text information; The text feature vector mapped by the matched second acoustic feature vector is concatenated with the first acoustic feature vector and then input into the language model for processing to obtain the corresponding speech recognition result for cockpit response.

[0007] In some implementations, the acoustic knowledge base is obtained by synthesizing speech based on text information in a preset text knowledge base, including: Collect the user's reference speech; synthesize the corresponding audio information from the text knowledge base based on the timbre features in the reference speech; extract the corresponding second acoustic feature vector based on the audio information to generate the user's corresponding acoustic knowledge base, wherein the second acoustic feature vector includes the user's timbre features.

[0008] In some implementations, the language model includes a pre-trained acoustic encoder; Based on the audio information, the corresponding second acoustic feature vector is extracted to generate the corresponding acoustic knowledge base, including: The network parameters of each layer of the acoustic encoder are fixed, and the audio information is input into the acoustic encoder to obtain the corresponding second acoustic feature vector.

[0009] In some implementations, the first acoustic feature vector and the second acoustic feature vector are obtained by encoding and embedding through the same acoustic encoder, and both the first acoustic feature vector and the second acoustic feature vector contain the timbre features of the same user.

[0010] In some implementations, a matching second acoustic feature vector is retrieved from a pre-built acoustic knowledge base, including: The acoustic knowledge base is searched according to preset matching rules to obtain a second acoustic feature vector that has feature similarity to the first acoustic feature vector.

[0011] In some implementations, the speech recognition results include the API results to be responded to and the parameters.

[0012] In some implementations, the method further includes: when the text information in the text knowledge base is updated, updating the second acoustic feature vector in the acoustic knowledge base accordingly, and saving the mapping relationship with the updated text feature vector.

[0013] A second aspect of this application provides a voice recognition device, comprising: The voice receiving module is used to receive voice requests from users inside the cockpit. The feature encoding module is used to encode the voice request to be recognized to obtain a first acoustic feature vector; The retrieval enhancement module is used to retrieve a matching second acoustic feature vector from a pre-built acoustic knowledge base based on the first acoustic feature vector. The result generation module is used to concatenate the text feature vector mapped by the matched second acoustic feature vector and the first acoustic feature vector, and then input the concatenation into the language model for processing to obtain the corresponding speech recognition result for the cockpit response. The storage module is used to store the acoustic knowledge base, which is obtained by speech synthesis based on text information in a preset text knowledge base. Each second acoustic feature vector in the acoustic knowledge base has a mapping relationship with the text feature vector corresponding to the text information.

[0014] A third aspect of this application provides a cockpit that includes a voice recognition device as described in the second aspect.

[0015] A fourth aspect of this application provides a vehicle, comprising: Processor; and A memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the speech recognition method as described in the first aspect above.

[0016] The fifth aspect of this application provides a computer-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the speech recognition method as described in the first aspect above.

[0017] The sixth aspect of this application provides a computer program product, the computer program product including computer instructions, which, when executed by a processor, implement the speech recognition method described above.

[0018] The technical solution provided in this application may include the following beneficial effects: The speech recognition method of this application first searches the acoustic knowledge base using the first acoustic feature vector corresponding to the speech request, i.e., performs acoustic feature matching under the same modality to quickly find a second acoustic feature vector similar to the first acoustic feature vector. Then, it further finds the uniquely mapped text feature vector in the text knowledge base, thereby quickly obtaining accurate external knowledge and integrating it into the speech request. This design enables RAG technology to efficiently and accurately complete retrieval in multimodal scenarios, and integrates the correct text feature vector into the first acoustic feature vector as input data for the language model, ensuring that the language model can output accurate speech recognition results. It also eliminates the intermediate speech-to-text processing step, meeting end-to-end speech recognition requirements, accelerating system response time, accurately responding to user needs, and optimizing user experience.

[0019] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0020] The above and other objects, features and advantages of this application will become more apparent from the following description of exemplary embodiments of this application in conjunction with the accompanying drawings, wherein the same reference numerals generally represent the same components in the exemplary embodiments of this application.

[0021] Figure 1 This is a first flowchart illustrating a speech recognition method shown in this application; Figure 2 This is a schematic diagram of the second process of a speech recognition method shown in this application; Figure 3 This is a schematic diagram of the application framework of a speech processing method shown in this application; Figure 4 This is a first structural schematic diagram of the speech recognition device shown in this application; Figure 5 This is a second structural schematic diagram of the speech recognition device shown in this application; Figure 6 This is a structural schematic diagram of the cockpit shown in this application; Figure 7 This is a schematic diagram of the vehicle structure shown in this application. Detailed Implementation

[0022] Embodiments of this application will now be described in more detail with reference to the accompanying drawings. While embodiments of this application are shown in the drawings, it should be understood that this application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to make this application more thorough and complete, and to fully convey the scope of this application to those skilled in the art.

[0023] The terminology used in this application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application. The singular forms “a,” “the,” and “the” used in this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used herein refers to and includes any or all possible combinations of one or more of the associated listed items.

[0024] It should be understood that although the terms "first," "second," "third," etc., may be used in this application to describe various information, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0025] In related technologies, to improve system response time and optimize user experience in in-vehicle dialogue scenarios, an end-to-end language model is adopted. This model directly outputs semantic commands based on the input audio for the vehicle's infotainment system to execute, thus eliminating the intermediate process of converting speech into text. However, precisely because there is no text output stage, it is impossible to use RAG technology to inject external knowledge into the language model, resulting in inaccurate speech recognition results for response execution and impacting user experience.

[0026] To address the aforementioned issues, this application provides a speech recognition method that can efficiently retrieve external knowledge, correctly identify emerging words in voice commands, ensure the accuracy of speech recognition results, and enable the system to reliably respond to user needs.

[0027] The technical solution of this application is described in detail below with reference to the accompanying drawings.

[0028] See Figure 1 This application discloses a speech recognition method, which includes: S110 receives voice requests from users inside the cockpit.

[0029] Taking a vehicle as an example, when a user interacts with the computer via voice in the cabin, the voice requests (i.e., user queries) made by the user in the cabin can be collected in real time through the microphone in the vehicle.

[0030] S120: The voice request to be recognized is feature-encoded to obtain the first acoustic feature vector.

[0031] In this application, a pre-trained acoustic encoder can be used to extract and encode features from a speech request, generating an embedding vector, which is the corresponding first acoustic feature vector. In other words, the first acoustic feature vector is an acoustic feature vector generated in real time based on the user's instantaneous speech.

[0032] S130, based on the first acoustic feature vector, a matching second acoustic feature vector is retrieved from a pre-constructed acoustic knowledge base; wherein, the acoustic knowledge base is obtained by speech synthesis based on text information in a pre-set text knowledge base, and each second acoustic feature vector in the acoustic knowledge base has a mapping relationship with the text feature vector corresponding to the text information.

[0033] The acoustic knowledge base in this application is a pre-built acoustic feature vector information database. This acoustic knowledge base can be pre-stored locally, i.e., in the vehicle, so that the RAG module can quickly perform retrieval and matching. The text knowledge base can be stored locally and / or on a cloud server.

[0034] Specifically, current RAG technology uses a text knowledge base that stores a large amount of text information. This application can pre-convert text to speech (TTS) using relevant technologies to generate corresponding audio information from the text information. Optionally, the converted audio information can be aggregated to form an audio knowledge base. Further, by performing feature encoding on each audio information separately, corresponding second acoustic feature vectors are obtained and aggregated for storage, generating a corresponding acoustic knowledge base. That is, the second acoustic feature vector refers to the vector pre-stored in the acoustic knowledge base. Simultaneously, the text information in the text knowledge base stores corresponding text feature vectors. Then, the text feature vectors of the text information are mapped one-to-one with the second acoustic feature vectors of the corresponding audio information to obtain the mapping relationship between the text feature vectors and the second acoustic feature vectors.

[0035] It should be noted that the text knowledge base can be updated and stored in real time or periodically, including various emerging terms and trending words on the internet. In some implementations, when the text information in the text knowledge base is updated, the second acoustic feature vector in the acoustic knowledge base is updated accordingly, and the mapping relationship between the updated text feature vector and the second acoustic feature vector is saved. It can be understood that the updated acoustic knowledge base is stored locally on the vehicle. By updating the acoustic knowledge base in real time or periodically, each updated text feature vector in the text knowledge base has a corresponding second acoustic feature vector in the acoustic knowledge base. This design ensures that the updated second acoustic feature vector is saved in the acoustic knowledge base in a timely manner, and simultaneously saves the mapping relationship between the updated text feature vector and the corresponding second acoustic feature vector, thereby ensuring the provision of the latest knowledge to enhance the timeliness of the retrieval function.

[0036] In this step, the first acoustic feature vector is used as the retrieval target. The retrieval is carried out in the pre-built acoustic knowledge base, and the second acoustic feature vector that has similarity to the first acoustic feature vector is found as the matching target acoustic feature vector.

[0037] S140: The text feature vector mapped by the matched second acoustic feature vector is concatenated with the first acoustic feature vector and then input into the language model for processing to obtain the corresponding speech recognition result for cockpit response.

[0038] It is understood that each second acoustic feature vector in the acoustic knowledge base has a uniquely mapped text feature vector. Therefore, after retrieving the second acoustic feature vector that matches the first acoustic feature vector in S130, the corresponding text feature vector can be further determined in the text knowledge base based on the mapping relationship. In this step, the text feature vector is concatenated with the first acoustic feature vector in S120 to obtain the corresponding concatenated vector as input data for the language model. It should be understood that when the text knowledge base is stored locally, it can be directly concatenated with the corresponding text feature vector. When the text knowledge base is stored in the cloud, the vehicle can receive the mapped text feature vector sent from the cloud for concatenation.

[0039] The text knowledge base stores various emerging words, including new internet slang and trending terms. If a user's voice request contains emerging words, the system can first search the acoustic knowledge base using the first acoustic feature vector corresponding to the voice request. This involves feature matching within the same modality to quickly find a second acoustic feature vector similar to the first. Then, it can further find the uniquely mapped text feature vector in the text knowledge base, thereby quickly obtaining accurate external knowledge to enhance the content generated by the language model.

[0040] The speech recognition method of this application enables RAG technology to efficiently and accurately complete retrieval in multimodal scenarios. It concatenates the correct text feature vector with the first acoustic feature vector and injects it into the language model, ensuring that the language model can output accurate speech recognition results. It also eliminates the intermediate speech-to-text processing step, meets end-to-end speech recognition, speeds up system response time, accurately responds to user needs, and optimizes user experience.

[0041] See Figure 2 Another embodiment shown in this application includes a speech recognition method, which includes: S210 receives voice requests from users inside the cockpit.

[0042] The speech recognition method of this application can be applied to the cockpit of a vehicle. A single vehicle can have one or more users, each with their own specific voice timbre, which then issues a speech request with that specific voice timbre. The following steps can be executed independently for each speech request issued by a different user.

[0043] For example, at time T0, user A issues a voice command: "I want to listen to Zhou Moumou's 'Blue and White Porcelain'." Assume that when the language model is trained, the song "Zhou Moumou's Blue and White Porcelain" has not yet been released, meaning this knowledge is not present in the language model's training data.

[0044] S220, the acoustic encoder of the language model performs feature encoding on the speech request to be recognized to obtain a first acoustic feature vector; wherein, the first acoustic feature vector contains the timbre features of the corresponding user.

[0045] The language model in this application can be a large-scale speech-semantic model in related technologies. By combining it with an acoustic encoder, it can directly process speech signals and generate corresponding speech recognition results without needing to convert speech requests into text using ASR technology, thus eliminating intermediate steps. In this step, the speech request is input into the acoustic encoder of the pre-trained language model for feature extraction and encoding embedding, thereby obtaining a first acoustic feature vector containing the timbre features of user A.

[0046] For example, at time T1, after user A's voice command passes through the acoustic encoder, a corresponding first acoustic feature vector (i.e., acoustic embedding) is generated.

[0047] S230, based on the first acoustic feature vector, a search is performed in the acoustic knowledge base according to a preset matching rule to obtain a second acoustic feature vector with feature similarity to the first acoustic feature vector and a corresponding mapped text feature vector; wherein, the acoustic knowledge base is obtained by synthesizing the text information in the preset text knowledge base according to the user's timbre characteristics.

[0048] The acoustic knowledge base of this application is mapped one-to-one with the timbre characteristics of user A. Different users have their own specific timbre, and thus have corresponding timbre characteristics, and consequently have their own corresponding acoustic knowledge base. When the user in S210 is user B, this step searches the pre-built acoustic knowledge base based on user B's timbre characteristics, and then achieves accurate retrieval based on the specific timbre.

[0049] To obtain an acoustic knowledge base for a user's specific timbre, some implementations involve collecting the user's reference speech; synthesizing corresponding audio information from text information in a text knowledge base based on the timbre features in the reference speech; and extracting a corresponding second acoustic feature vector from the audio information to generate the user's corresponding acoustic knowledge base, where the second acoustic feature vector includes the user's timbre features. Specifically, to facilitate rapid acquisition of the user's timbre, reference speech recorded by the user during the initial account registration phase can be used, or reference speech recorded in real-time. For example, during the user registration phase, the user pre-records several sentences of reference speech, such as recording three "Hello Xiao P" phrases. The reference speech can be customized and is not limited here. After recording, the reference speech is input into a text-to-speech model (e.g., a TTS model) in related technologies. The TTS model extracts the timbre features from the reference speech and synthesizes audio information with the user's specific timbre from the text information in the text knowledge base.

[0050] Furthermore, in some implementations, the network parameters of each layer of the acoustic encoder are fixed, and audio information is input into the acoustic encoder to obtain the corresponding second acoustic feature vector. Specifically, audio information with the user's timbre is input into a pre-trained acoustic encoder, the network parameters of the acoustic encoder are frozen, and a second acoustic feature vector containing the user's timbre is obtained, forming an acoustic embedding library, i.e., an acoustic knowledge base. This design can alleviate the differences caused by different user timbres and improve the accuracy of knowledge retrieval when RAG technology performs knowledge retrieval based on the acoustic knowledge base composed of user timbre features.

[0051] In some implementations, the first acoustic feature vector and the second acoustic feature vector are obtained by encoding and embedding using the same acoustic encoder, and both the first and second acoustic feature vectors contain the timbre features of the same user. It is understood that, to avoid changes in the extracted acoustic feature vectors due to parameter changes in the acoustic encoder, the first and second acoustic feature vectors of this application are obtained by feature encoding using the same acoustic encoder. Furthermore, the first acoustic feature vector corresponding to a voice request from the same user has the same timbre features as the second acoustic feature vector in the pre-built acoustic knowledge base, thereby ensuring that a matching second acoustic feature vector can be quickly found during retrieval, improving the accuracy of retrieval results, avoiding interference caused by acoustic feature vectors with different timbres, and improving retrieval efficiency.

[0052] In this application, the first acoustic feature can be paired with each second acoustic feature vector in the acoustic knowledge base, and the cosine similarity of each pair can be calculated. The second acoustic feature vector in the pair with the highest similarity is then used as the target acoustic feature vector for the current matching. Optionally, various methods such as Euclidean distance, Manhattan distance, Mahalanobis distance, dynamic time warping, Gaussian mixture models, and deep learning methods can be used to find the most similar second acoustic feature vector. This is only an example and is not a limitation. It can be understood that after retrieving the matching second acoustic feature vector in the acoustic knowledge base, the corresponding text feature vector in the text knowledge base can be determined according to the mapping relationship.

[0053] For example, by calling the RAG module, external knowledge corresponding to Zhou Moumou and the song "Qinghua" in the text knowledge base can be retrieved from the acoustic knowledge base by acoustic feature similarity.

[0054] S240, the matched text feature vector and the first acoustic feature vector are concatenated and input into the language model for processing to obtain the corresponding speech recognition result; wherein, the speech recognition result includes the API result to be responded to and the parameters.

[0055] In this step, the corresponding text feature vector (i.e., text embedding) is concatenated with the first acoustic feature vector (i.e., acoustic embedding) in S220 above to form a concatenated vector, which is then injected into the language model.

[0056] It can be understood that, although when the language model finished pre-training, it had not learned knowledge related to *Qing Hua*, a new song by Zhou Moumou, at time T2, through knowledge retrieval performed based on the first acoustic feature vector in the acoustic knowledge base during the RAG stage, the new song *Qing Hua* by Zhou Moumou has been injected into the language model as external knowledge. Therefore, when the language model parses the user's voice request, it can accurately parse the song name as *Qing Hua*, instead of other words with the same pronunciation such as *Qing Hua* (clear flower), *Qing Hua* (light flower), *Qing Hua* (cyanide) and other words. At this time, the language model can smoothly output an accurate speech recognition result.

[0057] In some embodiments, the speech recognition result includes an API result to be responded to and parameters. In a smart cockpit, Application Programming Interface (API) results include, but are not limited to, text response, voice response, operation execution, status information, navigation instructions, multimedia control, safety warning, personalized service, interactive feedback and system status update. After determining the API result to be responded to, the specific parameters that need to be passed when calling the API are also obtained.

[0058] Correspondingly, the speech recognition result of this example is: API result: MusicSearchPlay, Argument result: {'name': 'Zhou Moumou', 'value': 'Qing Hua'}.

[0059] S250: input the API result and parameters from the speech recognition result into the function calling module of the cockpit to call the corresponding function or service.

[0060] It can be understood that, after parsing out the API result to be responded to and the Arguments parameters, the system inputs the information into the corresponding function call module, where the function call module is responsible for actually calling the backend service or executing specific operations to complete the user's voice request.

[0061] For example, at time T3, the aforementioned API result and Arguments parameters are input into the corresponding function call module to play the song "Qing Hua" by Zhou Moumou, completing the response to the instruction.

[0062] This example demonstrates that when the language model lacks emerging vocabulary for training data, it updates the second acoustic feature vector in the acoustic knowledge base. This serves as a medium, enabling the RAG module to quickly complete retrieval and matching within the comprehensively updated acoustic knowledge base. Subsequently, it finds the mapped text feature vector in the text knowledge base as external knowledge and injects it into the language model, achieving enhanced retrieval functionality. This design requires only simple updates to the acoustic knowledge base, ensuring consistency between locally stored data and the latest knowledge. This allows the language model, based on RAG technology, to efficiently and accurately output speech recognition results, accurately responding to user needs and significantly reducing the training cost of the language model. Furthermore, the text-to-speech synthesis module uses collected user speech to synthesize an acoustic knowledge base with specific timbres, facilitating more precise RAG knowledge retrieval and injection based on the user's timbre characteristics, ensuring the accuracy of the retrieval results.

[0063] Corresponding to the aforementioned application function implementation method embodiments, this application also provides a voice recognition device and corresponding embodiments.

[0064] Figure 4 This is a schematic diagram of the structure of the speech recognition device shown in this application.

[0065] See Figure 4 The speech recognition device 400 shown in this application includes a speech receiving module 410, a feature encoding module 420, a retrieval enhancement module 430, a result generation module 440, and a storage module 450. Wherein: The voice receiving module 410 is used to receive voice requests from users inside the cockpit.

[0066] The feature encoding module 420 is used to encode the voice request to be recognized to obtain the first acoustic feature vector.

[0067] The retrieval enhancement module 430 is used to retrieve a matching second acoustic feature vector from a pre-built acoustic knowledge base based on the first acoustic feature vector.

[0068] The result generation module 440 is used to concatenate the text feature vector mapped by the matched second acoustic feature vector and the first acoustic feature vector, and then input the concatenation into the language model for processing to obtain the corresponding speech recognition result for cockpit response.

[0069] The storage module 450 is used to store the acoustic knowledge base, which is obtained by speech synthesis based on the text information in the preset text knowledge base. Each second acoustic feature vector in the acoustic knowledge base has a mapping relationship with the text feature vector corresponding to the text information.

[0070] See Figure 5In some embodiments, the apparatus of this application further includes a text-to-speech module 460, used to synthesize corresponding audio information from text information in a text knowledge base based on the timbre features in the collected user's reference speech; the feature encoding module 420 is further used to extract the corresponding second acoustic feature vector based on the audio information to generate an acoustic knowledge base corresponding to the user, wherein the second acoustic feature vector includes the user's timbre features. The feature encoding module 420 is the acoustic encoder of the language model, and the first acoustic feature vector and the second acoustic feature vector are obtained by encoding and embedding through the same acoustic encoder. Further, the text-to-speech module 460 is also used to update the second acoustic feature vector in the acoustic knowledge base accordingly when the text information in the text knowledge base is updated, and to save the mapping relationship with the updated text feature vector. Optionally, the storage module 450 is also used to store the text knowledge base.

[0071] In some implementations, the retrieval enhancement module 430 is used to perform a search in the acoustic knowledge base according to preset matching rules to obtain a second acoustic feature vector that has feature similarity to the first acoustic feature vector. The retrieval enhancement module is the RAG module.

[0072] In some embodiments, the result generation module 440 of this application includes a feature concatenation module 441 and a language generation module 442. The feature concatenation module 441 is used to concatenate the text feature vector mapped by the matched second acoustic feature vector and the first acoustic feature vector to generate concatenated features; the language generation module 442 is used to generate the API result and parameters to be responded to based on the concatenated features.

[0073] The speech recognition device provided in this application searches an acoustic knowledge base using a first acoustic feature vector containing the user's voice timbre, finds a matching second acoustic feature vector, and then concatenates the mapped text feature vector with the first acoustic feature vector before injecting it into a language model to achieve enhanced retrieval functionality. This design requires only simple updates to the acoustic knowledge base, ensuring consistency between locally stored data and the latest knowledge, to efficiently and accurately output speech recognition results, enabling the system to accurately respond to user needs.

[0074] Figure 6 This is a structural schematic diagram of the cockpit shown in this application.

[0075] like Figure 6 As shown, the cockpit 600 provided in this application may include the above-mentioned features. Figure 4 or Figure 5 The voice recognition device 400 shown.

[0076] Regarding the apparatus in the above embodiments, the specific manner in which each module performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated further here.

[0077] Figure 7 This is a schematic diagram of the structure of the electronic device shown in this application.

[0078] See Figure 7 The vehicle 1000 includes a memory 1010 and a processor 1020.

[0079] The processor 1020 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor.

[0080] Memory 1010 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM may store static data or instructions required by processor 1020 or other modules of the computer. Permanent storage devices may be read-write storage devices. Permanent storage devices may be non-volatile storage devices that retain stored instructions and data even when the computer is powered off. In some embodiments, permanent storage devices use mass storage devices (e.g., magnetic or optical disks, flash memory) as permanent storage devices. In other embodiments, permanent storage devices may be removable storage devices (e.g., floppy disks, optical drives). System memory may be a read-write storage device or a volatile read-write storage device, such as dynamic random access memory. System memory may store some or all of the instructions and data required by the processor during operation. Furthermore, memory 1010 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (e.g., DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and disks and / or optical disks may also be used. In some embodiments, the memory 1010 may include a removable storage device that is readable and / or writable, such as a laser disc (CD), a read-only digital multifunction optical disc (e.g., DVD-ROM, dual-layer DVD-ROM), a read-only Blu-ray disc, an ultra-high density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not contain carrier waves or transient electronic signals transmitted wirelessly or via wired connections.

[0081] The memory 1010 stores executable code, which, when processed by the processor 1020, can cause the processor 1020 to execute part or all of the methods described above.

[0082] Furthermore, the method according to this application can also be implemented as a computer program or computer program product, which includes computer program code instructions for performing some or all of the steps in the method described above.

[0083] Alternatively, this application may be implemented as a computer-readable storage medium (or a non-transitory machine-readable storage medium or a machine-readable storage medium) storing executable code (or computer program or computer instruction code) thereon, which, when executed by a processor of an electronic device (or server, etc.), causes the processor to perform part or all of the steps of the methods described above according to this application.

[0084] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, and are not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A speech recognition method, characterized in that, include: Receive voice requests from users inside the cockpit; The voice request to be identified is feature-encoded to obtain a first acoustic feature vector; Based on the first acoustic feature vector, a matching second acoustic feature vector is retrieved from a pre-constructed acoustic knowledge base; wherein, the acoustic knowledge base is obtained by speech synthesis based on text information in a preset text knowledge base, and each second acoustic feature vector in the acoustic knowledge base has a mapping relationship with the text feature vector corresponding to the text information; The text feature vector mapped by the matched second acoustic feature vector is concatenated with the first acoustic feature vector and then input into a language model for processing to obtain the corresponding speech recognition result for the cockpit response.

2. The method according to claim 1, characterized in that, The acoustic knowledge base is obtained by speech synthesis based on text information in a preset text knowledge base, including: Collect the user's reference voice; Based on the timbre features in the reference speech, the text information in the text knowledge base is synthesized into corresponding audio information; Based on the audio information, a second acoustic feature vector is extracted to generate the acoustic knowledge base corresponding to the user, wherein the second acoustic feature vector includes the user's timbre features.

3. The method according to claim 2, characterized in that, The language model includes a pre-trained acoustic encoder; The step of extracting the corresponding second acoustic feature vector based on the audio information and generating the corresponding acoustic knowledge base includes: The network parameters of each layer of the acoustic encoder are fixed, and the audio information is input into the acoustic encoder to obtain the corresponding second acoustic feature vector.

4. The method according to claim 1, characterized in that: The first acoustic feature vector and the second acoustic feature vector are obtained by encoding and embedding through the same acoustic encoder, and both the first acoustic feature vector and the second acoustic feature vector contain the timbre features of the same user.

5. The method according to claim 1, characterized in that, The step of retrieving a matching second acoustic feature vector from a pre-built acoustic knowledge base includes: The acoustic knowledge base is searched according to preset matching rules to obtain a second acoustic feature vector that has feature similarity to the first acoustic feature vector.

6. The method according to claim 1, characterized in that: The speech recognition results include the API results and parameters to be responded to.

7. The method according to any one of claims 1 to 6, characterized in that, The method further includes: When the text information in the text knowledge base is updated, the second acoustic feature vector in the acoustic knowledge base is updated accordingly, and the mapping relationship with the updated text feature vector is saved.

8. A voice recognition device, characterized in that, include: The voice receiving module is used to receive voice requests from users inside the cockpit. The feature encoding module is used to encode the voice request to be recognized to obtain a first acoustic feature vector; The retrieval enhancement module is used to retrieve a matching second acoustic feature vector from a pre-built acoustic knowledge base based on the first acoustic feature vector. The result generation module is used to concatenate the text feature vector mapped by the matched second acoustic feature vector and the first acoustic feature vector, and then input the concatenation into the language model for processing to obtain the corresponding speech recognition result for the cockpit response. The storage module is used to store the acoustic knowledge base, which is obtained by speech synthesis based on text information in a preset text knowledge base. Each second acoustic feature vector in the acoustic knowledge base has a mapping relationship with the text feature vector corresponding to the text information.

9. A cockpit, characterized in that, Includes the speech recognition device as described in claim 8.

10. A vehicle, characterized in that, include: processor; as well as A memory having executable code stored thereon, which, when executed by the processor, causes the processor to perform the method as described in any one of claims 1 to 7.

11. A computer-readable storage medium having executable code stored thereon, which, when executed by a processor of an electronic device, causes the processor to perform the speech recognition method as described in any one of claims 1 to 7.

12. A computer program product, characterized in that, The computer program product includes computer instructions that, when executed by a processor, implement the speech recognition method according to any one of claims 1 to 7.

Images