Speech recognition method and device, storage medium and equipment

Abstract

The invention discloses a voice recognition method and device, a storage medium and equipment, and the method comprises the steps: firstly obtaining the driving data of a target vehicle, and extracting a target feature vector from the driving data; wherein the driving data comprises state information of the target vehicle and environment data information of the target vehicle, then obtaining a target voice sent by a vehicle-mounted user on the target vehicle, and carrying out primary recognition on the target voice to obtain N candidate text recognition results; inputting the target feature vector and the N candidate text recognition results into a semantic confidence prediction model, predicting to obtain text confidence prediction results corresponding to the N candidate text recognition results, and performing secondary recognition on the target speech by using the text confidence prediction results corresponding to the N candidate text recognition results to obtain a target speech recognition result; therefore, the final recognition result corresponding to the target voice is screened out from the N candidate text recognition results, the recognition accuracy is improved, and the riding experience and the driving safety of the vehicle-mounted user are enhanced.

Description

Technical Field

[0001] The present application relates to the field of vehicle technology, and in particular to a speech recognition method, apparatus, storage medium and device. Background Art

[0002] With the improvement of people's living standards and the rapid development of social economy, the utilization rate of cars has gradually increased, and people's demand for intelligent car riding has also increased. Among them, the in-vehicle voice recognition system allows the driver to control the vehicle to perform various functions without leaving the steering wheel, such as setting navigation, playing music or answering calls, etc., only by issuing voice commands. Although it can improve the driver's driving safety, sometimes the system may misidentify the driver's instructions, especially those commands with similar pronunciations are more likely to be confused, affecting the user's driving experience and may even affect driving safety in an emergency.

[0003] Therefore, it is necessary to more accurately recognize the voice of in-vehicle users in order to improve the riding experience and driving safety. However, the existing methods for improving the accuracy of speech recognition results usually rely mainly on preset industry error correction vocabulary or the acoustic and semantic features of characters. Although these methods have improved the accuracy of speech recognition result correction to a certain extent, they do not take into account the changes in the dynamic environment and real-time status of the vehicle. In vehicle usage scenarios, due to the uncertainty of factors such as road conditions, traffic conditions, and weather changes, fixed recognition result correction strategies may not be flexible enough and cannot adapt to the rapidly changing driving environment, reducing the accuracy of speech recognition results, resulting in poor driving and riding experience for users and affecting the driving safety of the vehicle. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to provide a speech recognition method, device, storage medium and equipment, which can effectively improve the accuracy of the speech recognition results of vehicle-mounted users, thereby enhancing the user's driving and riding experience and improving the vehicle's driving safety.

[0005] The present application provides a speech recognition method, including:

[0006] Acquire the driving data of the target vehicle and extract the target feature vector corresponding to the driving data; the driving data includes the self state information of the target vehicle and the environmental data information of the target vehicle;

[0007] Acquire a target speech to be recognized issued by a user on the target vehicle, and perform initial recognition on the target speech to obtain N candidate text recognition results; wherein N is a positive integer greater than 1;

[0008] Inputting the target feature vector and N candidate text recognition results into a pre-built semantic confidence prediction model to predict text confidence prediction results corresponding to each of the N candidate text recognition results;

[0009] The target speech is recognized a second time using the text confidence prediction results corresponding to each of the N candidate text recognition results to obtain a final recognition result corresponding to the target speech.

[0010] In a possible implementation, extracting the target feature vector corresponding to the driving data includes:

[0011] Encoding the self-state information of the target vehicle to obtain a running state feature vector of the target vehicle;

[0012] Encoding the environmental data information of the target vehicle to obtain an environmental information feature vector;

[0013] The running state feature vector and the environmental information feature vector of the target vehicle are fused to obtain a fused feature vector as the target feature vector corresponding to the driving data.

[0014] In a possible implementation, the operation state feature vector of the target vehicle and the environment information feature vector are fused to obtain a fused feature vector as the target feature vector corresponding to the driving data, including:

[0015] The running state feature vector and the environmental information feature vector of the target vehicle are concatenated or weighted summed to obtain a fused feature vector as the target feature vector corresponding to the driving data.

[0016] In a possible implementation, the performing of the initial recognition on the target speech to obtain N candidate text recognition results includes:

[0017] The acoustic features of the target speech are extracted; and according to the acoustic features, the target speech is decoded to obtain N candidate text recognition results.

[0018] In a possible implementation, the semantic confidence prediction model is constructed as follows:

[0019] Acquire driving data of a sample vehicle; and extract a sample feature vector corresponding to the driving data of the sample vehicle; the driving data of the sample vehicle includes the self state information of the sample vehicle and the environmental data information of the sample vehicle;

[0020] Acquire a sample speech uttered by a user on the sample vehicle, and determine a true semantic confidence of a text recognition result corresponding to the sample speech;

[0021] The initial semantic confidence prediction model is trained according to the sample feature vector, the text recognition result corresponding to the sample speech and its corresponding true semantic confidence to generate the semantic confidence prediction model.

[0022] In a possible implementation, the method further includes:

[0023] Acquire the driving data of the verification vehicle; and extract the verification feature vector corresponding to the driving data of the verification vehicle; the driving data of the verification vehicle includes the self state information of the verification vehicle and the environmental data information of the verification vehicle;

[0024] Acquire a verification voice uttered by a user on the verification vehicle, and determine a true semantic confidence level of a text recognition result corresponding to the verification voice;

[0025] Inputting the verification feature vector and the text recognition result corresponding to the verification speech into the semantic confidence prediction model to obtain the prediction confidence of the text recognition result corresponding to the verification speech;

[0026] When the predicted confidence of the text recognition result corresponding to the verification speech is inconsistent with the true semantic confidence, the driving data of the verification vehicle is re-used as the driving data of the sample vehicle, and the verification speech is re-used as the sample speech to update the semantic confidence prediction model.

[0027] In a possible implementation, the method of performing secondary recognition on the target speech by using the text confidence prediction results corresponding to each of the N candidate text recognition results to obtain a final recognition result corresponding to the target speech includes:

[0028] Performing a weighted sum calculation on the probability score of the initial recognition corresponding to each candidate text recognition result in the N candidate text recognition results and the text confidence prediction result to obtain a recognition score corresponding to each candidate text recognition result;

[0029] The candidate text recognition result corresponding to the highest recognition score is selected from the N candidate text recognition results as the final recognition result corresponding to the target speech.

[0030] The present application also provides a speech recognition device, the device comprising:

[0031] A first acquisition unit is used to acquire the driving data of the target vehicle and extract the target feature vector corresponding to the driving data; the driving data includes the self state information of the target vehicle and the environmental data information of the target vehicle;

[0032] A second acquisition unit is used to acquire a target speech to be recognized issued by a user on the target vehicle, and perform initial recognition on the target speech to obtain N candidate text recognition results; wherein N is a positive integer greater than 1;

[0033] A prediction unit, used for inputting the target feature vector and N candidate text recognition results into a pre-built semantic confidence prediction model, and predicting text confidence prediction results corresponding to each of the N candidate text recognition results;

[0034] The recognition unit is used to perform secondary recognition on the target speech by using the text confidence prediction results corresponding to each of the N candidate text recognition results to obtain a final recognition result corresponding to the target speech.

[0035] In a possible implementation, the first acquiring unit includes:

[0036] A first encoding subunit is used to encode the self state information of the target vehicle to obtain a running state feature vector of the target vehicle;

[0037] A second encoding subunit is used to encode the environmental data information of the target vehicle to obtain an environmental information feature vector;

[0038] The fusion subunit is used to fuse the running state feature vector and the environmental information feature vector of the target vehicle to obtain a fused feature vector as the target feature vector corresponding to the driving data.

[0039] In a possible implementation, the fusion subunit is specifically used for:

[0040] The running state feature vector and the environmental information feature vector of the target vehicle are concatenated or weighted summed to obtain a fused feature vector as the target feature vector corresponding to the driving data.

[0041] In a possible implementation manner, the second obtaining unit is specifically configured to:

[0042] The acoustic features of the target speech are extracted; and according to the acoustic features, the target speech is decoded to obtain N candidate text recognition results.

[0043] In a possible implementation manner, the device further includes:

[0044] A third acquisition unit is used to acquire driving data of a sample vehicle; and extract a sample feature vector corresponding to the driving data of the sample vehicle; the driving data of the sample vehicle includes the self state information of the sample vehicle and the environmental data information of the sample vehicle;

[0045] A fourth acquisition unit, configured to acquire a sample speech uttered by a user on the sample vehicle, and determine a true semantic confidence of a text recognition result corresponding to the sample speech;

[0046] The training unit is used to train the initial semantic confidence prediction model according to the sample feature vector, the text recognition result corresponding to the sample speech and its corresponding true semantic confidence, so as to generate the semantic confidence prediction model.

[0047] In a possible implementation manner, the device further includes:

[0048] A fifth acquisition unit is used to acquire the driving data of the verification vehicle; and extract the verification feature vector corresponding to the driving data of the verification vehicle; the driving data of the verification vehicle includes the self state information of the verification vehicle and the environmental data information of the verification vehicle;

[0049] a sixth acquisition unit, configured to acquire a verification voice issued by a user on the verification vehicle, and determine a true semantic confidence level of a text recognition result corresponding to the verification voice;

[0050] An obtaining unit, used for inputting the verification feature vector and the text recognition result corresponding to the verification speech into the semantic confidence prediction model to obtain the prediction confidence of the text recognition result corresponding to the verification speech;

[0051] An updating unit is used to update the semantic confidence prediction model by reusing the driving data of the verification vehicle as the driving data of the sample vehicle and reusing the verification speech as the sample speech when the predicted confidence of the text recognition result corresponding to the verification speech is inconsistent with the true semantic confidence.

[0052] In a possible implementation, the identification unit includes:

[0053] A calculation subunit, used for performing weighted sum calculation on the probability score of the initial recognition corresponding to each candidate text recognition result in the N candidate text recognition results and the text confidence prediction result, to obtain a recognition score corresponding to each candidate text recognition result;

[0054] The selection subunit is used to select the candidate text recognition result corresponding to the highest recognition score from the N candidate text recognition results as the final recognition result corresponding to the target speech.

[0055] The embodiment of the present application also provides a speech recognition device, including: a processor, a memory, and a system bus;

[0056] The processor and the memory are connected via the system bus;

[0057] The memory is used to store one or more programs, and the one or more programs include instructions. When the instructions are executed by the processor, the processor executes any one of the implementations of the above-mentioned speech recognition method.

[0058] An embodiment of the present application also provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes any one of the implementations of the above-mentioned speech recognition method.

[0059] The embodiment of the present application also provides a computer program product. When the computer program product is run on a terminal device, the terminal device executes any one of the implementations of the above-mentioned speech recognition method.

[0060] A speech recognition method, apparatus, storage medium and device provided in an embodiment of the present application first obtain driving data of a target vehicle and extract a target feature vector corresponding to the driving data; wherein the driving data includes the target vehicle's own state information and the target vehicle's environmental data information, then obtain a target speech to be recognized issued by a user on the target vehicle, and perform an initial recognition on the target speech to obtain N candidate text recognition results; wherein N is a positive integer greater than 1; then, the target feature vector and the N candidate text recognition results are input into a pre-constructed semantic confidence prediction model, and the text confidence prediction results corresponding to each of the N candidate text recognition results are predicted, and then the target speech is secondary recognized using the text confidence prediction results corresponding to each of the N candidate text recognition results to obtain a final recognition result corresponding to the target speech.

[0061] It can be seen that, since the present application introduces driving data such as the target vehicle's own state information and the target vehicle's environmental data information as important recognition basis when recognizing the target speech emitted by the on-board user of the target vehicle, and uses the pre-constructed semantic confidence prediction model to accurately determine the confidence corresponding to the N candidate text recognition results obtained in the initial recognition as the screening basis, it is possible to screen out more accurate candidate text recognition results from the N candidate text recognition results as the final recognition result corresponding to the target speech, effectively improving the recognition accuracy of the target speech, thereby enhancing the driving and riding experience of the on-board user, and improving the driving safety of the target vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0063] Figure 1 A flowchart of a speech recognition method provided in an embodiment of the present application;

[0064] Figure 2 An example diagram of the process of encoding the target vehicle's own state information to obtain the target vehicle's operating state feature vector provided in the embodiment of the present application;

[0065] Figure 3 An example diagram of a process for inputting a target feature vector and three candidate text recognition results into a semantic confidence prediction model to predict text confidence prediction results corresponding to the three candidate text recognition results, provided in an embodiment of the present application;

[0066] Figure 4 An example diagram of the training process of the semantic confidence prediction model provided in an embodiment of the present application;

[0067] Figure 5 A schematic diagram of the composition of a speech recognition device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0068] With the popularization of automobiles, people have higher and higher requirements for vehicle intelligence and safety. Among them, the in-vehicle voice recognition system, as a core component for realizing automobile intelligence, is gradually being widely used in a variety of models. The introduction of the in-vehicle voice recognition system greatly facilitates the driver's operating experience. It enables the driver to control the vehicle to perform various functions, such as setting navigation, playing music or answering calls, without leaving the steering wheel, just by issuing voice commands. Such a design not only improves driving safety, but also makes the entire driving process easier and more enjoyable.

[0069] However, although existing in-vehicle voice recognition systems can bring some benefits, there are still some problems in actual use. The most notable problem is the occurrence of misrecognition. Due to the complexity of speech itself and the influence of the surrounding environment, these systems sometimes misrecognize the driver's instructions, especially those commands with similar pronunciations are more easily confused. For example, when driving at night, if the driver wants to "turn on the lights" through voice commands, the in-vehicle voice recognition system may mistakenly recognize it as "open the windows". This misjudgment will not only affect the driver's driving experience, but may even affect driving safety in an emergency.

[0070] Therefore, it is necessary to improve the recognition effect of the existing in-vehicle voice recognition system to effectively improve the recognition accuracy and reduce the false recognition rate. Such an in-vehicle voice recognition system can better adapt to the ever-changing driving needs and conditions, meet the needs of the growing automotive intelligent market, and ensure the driver's safe driving experience.

[0071] However, the existing methods for improving the accuracy of speech recognition results usually rely mainly on preset industry error correction vocabulary or the acoustic and semantic features of characters. Although these methods have improved the accuracy of speech recognition error correction to a certain extent, they ignore the importance of vehicle status information and driving environment to the understanding of voice commands. For example, when the vehicle is driving at high speed, the driver may issue emergency safety-related instructions, and such instructions need to be processed and responded to first. However, existing methods for improving speech recognition may not be able to effectively distinguish these emergency instructions from other non-emergency instructions, resulting in delayed or incorrect responses.

[0072] It can be seen that in vehicle usage scenarios, due to the uncertainty of factors such as road conditions, traffic conditions, weather changes, etc., fixed recognition result correction strategies may not be flexible enough to adapt to the rapidly changing driving environment, reducing the accuracy of speech recognition results, resulting in poor user driving and riding experience and affecting the vehicle's driving safety.

[0073] To address the above-mentioned defects, the present application provides a speech recognition method, which first obtains the driving data of a target vehicle and extracts a target feature vector corresponding to the driving data; wherein the driving data includes the target vehicle's own state information and the environmental data information of the target vehicle, and then obtains the target speech to be recognized issued by the on-board user of the target vehicle, and performs an initial recognition on the target speech to obtain N candidate text recognition results; wherein N is a positive integer greater than 1; then, the target feature vector and the N candidate text recognition results are input into a pre-constructed semantic confidence prediction model, and the text confidence prediction results corresponding to each of the N candidate text recognition results are predicted, and then the target speech is secondary recognized using the text confidence prediction results corresponding to each of the N candidate text recognition results to obtain a final recognition result corresponding to the target speech.

[0074] It can be seen that, since the present application introduces driving data such as the target vehicle's own state information and the target vehicle's environmental data information as important recognition basis when recognizing the target speech emitted by the on-board user of the target vehicle, and uses the pre-constructed semantic confidence prediction model to accurately determine the confidence corresponding to the N candidate text recognition results obtained in the initial recognition as the screening basis, it is possible to screen out more accurate candidate text recognition results from the N candidate text recognition results as the final recognition result corresponding to the target speech, effectively improving the recognition accuracy of the target speech, thereby enhancing the driving and riding experience of the on-board user, and improving the driving safety of the target vehicle.

[0075] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0076] First embodiment

[0077] See also Figure 1 , is a flow chart of a speech recognition method provided in this embodiment, the method comprising the following steps:

[0078] S101: Acquire driving data of a target vehicle and extract a target feature vector corresponding to the driving data; wherein the driving data includes the target vehicle's own state information and the target vehicle's environment data information.

[0079] In this embodiment, any vehicle for which the vehicle-mounted user voice recognition is performed by this embodiment is defined as a target vehicle, and any voice that needs to be recognized by the vehicle-mounted user on the target vehicle is defined as the target voice to be recognized. It should be noted that this embodiment does not limit the language type of the target voice, for example, the target voice can be a voice composed of Chinese or English, etc.; at the same time, this embodiment does not limit the length of the target voice, for example, the target voice can be a sentence or multiple sentences, etc. It can be understood that the target voice can be obtained by recording according to actual needs, for example, the vehicle-mounted microphone pre-installed on the target vehicle can be used to collect the voice emitted by the vehicle-mounted user as the target voice, etc. It should also be noted that in order to improve the recognition accuracy of the target voice, a noise suppression function can be pre-set on the target vehicle to reduce the interference of the noise in the car on the target voice signal, or an advanced signal processing algorithm can be used to accurately obtain the voice signal emitted by the vehicle-mounted user in a noisy environment.

[0080] Furthermore, in order to improve the recognition accuracy of the target speech, while acquiring the target speech, it is also necessary to use various sensors pre-installed on the target vehicle to acquire the driving data of the target vehicle during driving, and use existing or future feature extraction methods to extract the target feature vector corresponding to the driving data to execute the subsequent step S103.

[0081] The target vehicle's driving data may include, but is not limited to, the target vehicle's own state information and the target vehicle's environmental data information. The target vehicle's own state information may include, but is not limited to, driving speed, acceleration, engine state, gear position, headlight state, driving position, etc. The target vehicle's environmental data information may include, but is not limited to, road conditions, traffic signs, whether there are pedestrians around, and the location of other vehicles.

[0082] Furthermore, it should be noted that the present application does not limit the method for obtaining the driving data of the target vehicle, which can usually be obtained by using various sensors pre-installed on the target vehicle and other vehicle-mounted systems. Furthermore, the present application does not limit the specific structure, quantity, and installation location of each sensor on the target vehicle. For example, the target vehicle can be pre-installed with sensors such as a vehicle speed sensor (VSS), an acceleration sensor, etc. to obtain various driving data such as the driving speed and acceleration of the target vehicle. It can also be pre-installed with on-board cameras, radars, and other sensors to perceive the surrounding environment in real time to obtain the environmental information of the target vehicle. Moreover, the number of each sensor can be one or more.

[0083] Specifically, an optional implementation method is that the specific implementation process of "obtaining the driving data of the target vehicle and extracting the target feature vector corresponding to the driving data" in step S101 may include the following steps S1011-S1013:

[0084] S1011: Encode the target vehicle's own state information to obtain a running state feature vector of the target vehicle.

[0085] In this implementation, the target vehicle's own state information (such as driving speed, acceleration, engine state, gear position, light state, driving position, etc.) included in the driving data can be encoded using existing or future encoding methods to obtain the running state feature vector corresponding to the target vehicle, which is used to execute the subsequent step S1013. Figure 2As shown, the target vehicle's own state information can be encoded using one-hot encoding to obtain encoding vectors such as "010000" for acceleration, "100000" for driving speed, "00010" for headlight state, and "001000" for engine state. These one-hot encoding vectors are then input into the neural network embedding layer to generate a 256-dimensional (the specific dimension is not limited, but 256 dimensions are used as an example) feature vector as the target vehicle's operating state feature vector, which is defined as carEmbH to represent the target vehicle's current operating state.

[0086] S1012: Encode the environmental data information of the target vehicle to obtain an environmental information feature vector.

[0087] In this implementation, the environmental data information of the target vehicle included in the driving data (such as road conditions, traffic signs, whether there are pedestrians around, the location of other vehicles, etc.) can also be encoded using existing or future encoding methods to obtain an environmental information feature vector, and defined as envEmbH to perform subsequent step S1013. For example, the environmental data information of the target vehicle can still be encoded using a one-hot encoding method to obtain a one-hot encoding vector, which is then input into the neural network Embedding layer to generate a 256-dimensional (the specific dimension is not limited, and 256 dimensions are used as an example) feature vector as an environmental information feature vector, which is used to represent the environmental information characteristics of the current surroundings of the target vehicle.

[0088] S1013: Fusing the running state feature vector of the target vehicle and the environmental information feature vector to obtain a fused feature vector as the target feature vector corresponding to the driving data.

[0089] In this implementation, after obtaining the target vehicle's operating status feature vector carEmbH through step S1011 and the environmental information feature vector envEmbH through step S1012, the target vehicle's operating status feature vector carEmbH and the environmental information feature vector envEmbH can be further fused using existing or future feature fusion methods to fully integrate the target vehicle's own status information and the surrounding environment information, and obtain a fused feature vector that more comprehensively and accurately reflects the actual situation of the target vehicle as the target feature vector corresponding to the driving data, for executing the subsequent step S103.

[0090] Specifically, the target vehicle's operating state feature vector carEmbH and the environmental information feature vector envEmbH can be spliced ​​in a preset order (the specific splicing order is not limited, such as the two can be directly spliced, etc.) to obtain a longer fused feature vector as the target feature vector corresponding to the driving data. This fusion method can directly combine the two types of information together, retaining the integrity of the original information. For example, the target vehicle's operating state feature vector carEmbH can be placed in front, and then the environmental information feature vector envEmbH can be connected, so that the semantic confidence prediction model mentioned in the subsequent step S103 can process these two parts of information in turn, and fully explore the associations and features therein.

[0091] Alternatively, the target vehicle's operating state feature vector carEmbH and environmental information feature vector envEmbH can be weighted and summed to obtain a fused feature vector as the target feature vector corresponding to the driving data, that is, different weights can be assigned to the target vehicle's operating state feature vector carEmbH and environmental information feature vector envEmbH, and then the summation operation can be performed. The weights can be adjusted according to the actual application scenario and the importance of the two types of information. For example, if the vehicle's surrounding environment information has a greater impact on the vehicle's decision-making under specific traffic conditions, a higher weight can be assigned to the environmental information feature vector envEmbH. This method can more flexibly adjust the contribution of the two types of information in the fusion result to adapt to different situations.

[0092] It should be noted that the present application does not limit the feature fusion method used for the target vehicle's operating status feature vector carEmbH and the environmental information feature vector envEmbH. According to different application scenarios and requirements, the most appropriate fusion method can be flexibly selected to process the fusion of the two feature vectors to obtain a more accurate fused feature vector.

[0093] S102: Acquire a target speech to be recognized uttered by a user on board a target vehicle, and perform initial recognition on the target speech to obtain N candidate text recognition results; wherein N is a positive integer greater than 1.

[0094] In this embodiment, after the target voice emitted by the in-vehicle user is collected by using a vehicle microphone pre-installed on the target vehicle or other collection methods, in order to improve the recognition accuracy of the target voice, the target voice is further recognized for the first time through preprocessing, feature extraction, processing using acoustic models and language models, and decoding, and N candidate text recognition results are obtained to achieve voice-to-text conversion for executing the subsequent step S103.

[0095] Specifically, an optional implementation method is that after acquiring the target speech, the acoustic features of the target speech can be extracted using existing or future feature extraction methods, and the target speech can be decoded based on the acoustic features to obtain N candidate text recognition results.

[0096] In this implementation, when extracting the acoustic features of the target audio data, the target voice data needs to be framed first to obtain the corresponding voice frame sequence, and then the framed voice frame sequence is pre-emphasized; and then the acoustic features of each voice frame are extracted in turn, wherein the acoustic features refer to feature data used to characterize the acoustic information of the corresponding voice frame, for example, Mel-scale Frequency Cepstral Coefficients (MFCC) features or Perceptual Linear Predictive (PLP) features, etc. Then, the probability of the voice unit is calculated through an acoustic model such as a deep neural network, combined with the language model optimization results, and finally N candidate text recognition results can be decoded.

[0097] It should be noted that the embodiments of the present application do not limit the method for extracting the acoustic features of the target audio data, nor do they limit the specific extraction process. An appropriate extraction method may be selected according to actual conditions, and corresponding feature extraction operations may be performed.

[0098] S103: Input the target feature vector and the N candidate text recognition results into a pre-built semantic confidence prediction model to predict text confidence prediction results corresponding to each of the N candidate text recognition results.

[0099] In this embodiment, after extracting the target feature vector corresponding to the driving data through step S101 and performing the initial recognition of the target speech through step S102 to obtain N candidate text recognition results, in order to improve the recognition effect of the target speech, the obtained target feature vector and N candidate text recognition results can be further input into the pre-built semantic confidence prediction model, which will make full use of its complex neural network architecture and advanced algorithms to conduct in-depth analysis and comprehensive consideration of these rich input information. In this way, the text confidence prediction results corresponding to each of the N candidate text recognition results can be accurately predicted and defined as SegPrab to execute the subsequent step S104, providing a key basis for the subsequent optimization and decision-making of the speech recognition results.

[0100] For example: Figure 3As shown, assuming that the target feature vector corresponding to the driving data extracted through step S101 is "high-speed driving, no pedestrians around, and slightly dim light", and the target speech is initially recognized through step S102, and three candidate text recognition results are obtained: "open the window", "turn on the lights", and "turn on the headlights". After the target feature vector and the three candidate text recognition results are input into the semantic confidence prediction model, the text confidence prediction results corresponding to each of the three candidate text recognition results can be predicted as follows: the prediction confidence corresponding to "open the window" is 0.65, the prediction confidence corresponding to "turn on the lights" is 0.95, and the prediction confidence corresponding to "turn on the headlights" is 0.85.

[0101] It should be noted that the specific structure and specific prediction process of the semantic confidence prediction model can be set according to actual conditions, and this embodiment does not limit this.

[0102] Next, this embodiment will introduce the construction process of the semantic confidence prediction model used in this application, which may specifically include the following steps (1)-(3):

[0103] Step (1): Acquire driving data of a sample vehicle; and extract a sample feature vector corresponding to the driving data of the sample vehicle; wherein the driving data of the sample vehicle includes the sample vehicle's own state information and the sample vehicle's environmental data information.

[0104] In this embodiment, in order to construct a semantic confidence prediction model, a lot of preparatory work needs to be done in advance. First, a large number of voice commands issued by drivers in sample vehicles under various driving scenarios need to be collected as sample voices, and the corresponding driving data of the sample vehicles when the drivers issue voice commands are obtained, including but not limited to the sample vehicle's own state information (such as vehicle speed, gear position, vehicle fault warning, etc.) and the environmental data information of the sample vehicle (such as road conditions, weather, temperature inside the vehicle, etc.), and the driving data of the sample vehicle is encoded using existing or future encoding methods to obtain the sample feature vector corresponding to the driving data of the sample vehicle, which is used to form training data for executing the subsequent step (3). Among them, the specific encoding process can refer to the implementation process of extracting the target feature vector in step S101, and it is only necessary to replace the driving data of the target vehicle with the driving data of the sample vehicle, which will not be repeated here.

[0105] Step (2): Obtain a sample speech uttered by a user on board a sample vehicle, and determine the true semantic confidence of the text recognition result corresponding to the sample speech.

[0106] In this embodiment, after obtaining the sample speech emitted by the on-board user (such as the driver) in the sample vehicle, each sample speech can be further recognized for the first time through preprocessing, feature extraction, processing using an acoustic model and a language model, and decoding, and the text recognition results corresponding to each sample speech can be obtained to achieve speech-to-text conversion to form training data, and the accurate semantic confidence (i.e., true semantic confidence) of the text recognition results corresponding to each sample speech can be manually annotated to perform the subsequent step (3). Among them, the specific process of the initial recognition of the sample speech can refer to the implementation process of the initial recognition of the target speech in step S102, and it is only necessary to replace the target speech with the sample speech, which will not be repeated here.

[0107] Step (3): Based on the sample feature vector, the text recognition result corresponding to the sample speech and its corresponding true semantic confidence, the initial semantic confidence prediction model is trained to generate a semantic confidence prediction model.

[0108] In this embodiment, after obtaining the sample feature vector, the text recognition result corresponding to the sample speech and its corresponding true semantic confidence, further, when performing model training, a sample feature vector corresponding to the driving data of a sample vehicle can be extracted from the training data in turn, and the first position of the text recognition result of the corresponding sample speech (constituting global information) is inserted as the model input, and the corresponding semantic confidence prediction result is used as the output. Multiple rounds of model training are performed, and the prediction results obtained in each round of training are compared with the corresponding true semantic confidence labels, and the model parameters of the initial semantic confidence prediction model are updated according to the difference between the two until the preset conditions are met, such as the value of the loss constraint function is very small and basically unchanged, then the update of the model parameters is stopped, the training of the semantic confidence prediction model is completed, and a trained semantic confidence prediction model is generated.

[0109] Among them, the present application does not limit the specific composition structure of the initial semantic confidence prediction model, and different model structures can be selected according to the actual deployment terminal performance. For example, the BERT model, Transformer model or Recurrent Neural Network (RNN) model can be used as the initial semantic confidence prediction model.

[0110] For example: Figure 4 As shown, the sample feature vector corresponding to the driving data of the sample vehicle (such as Figure 4 The car_env_fuse_EmbH shown in FIG. 1 is inserted into the text recognition result of the corresponding sample speech (such as Figure 4 As shown in the figure, the first one is “turn on the lights”, and the global information is input into the initial semantic confidence prediction model (such as Figure 4 BERT, Transformer, RNN shown in the figure), and obtain the text recognition result output by the model (such as Figure 4 The prediction results of the semantic confidence corresponding to the sentence “turn on the lights” shown in Figure 4 The prediction confidence is 0.95 as shown in the figure), then the prediction result can be compared with the corresponding true semantic confidence label, and the model parameters of the initial semantic confidence prediction model can be updated according to the difference between the two, so as to train the initial semantic confidence prediction model and generate a semantic confidence prediction model.

[0111] On this basis, after the semantic confidence prediction model is generated based on the driving data of the sample vehicle and the sample speech uttered by the onboard user of the sample vehicle, the generated semantic confidence prediction model can be further verified using the driving data of the verification vehicle and the verification speech uttered by the onboard user of the verification vehicle. The specific verification process may include the following steps ①-④:

[0112] Step ①: Acquire the driving data of the verification vehicle; and extract the verification feature vector corresponding to the driving data of the verification vehicle; wherein the driving data of the verification vehicle includes the verification vehicle's own state information and the verification vehicle's environmental data information.

[0113] In this embodiment, in order to verify the semantic confidence prediction model, it is first necessary to obtain the driving data of the verification vehicle and the verification voice issued by the on-board user of the verification vehicle, such as obtaining the voice command issued by the driver in the verification vehicle as the verification voice, and obtaining the corresponding driving data of the verification vehicle when the voice command issued by the driver is obtained, including but not limited to the verification vehicle's own state information (such as driving speed, acceleration, engine status, etc.) and the verification vehicle's environmental data information (such as road conditions, traffic signs, whether there are pedestrians around, etc.). Among them, the driving data of the verification vehicle refers to the vehicle driving data that can be used to verify the semantic confidence prediction model. After obtaining the driving data of these verification vehicles, the driving data of the verification vehicle can be encoded using existing or future encoding methods to extract the verification feature vector corresponding to the driving data of the verification vehicle, and the subsequent step ③ can be continued.

[0114] Step ②: Obtain the verification voice uttered by the user on the verification vehicle, and determine the true semantic confidence of the text recognition result corresponding to the verification voice.

[0115] In this embodiment, after obtaining the verification voice issued by the on-board user (such as the driver) in the verification vehicle, each verification voice can be further recognized for the first time through preprocessing, feature extraction, processing using acoustic models and language models, and decoding, and the text recognition results corresponding to each verification voice can be obtained to achieve speech-to-text conversion, and the accurate semantic confidence (i.e., true semantic confidence) of the text recognition result corresponding to each verification voice can be manually marked to execute the subsequent step ④.

[0116] Step ③: Input the verification feature vector and the text recognition result corresponding to the verification speech into the semantic confidence prediction model to obtain the prediction confidence of the text recognition result corresponding to the verification speech.

[0117] After extracting the verification feature vector corresponding to the driving data of the verification vehicle through step ①, the verification feature vector and the text recognition result corresponding to the verification voice can be further input into the semantic confidence prediction model to obtain the prediction confidence of the text recognition result corresponding to the verification voice, so as to execute the subsequent step ④.

[0118] Step ④: When the prediction confidence of the text recognition result corresponding to the verification speech is inconsistent with the true semantic confidence, the driving data of the verification vehicle is used again as the driving data of the sample vehicle, and the verification speech is used again as the sample speech to update the semantic confidence prediction model.

[0119] After obtaining the prediction confidence of the text recognition result corresponding to the verification voice through step ③, if the prediction confidence is inconsistent with the true semantic confidence of the text recognition result corresponding to the verification voice determined through step ②, the driving data of the verification vehicle can be used again as the driving data of the sample vehicle, and the verification voice can be used again as the sample voice to update the parameters of the semantic confidence prediction model to further improve the prediction accuracy and precision of the semantic confidence prediction model.

[0120] S104: Using the text confidence prediction results corresponding to each of the N candidate text recognition results, perform secondary recognition on the target speech to obtain a final recognition result corresponding to the target speech.

[0121] In this embodiment, after predicting the text confidence prediction results corresponding to each of the N candidate text recognition results through step S103, the existing or future data fusion method can be further used to fuse the probability score of the initial recognition corresponding to each candidate text recognition result in the N candidate text recognition results with the text confidence prediction result, so as to perform a secondary recognition on the target speech according to the processing result. After comprehensively considering the possibility and reliability of the speech recognition results, a more accurate final recognition result corresponding to the target speech is obtained, thereby effectively improving the accuracy of the speech recognition results of the on-board users of the target vehicle, thereby enhancing the driving and riding experience of the on-board users and improving the driving safety of the target vehicle.

[0122] Specifically, an optional implementation method is to first perform a weighted sum calculation on the probability score of the initial recognition corresponding to each candidate text recognition result in the N candidate text recognition results and the text confidence prediction result to obtain the recognition score corresponding to each candidate text recognition result. The specific calculation formula is as follows:

[0123] finalScore=ω1*ASRProb+ω2*SegProb

[0124] Among them, finalScore represents the recognition score corresponding to each candidate text recognition result; ASRProb represents the probability score of the initial recognition corresponding to each candidate text recognition result; SegProb represents the text confidence prediction result corresponding to each candidate text recognition result; ω1 and ω2 represent the weight coefficients of ASRProb and SegProb respectively. The specific values ​​are not limited and can be set according to actual conditions and experience values. For example, the two can be set to 0.6 and 0.4 respectively.

[0125] In addition, other more complex fusion methods, such as fusion models based on deep learning, can also be used for fusion processing. In this way, after re-scoring each candidate text recognition result, the candidate text recognition result corresponding to the highest recognition score (finalScore) can be selected from all the re-scored N candidate text recognition results as the final recognition result corresponding to the target speech.

[0126] In summary, the present embodiment provides a speech recognition method, which first obtains the driving data of the target vehicle and extracts the target feature vector corresponding to the driving data; wherein the driving data includes the target vehicle's own state information and the environmental data information of the target vehicle, and then obtains the target speech to be recognized issued by the on-board user of the target vehicle, and performs an initial recognition on the target speech to obtain N candidate text recognition results; wherein N is a positive integer greater than 1; then, the target feature vector and the N candidate text recognition results are input into a pre-constructed semantic confidence prediction model, and the text confidence prediction results corresponding to each of the N candidate text recognition results are predicted, and then the target speech is secondary recognized using the text confidence prediction results corresponding to each of the N candidate text recognition results to obtain the final recognition result corresponding to the target speech.

[0127] It can be seen that, since the present application introduces driving data such as the target vehicle's own state information and the target vehicle's environmental data information as important recognition basis when recognizing the target speech emitted by the on-board user of the target vehicle, and uses the pre-constructed semantic confidence prediction model to accurately determine the confidence corresponding to the N candidate text recognition results obtained in the initial recognition as the screening basis, it is possible to screen out more accurate candidate text recognition results from the N candidate text recognition results as the final recognition result corresponding to the target speech, effectively improving the recognition accuracy of the target speech, thereby enhancing the driving and riding experience of the on-board user, and improving the driving safety of the target vehicle.

[0128] Second embodiment

[0129] This embodiment will introduce a speech recognition device. For related content, please refer to the above method embodiment.

[0130] See also Figure 5 , is a schematic diagram of the composition of a speech recognition device provided in this embodiment, the device 500 includes:

[0131] The first acquisition unit 501 is used to acquire the driving data of the target vehicle and extract the target feature vector corresponding to the driving data; the driving data includes the self state information of the target vehicle and the environmental data information of the target vehicle;

[0132] The second acquisition unit 502 is used to acquire the target speech to be recognized uttered by the on-board user of the target vehicle, and perform initial recognition on the target speech to obtain N candidate text recognition results; wherein N is a positive integer greater than 1;

[0133] The prediction unit 503 is used to input the target feature vector and the N candidate text recognition results into a pre-built semantic confidence prediction model to predict the text confidence prediction results corresponding to the N candidate text recognition results.

[0134] The recognition unit 504 is used to perform secondary recognition on the target speech by using the text confidence prediction results corresponding to each of the N candidate text recognition results to obtain a final recognition result corresponding to the target speech.

[0135] In an implementation of this embodiment, the first acquiring unit 501 includes:

[0136] A first encoding subunit is used to encode the self state information of the target vehicle to obtain a running state feature vector of the target vehicle;

[0137] A second encoding subunit is used to encode the environmental data information of the target vehicle to obtain an environmental information feature vector;

[0138] The fusion subunit is used to fuse the running state feature vector and the environmental information feature vector of the target vehicle to obtain a fused feature vector as the target feature vector corresponding to the driving data.

[0139] In an implementation of this embodiment, the fusion subunit is specifically used for:

[0140] The running state feature vector and the environmental information feature vector of the target vehicle are concatenated or weighted summed to obtain a fused feature vector as the target feature vector corresponding to the driving data.

[0141] In an implementation of this embodiment, the second obtaining unit 502 is specifically configured to:

[0142] The acoustic features of the target speech are extracted; and according to the acoustic features, the target speech is decoded to obtain N candidate text recognition results.

[0143] In an implementation of this embodiment, the device further includes:

[0144] A third acquisition unit is used to acquire driving data of a sample vehicle; and extract a sample feature vector corresponding to the driving data of the sample vehicle; the driving data of the sample vehicle includes the self state information of the sample vehicle and the environmental data information of the sample vehicle;

[0145] A fourth acquisition unit, configured to acquire a sample speech uttered by a user on the sample vehicle, and determine a true semantic confidence of a text recognition result corresponding to the sample speech;

[0146] The training unit is used to train the initial semantic confidence prediction model according to the sample feature vector, the text recognition result corresponding to the sample speech and its corresponding true semantic confidence, so as to generate the semantic confidence prediction model.

[0147] In an implementation of this embodiment, the device further includes:

[0148] A fifth acquisition unit is used to acquire the driving data of the verification vehicle; and extract the verification feature vector corresponding to the driving data of the verification vehicle; the driving data of the verification vehicle includes the self state information of the verification vehicle and the environmental data information of the verification vehicle;

[0149] a sixth acquisition unit, configured to acquire a verification voice issued by a user on the verification vehicle, and determine a true semantic confidence level of a text recognition result corresponding to the verification voice;

[0150] An obtaining unit, used for inputting the verification feature vector and the text recognition result corresponding to the verification speech into the semantic confidence prediction model to obtain the prediction confidence of the text recognition result corresponding to the verification speech;

[0151] An updating unit is used to update the semantic confidence prediction model by reusing the driving data of the verification vehicle as the driving data of the sample vehicle and reusing the verification speech as the sample speech when the predicted confidence of the text recognition result corresponding to the verification speech is inconsistent with the true semantic confidence.

[0152] In an implementation of this embodiment, the identification unit 504 includes:

[0153] A calculation subunit, used for performing weighted sum calculation on the probability score of the initial recognition corresponding to each candidate text recognition result in the N candidate text recognition results and the text confidence prediction result, to obtain a recognition score corresponding to each candidate text recognition result;

[0154] The selection subunit is used to select the candidate text recognition result corresponding to the highest recognition score from the N candidate text recognition results as the final recognition result corresponding to the target speech.

[0155] Furthermore, an embodiment of the present application also provides a speech recognition device, including: a processor, a memory, and a system bus;

[0156] The processor and the memory are connected via the system bus;

[0157] The memory is used to store one or more programs, and the one or more programs include instructions. When the instructions are executed by the processor, the processor executes any one of the implementation methods of the above-mentioned speech recognition method.

[0158] Furthermore, an embodiment of the present application also provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes any one of the implementation methods of the above-mentioned speech recognition method.

[0159] Furthermore, an embodiment of the present application also provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute any one of the implementation methods of the above-mentioned speech recognition method.

[0160] It can be known from the description of the above implementation mode that those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment method can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product can be stored in a storage medium such as ROM / RAM, a disk, an optical disk, etc., including several instructions for enabling a computer device (which can be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the methods described in the various embodiments of the present application or certain parts of the embodiments.

[0161] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part description.

[0162] It should also be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0163] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A speech recognition method, characterized in that: include: Acquire the driving data of the target vehicle and extract the target feature vector corresponding to the driving data; the driving data includes the self state information of the target vehicle and the environmental data information of the target vehicle; Acquire a target speech to be recognized issued by a user on the target vehicle, and perform initial recognition on the target speech to obtain N candidate text recognition results; wherein N is a positive integer greater than 1; Inputting the target feature vector and N candidate text recognition results into a pre-built semantic confidence prediction model to predict text confidence prediction results corresponding to each of the N candidate text recognition results; The target speech is recognized a second time using the text confidence prediction results corresponding to each of the N candidate text recognition results to obtain a final recognition result corresponding to the target speech.

2. The method according to claim 1, characterized in that The extracting the target feature vector corresponding to the driving data includes: Encoding the self-state information of the target vehicle to obtain a running state feature vector of the target vehicle; Encoding the environmental data information of the target vehicle to obtain an environmental information feature vector; The running state feature vector and the environmental information feature vector of the target vehicle are fused to obtain a fused feature vector as the target feature vector corresponding to the driving data.

3. The method according to claim 2, characterized in that The step of fusing the running state feature vector and the environmental information feature vector of the target vehicle to obtain a fused feature vector as the target feature vector corresponding to the driving data includes: The running state feature vector and the environmental information feature vector of the target vehicle are concatenated or weighted summed to obtain a fused feature vector as the target feature vector corresponding to the driving data.

4. The method according to claim 1, characterized in that: The target speech is initially recognized to obtain N candidate text recognition results, including: The acoustic features of the target speech are extracted; and according to the acoustic features, the target speech is decoded to obtain N candidate text recognition results.

5. The method according to claim 1, characterized in that The semantic confidence prediction model is constructed as follows: Acquire driving data of a sample vehicle; and extract a sample feature vector corresponding to the driving data of the sample vehicle; the driving data of the sample vehicle includes the self state information of the sample vehicle and the environmental data information of the sample vehicle; Acquire a sample speech uttered by a user on the sample vehicle, and determine a true semantic confidence of a text recognition result corresponding to the sample speech; The initial semantic confidence prediction model is trained according to the sample feature vector, the text recognition result corresponding to the sample speech and its corresponding true semantic confidence to generate the semantic confidence prediction model.

6. The method according to claim 5, characterized in that The method further comprises: Acquire the driving data of the verification vehicle; and extract the verification feature vector corresponding to the driving data of the verification vehicle; the driving data of the verification vehicle includes the self state information of the verification vehicle and the environmental data information of the verification vehicle; Acquire a verification voice uttered by a user on the verification vehicle, and determine a true semantic confidence level of a text recognition result corresponding to the verification voice; Inputting the verification feature vector and the text recognition result corresponding to the verification speech into the semantic confidence prediction model to obtain the prediction confidence of the text recognition result corresponding to the verification speech; When the predicted confidence of the text recognition result corresponding to the verification speech is inconsistent with the true semantic confidence, the driving data of the verification vehicle is re-used as the driving data of the sample vehicle, and the verification speech is re-used as the sample speech to update the semantic confidence prediction model.

7. The method according to any one of claims 1 to 6, characterized in that: The method of performing secondary recognition on the target speech by using the text confidence prediction results corresponding to each of the N candidate text recognition results to obtain a final recognition result corresponding to the target speech includes: Performing a weighted sum calculation on the probability score of the initial recognition corresponding to each candidate text recognition result in the N candidate text recognition results and the text confidence prediction result to obtain a recognition score corresponding to each candidate text recognition result; The candidate text recognition result corresponding to the highest recognition score is selected from the N candidate text recognition results as the final recognition result corresponding to the target speech.

8. A speech recognition device, characterized in that: include: A first acquisition unit is used to acquire the driving data of the target vehicle and extract the target feature vector corresponding to the driving data; the driving data includes the self state information of the target vehicle and the environmental data information of the target vehicle; A second acquisition unit is used to acquire a target speech to be recognized issued by a user on the target vehicle, and perform initial recognition on the target speech to obtain N candidate text recognition results; wherein N is a positive integer greater than 1; A prediction unit, used for inputting the target feature vector and N candidate text recognition results into a pre-built semantic confidence prediction model, and predicting text confidence prediction results corresponding to each of the N candidate text recognition results; The recognition unit is used to perform secondary recognition on the target speech by using the text confidence prediction results corresponding to each of the N candidate text recognition results to obtain a final recognition result corresponding to the target speech.

9. A speech recognition device, characterized in that: include: Processor, memory, system bus; The processor and the memory are connected via the system bus; The memory is used to store one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by the processor, the processor executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device executes the method according to any one of claims 1 to 7.

Images