Speech recognition method, device, storage medium and equipment

Abstract

The present application discloses a speech recognition method, apparatus, storage medium and equipment, which comprises: firstly obtaining driving data of a target vehicle and extracting a target feature vector therefrom; wherein the driving data includes the target vehicle's own state information and the data information of the environment in which it is located; then obtaining a target speech emitted by an onboard user of the target vehicle, and performing an initial recognition on the target speech to obtain N candidate text recognition results; then inputting the target feature vector and the N candidate text recognition results into a semantic confidence prediction model to predict the text confidence prediction results corresponding to each of the N candidate text recognition results; then performing a secondary recognition on the target speech using the text confidence prediction results corresponding to each of the N candidate text recognition results, so as to screen out a final recognition result corresponding to the target speech from the N candidate text recognition results, thereby improving the recognition accuracy and enhancing the riding experience and driving safety of the onboard user.

Description

Technical Field

[0001] The present application relates to the field of vehicle technology, and in particular to a speech recognition method, apparatus, storage medium, and device. Background Art

[0002] With the improvement of people's living standards and the rapid development of the social economy, the use of cars has gradually increased, and people's demand for intelligent driving has also become increasingly higher. In-vehicle voice recognition systems allow drivers to control various vehicle functions, such as setting navigation, playing music, or answering calls, without leaving the steering wheel, simply by issuing voice commands. While this can improve driving safety, the system may sometimes misidentify the driver's commands, especially those with similar pronunciations, which are more likely to be confused, affecting the user's driving experience and even affecting driving safety in emergency situations.

[0003] Therefore, it is necessary to more accurately recognize the voices of in-vehicle users to improve the riding experience and driving safety. However, existing methods for improving the accuracy of speech recognition results usually rely mainly on preset industry error correction vocabulary or the acoustic and semantic features of characters. Although these methods have improved the accuracy of speech recognition error correction to a certain extent, they do not take into account the changes in the vehicle's dynamic environment and real-time status. In vehicle usage scenarios, due to the uncertainty of factors such as road conditions, traffic conditions, and weather changes, fixed recognition result error correction strategies may not be flexible enough to adapt to rapidly changing driving environments, reducing the accuracy of speech recognition results, resulting in a poor driving and riding experience for users and affecting the vehicle's driving safety. Summary of the Invention

[0004] The main purpose of the embodiments of the present application is to provide a speech recognition method, device, storage medium and equipment, which can effectively improve the accuracy of the speech recognition results of vehicle-mounted users, thereby enhancing the user's driving and riding experience and improving the vehicle's driving safety.

[0005] The present invention provides a speech recognition method, including:

[0006] Acquire driving data of a target vehicle and extract a target feature vector corresponding to the driving data; the driving data includes the target vehicle's own state information and the target vehicle's environment data information;

[0007] Acquire a target speech to be recognized uttered by a user on the target vehicle, and perform initial recognition on the target speech to obtain N candidate text recognition results, where N is a positive integer greater than 1;

[0008] Inputting the target feature vector and N candidate text recognition results into a pre-built semantic confidence prediction model to predict text confidence prediction results corresponding to each of the N candidate text recognition results;

[0009] The target speech is recognized a second time using the text confidence prediction results corresponding to each of the N candidate text recognition results to obtain a final recognition result corresponding to the target speech.

[0010] In one possible implementation, extracting the target feature vector corresponding to the driving data includes:

[0011] Encoding the state information of the target vehicle to obtain a running state feature vector of the target vehicle;

[0012] Encoding the environmental data information of the target vehicle to obtain an environmental information feature vector;

[0013] The running state feature vector and the environmental information feature vector of the target vehicle are fused to obtain a fused feature vector as the target feature vector corresponding to the driving data.

[0014] In a possible implementation, fusing the target vehicle's operating state feature vector and the environmental information feature vector to obtain a fused feature vector as the target feature vector corresponding to the driving data includes:

[0015] The running state feature vector and the environmental information feature vector of the target vehicle are concatenated or weightedly summed to obtain a fused feature vector as the target feature vector corresponding to the driving data.

[0016] In one possible implementation, the performing of the initial recognition on the target speech to obtain N candidate text recognition results includes:

[0017] Extracting acoustic features of the target speech; and decoding the target speech based on the acoustic features to obtain N candidate text recognition results.

[0018] In one possible implementation, the semantic confidence prediction model is constructed as follows:

[0019] Acquire driving data of a sample vehicle; and extract a sample feature vector corresponding to the driving data of the sample vehicle; the driving data of the sample vehicle includes state information of the sample vehicle itself and environmental data information of the sample vehicle;

[0020] Obtaining a sample speech uttered by a user on the sample vehicle, and determining a true semantic confidence level of a text recognition result corresponding to the sample speech;

[0021] An initial semantic confidence prediction model is trained according to the sample feature vector, the text recognition result corresponding to the sample speech and its corresponding true semantic confidence to generate the semantic confidence prediction model.

[0022] In a possible implementation, the method further includes:

[0023] Acquire driving data of a verification vehicle; and extract a verification feature vector corresponding to the driving data of the verification vehicle; the driving data of the verification vehicle includes the state information of the verification vehicle itself and the environmental data information of the verification vehicle;

[0024] Acquire a verification voice uttered by a user on board the verification vehicle, and determine a true semantic confidence level of a text recognition result corresponding to the verification voice;

[0025] Inputting the verification feature vector and the text recognition result corresponding to the verification speech into the semantic confidence prediction model to obtain the prediction confidence of the text recognition result corresponding to the verification speech;

[0026] When the predicted confidence of the text recognition result corresponding to the verification voice is inconsistent with the true semantic confidence, the driving data of the verification vehicle is re-used as the driving data of the sample vehicle, and the verification voice is re-used as the sample voice to update the semantic confidence prediction model.

[0027] In one possible implementation, performing secondary recognition on the target speech using the text confidence prediction results corresponding to each of the N candidate text recognition results to obtain a final recognition result corresponding to the target speech includes:

[0028] Performing a weighted sum calculation on the probability score of the initial recognition corresponding to each candidate text recognition result and the text confidence prediction result of the N candidate text recognition results to obtain a recognition score corresponding to each candidate text recognition result;

[0029] The candidate text recognition result corresponding to the highest recognition score is selected from the N candidate text recognition results as the final recognition result corresponding to the target speech.

[0030] The present application also provides a speech recognition device, comprising:

[0031] A first acquisition unit is configured to acquire driving data of a target vehicle and extract a target feature vector corresponding to the driving data; the driving data includes state information of the target vehicle and environmental data information of the target vehicle;

[0032] a second acquiring unit, configured to acquire a target speech to be recognized uttered by a user on board the target vehicle, and perform initial recognition on the target speech to obtain N candidate text recognition results; wherein N is a positive integer greater than 1;

[0033] A prediction unit, configured to input the target feature vector and N candidate text recognition results into a pre-built semantic confidence prediction model, and predict text confidence prediction results corresponding to each of the N candidate text recognition results;

[0034] The recognition unit is used to perform secondary recognition on the target speech using the text confidence prediction results corresponding to each of the N candidate text recognition results to obtain a final recognition result corresponding to the target speech.

[0035] In a possible implementation, the first acquiring unit includes:

[0036] A first encoding subunit is used to encode the state information of the target vehicle to obtain a running state feature vector of the target vehicle;

[0037] A second encoding subunit is used to encode the environmental data information of the target vehicle to obtain an environmental information feature vector;

[0038] The fusion subunit is used to fuse the running state feature vector and the environmental information feature vector of the target vehicle to obtain a fused feature vector as the target feature vector corresponding to the driving data.

[0039] In a possible implementation, the fusion subunit is specifically used to:

[0040] The running state feature vector and the environmental information feature vector of the target vehicle are concatenated or weightedly summed to obtain a fused feature vector as the target feature vector corresponding to the driving data.

[0041] In a possible implementation, the second acquiring unit is specifically configured to:

[0042] Extracting acoustic features of the target speech; and decoding the target speech based on the acoustic features to obtain N candidate text recognition results.

[0043] In a possible implementation, the apparatus further includes:

[0044] A third acquisition unit is configured to acquire driving data of a sample vehicle and extract a sample feature vector corresponding to the driving data of the sample vehicle; the driving data of the sample vehicle includes state information of the sample vehicle itself and environmental data information of the sample vehicle;

[0045] a fourth acquiring unit, configured to acquire a sample speech uttered by a user on board the sample vehicle, and determine a true semantic confidence level of a text recognition result corresponding to the sample speech;

[0046] The training unit is used to train the initial semantic confidence prediction model according to the sample feature vector, the text recognition result corresponding to the sample speech and its corresponding true semantic confidence, so as to generate the semantic confidence prediction model.

[0047] In a possible implementation, the apparatus further includes:

[0048] a fifth acquisition unit, configured to acquire driving data of the verification vehicle; and extract a verification feature vector corresponding to the driving data of the verification vehicle; the driving data of the verification vehicle includes the state information of the verification vehicle itself and the environmental data information of the verification vehicle;

[0049] a sixth acquiring unit, configured to acquire a verification voice uttered by a user on board the verification vehicle, and determine a true semantic confidence level of a text recognition result corresponding to the verification voice;

[0050] an obtaining unit, configured to input the verification feature vector and the text recognition result corresponding to the verification speech into the semantic confidence prediction model to obtain the prediction confidence of the text recognition result corresponding to the verification speech;

[0051] An updating unit is used to update the semantic confidence prediction model by reusing the driving data of the verification vehicle as the driving data of the sample vehicle and reusing the verification speech as the sample speech when the predicted confidence of the text recognition result corresponding to the verification speech is inconsistent with the true semantic confidence.

[0052] In one possible implementation, the identification unit includes:

[0053] a calculation subunit, configured to perform a weighted sum calculation on the probability score of the initial recognition corresponding to each candidate text recognition result and the text confidence prediction result of the N candidate text recognition results to obtain a recognition score corresponding to each candidate text recognition result;

[0054] The selection subunit is used to select the candidate text recognition result corresponding to the highest recognition score from the N candidate text recognition results as the final recognition result corresponding to the target speech.

[0055] The embodiment of the present application also provides a speech recognition device, comprising: a processor, a memory, and a system bus;

[0056] The processor and the memory are connected via the system bus;

[0057] The memory is used to store one or more programs, and the one or more programs include instructions. When the instructions are executed by the processor, the processor performs any one implementation of the above-mentioned speech recognition method.

[0058] An embodiment of the present application also provides a computer-readable storage medium, which stores instructions. When the instructions are executed on a terminal device, the terminal device executes any one of the implementation methods of the above-mentioned speech recognition method.

[0059] An embodiment of the present application further provides a computer program product, which, when executed on a terminal device, enables the terminal device to execute any one of the implementations of the above-mentioned speech recognition method.

[0060] A speech recognition method, apparatus, storage medium, and device provided in an embodiment of the present application first obtain driving data of a target vehicle and extract a target feature vector corresponding to the driving data; wherein the driving data includes the target vehicle's own state information and the target vehicle's environmental data information, then obtains a target speech to be recognized emitted by a user on board the target vehicle, and performs an initial recognition on the target speech to obtain N candidate text recognition results; wherein N is a positive integer greater than 1; then, the target feature vector and the N candidate text recognition results are input into a pre-constructed semantic confidence prediction model, and the text confidence prediction results corresponding to each of the N candidate text recognition results are predicted, and then the target speech is secondary recognized using the text confidence prediction results corresponding to each of the N candidate text recognition results to obtain a final recognition result corresponding to the target speech.

[0061] It can be seen that, since this application introduces driving data such as the target vehicle's own state information and the target vehicle's environmental data information as important recognition basis when recognizing the target voice emitted by the on-board user of the target vehicle, and uses the pre-built semantic confidence prediction model to accurately determine the confidence corresponding to the N candidate text recognition results obtained in the initial recognition as the screening basis, it can screen out more accurate candidate text recognition results from the N candidate text recognition results as the final recognition result corresponding to the target voice, effectively improving the recognition accuracy of the target voice, thereby enhancing the driving and riding experience of the on-board user, and improving the driving safety of the target vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0063] Figure 1 A flowchart of a speech recognition method provided in an embodiment of the present application;

[0064] Figure 2 An example diagram of the process of encoding the target vehicle's own state information to obtain the target vehicle's operating state feature vector provided in an embodiment of the present application;

[0065] Figure 3 An example diagram of the process of inputting a target feature vector and three candidate text recognition results into a semantic confidence prediction model to predict the text confidence prediction results corresponding to each of the three candidate text recognition results, provided in an embodiment of the present application;

[0066] Figure 4 This is an example diagram of the training process of the semantic confidence prediction model provided in an embodiment of the present application;

[0067] Figure 5 A schematic diagram of the composition of a speech recognition device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0068] With the increasing popularity of automobiles, people's demands for vehicle intelligence and safety are becoming increasingly demanding. In-vehicle voice recognition systems, as a core component in achieving vehicle intelligence, are gradually being widely adopted in a variety of vehicle models. The introduction of in-vehicle voice recognition systems has greatly facilitated the driver's operating experience. They allow drivers to control various vehicle functions, such as setting navigation, playing music, or answering calls, simply by issuing voice commands without leaving the steering wheel. This design not only improves driving safety but also makes the entire driving process easier and more enjoyable.

[0069] However, despite the benefits of existing in-vehicle voice recognition systems, they still face some challenges in practical use. The most significant issue is misidentification. Due to the complexity of speech and the influence of the surrounding environment, these systems sometimes misidentify the driver's instructions, especially those with similar pronunciations. For example, when driving at night, if the driver wants to "turn on the lights" through voice command, the in-vehicle voice recognition system may mistakenly recognize it as "open the windows." This misjudgment not only affects the driver's driving experience but can even affect driving safety in an emergency.

[0070] Therefore, it is necessary to improve the recognition performance of existing in-vehicle voice recognition systems to effectively increase recognition accuracy and reduce false recognition rates. Such in-vehicle voice recognition systems can better adapt to changing driving needs and conditions, meet the needs of the growing automotive intelligent market, and ensure a safe driving experience for drivers.

[0071] However, existing methods for improving the accuracy of speech recognition results usually rely mainly on preset industry error correction vocabulary or the acoustic and semantic features of characters. Although these methods have improved the accuracy of speech recognition error correction to a certain extent, they ignore the importance of vehicle status information and driving environment for understanding voice commands. For example, when a vehicle is driving at high speed, the driver may issue emergency safety-related commands, and such commands need to be processed and responded to first. However, existing methods for improving speech recognition may not be able to effectively distinguish these emergency commands from other non-emergency commands, resulting in delayed or erroneous responses.

[0072] It can be seen that in vehicle usage scenarios, due to the uncertainty of factors such as road conditions, traffic conditions, and weather changes, fixed recognition result correction strategies may not be flexible enough and cannot adapt to the rapidly changing driving environment, reducing the accuracy of voice recognition results, resulting in poor driving and riding experience for users and affecting the vehicle's driving safety.

[0073] To address the above-mentioned defects, the present application provides a speech recognition method, which first obtains the driving data of a target vehicle and extracts a target feature vector corresponding to the driving data; wherein the driving data includes the target vehicle's own state information and the environmental data information of the target vehicle, and then obtains the target speech to be recognized emitted by the on-board user of the target vehicle, and performs an initial recognition on the target speech to obtain N candidate text recognition results; wherein N is a positive integer greater than 1; then, the target feature vector and the N candidate text recognition results are input into a pre-constructed semantic confidence prediction model, and the text confidence prediction results corresponding to each of the N candidate text recognition results are predicted, and then the target speech is secondary recognized using the text confidence prediction results corresponding to each of the N candidate text recognition results to obtain the final recognition result corresponding to the target speech.

[0074] It can be seen that, since this application introduces driving data such as the target vehicle's own state information and the target vehicle's environmental data information as important recognition basis when recognizing the target voice emitted by the on-board user of the target vehicle, and uses the pre-built semantic confidence prediction model to accurately determine the confidence corresponding to the N candidate text recognition results obtained in the initial recognition as the screening basis, it can screen out more accurate candidate text recognition results from the N candidate text recognition results as the final recognition result corresponding to the target voice, effectively improving the recognition accuracy of the target voice, thereby enhancing the driving and riding experience of the on-board user, and improving the driving safety of the target vehicle.

[0075] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0076] First embodiment

[0077] See also Figure 1 , is a flow chart of a speech recognition method provided in this embodiment, the method comprising the following steps:

[0078] S101: Acquire driving data of a target vehicle and extract a target feature vector corresponding to the driving data; wherein the driving data includes the target vehicle's own state information and the target vehicle's environment data information.

[0079] In this embodiment, any vehicle undergoing in-vehicle user speech recognition using this embodiment is defined as a target vehicle, and any speech uttered by a user in the target vehicle that requires recognition is defined as a target speech to be recognized. It should be noted that this embodiment does not restrict the language type of the target speech; for example, the target speech can be in Chinese or English. Furthermore, this embodiment does not restrict the length of the target speech; for example, the target speech can be a single sentence or multiple sentences. It is understood that the target speech can be obtained through recording or other means as needed. For example, a pre-installed microphone in the target vehicle can be used to capture the speech of the user as the target speech. It should also be noted that to improve the recognition accuracy of the target speech, a noise suppression function can be pre-installed in the target vehicle to reduce interference from interior noise on the target speech signal. Alternatively, advanced signal processing algorithms can be employed to accurately capture the speech signal of the user in the vehicle even in noisy environments.

[0080] In addition, in order to improve the recognition accuracy of the target speech, while obtaining the target speech, it is also necessary to use various sensors pre-installed on the target vehicle to obtain the driving data of the target vehicle during driving, and use existing or future feature extraction methods to extract the target feature vector corresponding to the driving data to execute the subsequent step S103.

[0081] The target vehicle's driving data may include, but is not limited to, the target vehicle's own state information and the target vehicle's surrounding environment data. The target vehicle's own state information may include, but is not limited to, driving speed, acceleration, engine status, gear position, headlight status, driving position, etc. The target vehicle's surrounding environment data may include, but is not limited to, road conditions, traffic signs, the presence of pedestrians nearby, and the location of other vehicles.

[0082] Furthermore, it should be noted that the present application does not limit the method for obtaining the driving data of the target vehicle. Generally, the data can be obtained by using various sensors pre-installed on the target vehicle and other vehicle-mounted systems. Furthermore, the present application does not limit the specific structure, quantity, and installation location of each sensor on the target vehicle. For example, the target vehicle can be pre-installed with a vehicle speed sensor (VSS), an acceleration sensor, and other sensors to obtain various driving data such as the target vehicle's driving speed and acceleration. It can also be pre-installed with on-board cameras, radars, and other sensors to perceive the surrounding environment in real time to obtain environmental information of the target vehicle. In addition, the number of each sensor can be one or more.

[0083] Specifically, in an optional implementation manner, the specific implementation process of "obtaining the driving data of the target vehicle and extracting the target feature vector corresponding to the driving data" in step S101 may include the following steps S1011-S1013:

[0084] S1011: Encode the target vehicle's own state information to obtain a running state feature vector of the target vehicle.

[0085] In this implementation, the target vehicle's own state information (such as driving speed, acceleration, engine state, gear position, headlight state, driving position, etc.) included in the driving data can be encoded using existing or future encoding methods to obtain the target vehicle's corresponding running state feature vector for executing the subsequent step S1013. For example, Figure 2As shown in the figure, the target vehicle's own state information can be encoded using one-hot encoding to obtain encoding vectors such as "010000" for acceleration, "100000" for driving speed, "00010" for headlight state, and "001000" for engine state. These one-hot encoding vectors are then input into the neural network embedding layer to generate a 256-dimensional feature vector (the specific dimension is not limited, 256 dimensions are used as an example) as the target vehicle's operating state feature vector, and defined as carEmbH, which is used to represent the target vehicle's current operating state.

[0086] S1012: Encode the environmental data information of the target vehicle to obtain an environmental information feature vector.

[0087] In this implementation, the environmental data information of the target vehicle included in the driving data (such as road conditions, traffic signs, whether there are pedestrians around, the location of other vehicles, etc.) can also be encoded using existing or future encoding methods to obtain an environmental information feature vector, which is defined as envEmbH to perform subsequent step S1013. For example, the environmental data information of the target vehicle can still be encoded using a one-hot encoding method to obtain a one-hot encoding vector, which is then input into the neural network Embedding layer to generate a 256-dimensional feature vector (the specific dimension is not limited, and 256 dimensions are used as an example) as an environmental information feature vector to represent the environmental information characteristics of the current surroundings of the target vehicle.

[0088] S1013: Fusing the target vehicle's operating state feature vector and the environmental information feature vector to obtain a fused feature vector as the target feature vector corresponding to the driving data.

[0089] In this implementation, after obtaining the target vehicle's operating status feature vector carEmbH through step S1011 and the environmental information feature vector envEmbH through step S1012, the target vehicle's operating status feature vector carEmbH and the environmental information feature vector envEmbH can be further fused using existing or future feature fusion methods to fully integrate the target vehicle's own status information and the surrounding environment information, and obtain a fused feature vector that more comprehensively and accurately reflects the actual situation of the target vehicle, which serves as the target feature vector corresponding to the driving data, and is used to execute the subsequent step S103.

[0090] Specifically, the target vehicle's operating state feature vector carEmbH and the environmental information feature vector envEmbH can be spliced ​​in a preset order (the specific splicing order is not limited, such as the two can be directly spliced, etc.) to form a longer fusion feature vector as the target feature vector corresponding to the driving data. This fusion method can directly combine the two types of information together, retaining the integrity of the original information. For example, the target vehicle's operating state feature vector carEmbH can be placed in front, and then the environmental information feature vector envEmbH can be connected, so that the semantic confidence prediction model mentioned in the subsequent step S103 can process these two parts of information in turn, and fully explore the associations and features therein.

[0091] Alternatively, the target vehicle's operating state feature vector carEmbH and the environmental information feature vector envEmbH can be weighted and summed to obtain a fused feature vector as the target feature vector corresponding to the driving data. That is, different weights can be assigned to the target vehicle's operating state feature vector carEmbH and the environmental information feature vector envEmbH, respectively, and then the summation operation can be performed. The weights can be adjusted according to the actual application scenario and the importance of the two types of information. For example, if the vehicle's surrounding environment information has a greater impact on the vehicle's decision-making under specific traffic conditions, a higher weight can be assigned to the environmental information feature vector envEmbH. This approach can more flexibly adjust the contribution of the two types of information in the fusion result to adapt to different situations.

[0092] It should be noted that this application does not limit the feature fusion method used for the target vehicle's operating status feature vector carEmbH and the environmental information feature vector envEmbH. According to different application scenarios and needs, the most appropriate fusion method can be flexibly selected to process the fusion of the two feature vectors to obtain a more accurate fusion feature vector.

[0093] S102: Obtain a target speech to be recognized uttered by a user on a target vehicle, perform initial recognition on the target speech, and obtain N candidate text recognition results; wherein N is a positive integer greater than 1.

[0094] In this embodiment, after the target voice emitted by the in-vehicle user is collected using a pre-installed vehicle microphone or other collection method on the target vehicle, in order to improve the recognition accuracy of the target voice, the target voice is further recognized initially through preprocessing, feature extraction, processing using acoustic models and language models, and decoding, and N candidate text recognition results are obtained to achieve voice-to-text conversion for executing the subsequent step S103.

[0095] Specifically, an optional implementation method is that after obtaining the target speech, the acoustic features of the target speech can be extracted using existing or future feature extraction methods, and the target speech can be decoded based on the acoustic features to obtain N candidate text recognition results.

[0096] In this implementation, when extracting the acoustic features of the target audio data, the target speech data must first be framed to obtain a corresponding speech frame sequence. The framed speech frame sequence is then pre-emphasized. The acoustic features of each speech frame are then extracted in sequence. These acoustic features are characteristic data used to characterize the acoustic information of the corresponding speech frame, such as Mel-scale Frequency Cepstral Coefficients (MFCCs) or Perceptual Linear Predictive (PLP) features. The probability of speech units is then calculated using an acoustic model such as a deep neural network. Combined with the language model optimization results, N candidate text recognition results are decoded.

[0097] It should be noted that the embodiments of the present application do not limit the method for extracting the acoustic features of the target audio data, nor do they limit the specific extraction process. An appropriate extraction method can be selected according to actual conditions, and corresponding feature extraction operations can be performed.

[0098] S103: Input the target feature vector and the N candidate text recognition results into a pre-built semantic confidence prediction model to predict the text confidence prediction results corresponding to the N candidate text recognition results.

[0099] In this embodiment, after extracting the target feature vector corresponding to the driving data through step S101 and performing the initial recognition of the target speech through step S102 to obtain N candidate text recognition results, in order to improve the recognition effect of the target speech, the obtained target feature vector and N candidate text recognition results can be further input into a pre-built semantic confidence prediction model. The semantic confidence prediction model will make full use of its complex neural network architecture and advanced algorithms to conduct in-depth analysis and comprehensive consideration of these rich input information. In this way, the text confidence prediction results corresponding to each of the N candidate text recognition results can be accurately predicted and defined as SegPrab to execute the subsequent step S104, providing a key basis for the subsequent optimization and decision-making of the speech recognition results.

[0100] For example: Figure 3As shown, assuming that the target feature vector corresponding to the driving data extracted through step S101 is "high-speed driving, no pedestrians around, and slightly dim light", and the target speech is initially recognized through step S102, the three candidate text recognition results are: "open the window", "turn on the lights", and "turn on the headlights". After the target feature vector and the three candidate text recognition results are input into the semantic confidence prediction model, the text confidence prediction results corresponding to the three candidate text recognition results can be predicted as follows: the prediction confidence corresponding to "open the window" is 0.65, the prediction confidence corresponding to "turn on the lights" is 0.95, and the prediction confidence corresponding to "turn on the headlights" is 0.85.

[0101] It should be noted that the specific composition and specific prediction process of the semantic confidence prediction model can be set according to actual conditions, and this embodiment does not limit this.

[0102] Next, this embodiment will introduce the construction process of the semantic confidence prediction model adopted in this application, which may specifically include the following steps (1)-(3):

[0103] Step (1): Acquire driving data of a sample vehicle; and extract a sample feature vector corresponding to the driving data of the sample vehicle; wherein the driving data of the sample vehicle includes the sample vehicle's own state information and the environmental data information of the sample vehicle.

[0104] In this embodiment, in order to construct a semantic confidence prediction model, a lot of preparatory work needs to be done in advance. First, it is necessary to collect a large number of voice commands issued by drivers in sample vehicles under various driving scenarios as sample voices, and obtain the corresponding driving data of the sample vehicles when the drivers issue voice commands, including but not limited to the sample vehicle's own state information (such as vehicle speed, gear position, vehicle fault warning, etc.) and the environmental data information of the sample vehicle (such as road conditions, weather, vehicle temperature, etc.), and use existing or future encoding methods to encode the driving data of the sample vehicles to obtain the sample feature vector corresponding to the driving data of the sample vehicles, which is used to constitute training data for executing subsequent steps (3). Among them, the specific encoding process can refer to the implementation process of extracting the target feature vector in step S101. It is only necessary to replace the driving data of the target vehicle with the driving data of the sample vehicle, which will not be repeated here.

[0105] Step (2): Obtain a sample speech uttered by a user on a sample vehicle, and determine the true semantic confidence of the text recognition result corresponding to the sample speech.

[0106] In this embodiment, after obtaining the sample speech emitted by the user (such as the driver) on the sample vehicle, each sample speech can be further recognized for the first time through the steps of preprocessing, feature extraction, processing using the acoustic model and language model, and decoding, and the text recognition results corresponding to each sample speech can be obtained to achieve speech-to-text conversion to form training data. The accurate semantic confidence (i.e., true semantic confidence) of the text recognition result corresponding to each sample speech can be manually annotated to perform the subsequent step (3). Among them, the specific process of the initial recognition of the sample speech can refer to the implementation process of the initial recognition of the target speech in step S102. It is only necessary to replace the target speech with the sample speech, and no further details are given here.

[0107] Step (3): Based on the sample feature vector, the text recognition result corresponding to the sample speech and its corresponding true semantic confidence, the initial semantic confidence prediction model is trained to generate a semantic confidence prediction model.

[0108] In this embodiment, after obtaining the sample feature vector, the text recognition result corresponding to the sample speech and its corresponding true semantic confidence, further, when performing model training, the sample feature vector corresponding to the driving data of a sample vehicle can be extracted from the training data in sequence, and the first position of the text recognition result of the corresponding sample speech (constituting global information) is inserted as the model input, and the corresponding semantic confidence prediction result is used as the output. Multiple rounds of model training are performed, and the prediction results obtained in each round of training are compared with the corresponding true semantic confidence labels. The model parameters of the initial semantic confidence prediction model are updated according to the difference between the two until the preset conditions are met, such as the value of the loss constraint function is very small and basically unchanged, then the update of the model parameters is stopped, the training of the semantic confidence prediction model is completed, and a trained semantic confidence prediction model is generated.

[0109] Among them, this application does not limit the specific composition structure of the initial semantic confidence prediction model, and different model structures can be selected according to the actual deployment terminal performance. For example, the BERT model, Transformer model or Recurrent Neural Network (RNN) model can be used as the initial semantic confidence prediction model.

[0110] For example: Figure 4 As shown, the sample feature vector corresponding to the driving data of the sample vehicle (such as Figure 4 The car_env_fuse_EmbH shown in the figure is inserted into the text recognition result of its corresponding sample speech (such as Figure 4 As shown in the figure, the first one is “turn on the lights”, which is used as global information to input the initial semantic confidence prediction model (such as Figure 4 BERT, Transformer, RNN shown in the figure), and get the text recognition results output by the model (such as Figure 4 The prediction results of the semantic confidence corresponding to “turn on the lights” shown in Figure 4 The prediction confidence is 0.95), then the prediction result can be compared with the corresponding true semantic confidence label, and the model parameters of the initial semantic confidence prediction model can be updated according to the difference between the two, so as to train the initial semantic confidence prediction model and generate a semantic confidence prediction model.

[0111] On this basis, after training and generating a semantic confidence prediction model based on the driving data of the sample vehicle and the sample speech uttered by the user on board the sample vehicle, the generated semantic confidence prediction model can be further verified using the driving data of the verification vehicle and the verification speech uttered by the user on board the verification vehicle. The specific verification process may include the following steps ①-④:

[0112] Step ①: Acquire the driving data of the verification vehicle; and extract the verification feature vector corresponding to the driving data of the verification vehicle; wherein the driving data of the verification vehicle includes the verification vehicle's own state information and the verification vehicle's environmental data information.

[0113] In this embodiment, in order to verify the semantic confidence prediction model, it is first necessary to obtain the driving data of the verification vehicle and the verification voice issued by the on-board user of the verification vehicle. For example, the voice command issued by the driver in the verification vehicle can be obtained as the verification voice, and the corresponding driving data of the verification vehicle when the voice command issued by the driver is obtained, including but not limited to the verification vehicle's own state information (such as driving speed, acceleration, engine status, etc.) and the verification vehicle's environmental data information (such as road conditions, traffic signs, whether there are pedestrians around, etc.). Among them, the driving data of the verification vehicle refers to the vehicle driving data that can be used to verify the semantic confidence prediction model. After obtaining the driving data of these verification vehicles, the driving data of the verification vehicle can be encoded using existing or future encoding methods to extract the verification feature vector corresponding to the driving data of the verification vehicle, and the subsequent step ③ can be continued.

[0114] Step 2: Obtain the verification voice uttered by the user on the verification vehicle, and determine the true semantic confidence of the text recognition result corresponding to the verification voice.

[0115] In this embodiment, after obtaining the verification voice issued by the user on board the verification vehicle (such as the driver), each verification voice can be further recognized for the first time through preprocessing, feature extraction, processing using acoustic models and language models, and decoding, and the text recognition results corresponding to each verification voice can be obtained to realize speech-to-text conversion. The accurate semantic confidence (i.e., true semantic confidence) of the text recognition result corresponding to each verification voice can be manually marked to execute the subsequent step ④.

[0116] Step 3: Input the verification feature vector and the text recognition result corresponding to the verification speech into the semantic confidence prediction model to obtain the prediction confidence of the text recognition result corresponding to the verification speech.

[0117] After extracting the verification feature vector corresponding to the driving data of the verification vehicle through step ①, the verification feature vector and the text recognition result corresponding to the verification voice can be further input into the semantic confidence prediction model to obtain the prediction confidence of the text recognition result corresponding to the verification voice, which is used to execute the subsequent step ④.

[0118] Step ④: When the prediction confidence of the text recognition result corresponding to the verification speech is inconsistent with the true semantic confidence, the driving data of the verification vehicle is used again as the driving data of the sample vehicle, and the verification speech is used again as the sample speech to update the semantic confidence prediction model.

[0119] After obtaining the predicted confidence of the text recognition result corresponding to the verification voice through step ③, if the predicted confidence is inconsistent with the true semantic confidence of the text recognition result corresponding to the verification voice determined through step ②, the driving data of the verification vehicle can be used again as the driving data of the sample vehicle, and the verification voice can be used again as the sample voice to update the parameters of the semantic confidence prediction model to further improve the prediction precision and accuracy of the semantic confidence prediction model.

[0120] S104: Using the text confidence prediction results corresponding to each of the N candidate text recognition results, perform secondary recognition on the target speech to obtain a final recognition result corresponding to the target speech.

[0121] In this embodiment, after predicting the text confidence prediction results corresponding to each of the N candidate text recognition results through step S103, the existing or future data fusion method can be further used to fuse the probability score of the initial recognition corresponding to each candidate text recognition result in the N candidate text recognition results with the text confidence prediction result, so as to perform a secondary recognition of the target speech based on the processing result. After comprehensively considering the possibility and reliability of the speech recognition results, a more accurate final recognition result corresponding to the target speech is obtained, thereby effectively improving the accuracy of the speech recognition results of the on-board users of the target vehicle, thereby enhancing the driving and riding experience of the on-board users and improving the driving safety of the target vehicle.

[0122] Specifically, an optional implementation method is to first perform a weighted sum calculation on the probability score of the initial recognition corresponding to each candidate text recognition result in the N candidate text recognition results and the text confidence prediction result to obtain the recognition score corresponding to each candidate text recognition result. The specific calculation formula is as follows:

[0123] finalScore=ω1*ASRProb+ω2*SegProb

[0124] Among them, finalScore represents the recognition score corresponding to each candidate text recognition result; ASRProb represents the probability score of the initial recognition corresponding to each candidate text recognition result; SegProb represents the text confidence prediction result corresponding to each candidate text recognition result; ω1 and ω2 represent the weight coefficients of ASRProb and SegProb respectively. The specific values ​​are not limited and can be set according to actual conditions and experience values. For example, the two can be set to 0.6 and 0.4 respectively.

[0125] In addition, other more complex fusion methods, such as fusion models based on deep learning, can also be used for fusion processing. In this way, after completing the re-scoring of each candidate text recognition result, the candidate text recognition result corresponding to the highest recognition score (finalScore) can be selected from all N re-scored candidate text recognition results as the final recognition result corresponding to the target speech.

[0126] In summary, the present embodiment provides a speech recognition method, which first obtains the driving data of the target vehicle and extracts the target feature vector corresponding to the driving data; wherein the driving data includes the target vehicle's own state information and the environmental data information of the target vehicle, and then obtains the target speech to be recognized issued by the on-board user of the target vehicle, and performs an initial recognition on the target speech to obtain N candidate text recognition results; wherein N is a positive integer greater than 1; then, the target feature vector and the N candidate text recognition results are input into a pre-constructed semantic confidence prediction model, and the text confidence prediction results corresponding to each of the N candidate text recognition results are predicted, and then the target speech is secondary recognized using the text confidence prediction results corresponding to each of the N candidate text recognition results to obtain the final recognition result corresponding to the target speech.

[0127] It can be seen that, since this application introduces driving data such as the target vehicle's own state information and the target vehicle's environmental data information as important recognition basis when recognizing the target voice emitted by the on-board user of the target vehicle, and uses the pre-built semantic confidence prediction model to accurately determine the confidence corresponding to the N candidate text recognition results obtained in the initial recognition as the screening basis, it can screen out more accurate candidate text recognition results from the N candidate text recognition results as the final recognition result corresponding to the target voice, effectively improving the recognition accuracy of the target voice, thereby enhancing the driving and riding experience of the on-board user, and improving the driving safety of the target vehicle.

[0128] Second embodiment

[0129] This embodiment will introduce a speech recognition device. For related content, please refer to the above method embodiment.

[0130] See also Figure 5 , is a schematic diagram of the composition of a speech recognition device provided in this embodiment, the device 500 includes:

[0131] The first acquisition unit 501 is used to acquire the driving data of the target vehicle and extract the target feature vector corresponding to the driving data; the driving data includes the state information of the target vehicle itself and the environmental data information of the target vehicle;

[0132] The second acquisition unit 502 is configured to acquire a target speech to be recognized uttered by a user on the target vehicle, and perform initial recognition on the target speech to obtain N candidate text recognition results, where N is a positive integer greater than 1;

[0133] The prediction unit 503 is configured to input the target feature vector and the N candidate text recognition results into a pre-built semantic confidence prediction model to predict the text confidence prediction results corresponding to the N candidate text recognition results.

[0134] The recognition unit 504 is configured to perform secondary recognition on the target speech using the text confidence prediction results corresponding to the N candidate text recognition results to obtain a final recognition result corresponding to the target speech.

[0135] In an implementation of this embodiment, the first acquiring unit 501 includes:

[0136] A first encoding subunit is used to encode the state information of the target vehicle to obtain a running state feature vector of the target vehicle;

[0137] A second encoding subunit is used to encode the environmental data information of the target vehicle to obtain an environmental information feature vector;

[0138] The fusion subunit is used to fuse the running state feature vector and the environmental information feature vector of the target vehicle to obtain a fused feature vector as the target feature vector corresponding to the driving data.

[0139] In one implementation of this embodiment, the fusion subunit is specifically configured to:

[0140] The running state feature vector and the environmental information feature vector of the target vehicle are concatenated or weightedly summed to obtain a fused feature vector as the target feature vector corresponding to the driving data.

[0141] In one implementation of this embodiment, the second acquiring unit 502 is specifically configured to:

[0142] Extracting acoustic features of the target speech; and decoding the target speech based on the acoustic features to obtain N candidate text recognition results.

[0143] In one implementation of this embodiment, the apparatus further includes:

[0144] A third acquisition unit is configured to acquire driving data of a sample vehicle and extract a sample feature vector corresponding to the driving data of the sample vehicle; the driving data of the sample vehicle includes state information of the sample vehicle itself and environmental data information of the sample vehicle;

[0145] a fourth acquiring unit, configured to acquire a sample speech uttered by a user on board the sample vehicle, and determine a true semantic confidence level of a text recognition result corresponding to the sample speech;

[0146] The training unit is used to train the initial semantic confidence prediction model according to the sample feature vector, the text recognition result corresponding to the sample speech and its corresponding true semantic confidence, so as to generate the semantic confidence prediction model.

[0147] In one implementation of this embodiment, the apparatus further includes:

[0148] a fifth acquisition unit, configured to acquire driving data of the verification vehicle; and extract a verification feature vector corresponding to the driving data of the verification vehicle; the driving data of the verification vehicle includes the state information of the verification vehicle itself and the environmental data information of the verification vehicle;

[0149] a sixth acquiring unit, configured to acquire a verification voice uttered by a user on board the verification vehicle, and determine a true semantic confidence level of a text recognition result corresponding to the verification voice;

[0150] an obtaining unit, configured to input the verification feature vector and the text recognition result corresponding to the verification speech into the semantic confidence prediction model to obtain the prediction confidence of the text recognition result corresponding to the verification speech;

[0151] An updating unit is used to update the semantic confidence prediction model by reusing the driving data of the verification vehicle as the driving data of the sample vehicle and reusing the verification speech as the sample speech when the predicted confidence of the text recognition result corresponding to the verification speech is inconsistent with the true semantic confidence.

[0152] In one implementation of this embodiment, the identification unit 504 includes:

[0153] a calculation subunit, configured to perform a weighted sum calculation on the probability score of the initial recognition corresponding to each candidate text recognition result and the text confidence prediction result of the N candidate text recognition results to obtain a recognition score corresponding to each candidate text recognition result;

[0154] The selection subunit is used to select the candidate text recognition result corresponding to the highest recognition score from the N candidate text recognition results as the final recognition result corresponding to the target speech.

[0155] Furthermore, an embodiment of the present application also provides a speech recognition device, comprising: a processor, a memory, and a system bus;

[0156] The processor and the memory are connected via the system bus;

[0157] The memory is used to store one or more programs, and the one or more programs include instructions. When the instructions are executed by the processor, the processor performs any one of the implementation methods of the above-mentioned speech recognition method.

[0158] Furthermore, an embodiment of the present application also provides a computer-readable storage medium, in which instructions are stored. When the instructions are executed on a terminal device, the terminal device executes any one of the implementation methods of the above-mentioned speech recognition method.

[0159] Furthermore, an embodiment of the present application also provides a computer program product, which, when running on a terminal device, enables the terminal device to execute any one of the implementation methods of the above-mentioned speech recognition method.

[0160] Through the description of the above embodiments, it can be seen that those skilled in the art can clearly understand that all or part of the steps in the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network communication device such as a media gateway, etc.) to execute the methods described in each embodiment of the present application or certain parts of the embodiments.

[0161] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the methods.

[0162] It should also be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0163] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A speech recognition method, characterized in that: include: Acquire driving data of a target vehicle and extract a target feature vector corresponding to the driving data; the driving data includes the target vehicle's own state information and the target vehicle's environment data information; Acquire a target speech to be recognized uttered by a user on the target vehicle, and perform initial recognition on the target speech to obtain N candidate text recognition results, where N is a positive integer greater than 1; Inputting the target feature vector and N candidate text recognition results into a pre-built semantic confidence prediction model to predict text confidence prediction results corresponding to each of the N candidate text recognition results; Using the text confidence prediction results corresponding to each of the N candidate text recognition results, perform secondary recognition on the target speech to obtain a final recognition result corresponding to the target speech; The semantic confidence prediction model is constructed as follows: Acquire driving data of a sample vehicle; and extract a sample feature vector corresponding to the driving data of the sample vehicle; the driving data of the sample vehicle includes state information of the sample vehicle itself and environmental data information of the sample vehicle; Obtaining a sample speech uttered by a user on the sample vehicle, and determining a true semantic confidence level of a text recognition result corresponding to the sample speech; An initial semantic confidence prediction model is trained according to the sample feature vector, the text recognition result corresponding to the sample speech and its corresponding true semantic confidence to generate the semantic confidence prediction model.

2. The method according to claim 1, characterized in that The extracting the target feature vector corresponding to the driving data includes: Encoding the state information of the target vehicle to obtain a running state feature vector of the target vehicle; Encoding the environmental data information of the target vehicle to obtain an environmental information feature vector; The running state feature vector and the environmental information feature vector of the target vehicle are fused to obtain a fused feature vector as the target feature vector corresponding to the driving data.

3. The method according to claim 2, characterized in that The step of fusing the target vehicle's operating state feature vector and the environmental information feature vector to obtain a fused feature vector as the target feature vector corresponding to the driving data includes: The running state feature vector and the environmental information feature vector of the target vehicle are concatenated or weightedly summed to obtain a fused feature vector as the target feature vector corresponding to the driving data.

4. The method according to claim 1, wherein The target speech is initially recognized to obtain N candidate text recognition results, including: Extracting acoustic features of the target speech; and decoding the target speech based on the acoustic features to obtain N candidate text recognition results.

5. The method according to claim 1, characterized in that The method further comprises: Acquire driving data of a verification vehicle; and extract a verification feature vector corresponding to the driving data of the verification vehicle; the driving data of the verification vehicle includes the state information of the verification vehicle itself and the environmental data information of the verification vehicle; Acquire a verification voice uttered by a user on board the verification vehicle, and determine a true semantic confidence level of a text recognition result corresponding to the verification voice; Inputting the verification feature vector and the text recognition result corresponding to the verification speech into the semantic confidence prediction model to obtain the prediction confidence of the text recognition result corresponding to the verification speech; When the predicted confidence of the text recognition result corresponding to the verification voice is inconsistent with the true semantic confidence, the driving data of the verification vehicle is re-used as the driving data of the sample vehicle, and the verification voice is re-used as the sample voice to update the semantic confidence prediction model.

6. The method according to any one of claims 1 to 5, characterized in that The method of performing secondary recognition on the target speech using the text confidence prediction results corresponding to each of the N candidate text recognition results to obtain a final recognition result corresponding to the target speech includes: Performing a weighted sum calculation on the probability score of the initial recognition corresponding to each candidate text recognition result and the text confidence prediction result of the N candidate text recognition results to obtain a recognition score corresponding to each candidate text recognition result; The candidate text recognition result corresponding to the highest recognition score is selected from the N candidate text recognition results as the final recognition result corresponding to the target speech.

7. A speech recognition device, characterized in that: include: A first acquisition unit is configured to acquire driving data of a target vehicle and extract a target feature vector corresponding to the driving data; the driving data includes state information of the target vehicle and environmental data information of the target vehicle; a second acquiring unit, configured to acquire a target speech to be recognized uttered by a user on board the target vehicle, and perform initial recognition on the target speech to obtain N candidate text recognition results; wherein N is a positive integer greater than 1; A prediction unit, configured to input the target feature vector and N candidate text recognition results into a pre-built semantic confidence prediction model, and predict text confidence prediction results corresponding to each of the N candidate text recognition results; A recognition unit, configured to perform secondary recognition on the target speech using the text confidence prediction results corresponding to each of the N candidate text recognition results to obtain a final recognition result corresponding to the target speech; The device further comprises: A third acquisition unit is configured to acquire driving data of a sample vehicle and extract a sample feature vector corresponding to the driving data of the sample vehicle; the driving data of the sample vehicle includes state information of the sample vehicle itself and environmental data information of the sample vehicle; a fourth acquiring unit, configured to acquire a sample speech uttered by a user on board the sample vehicle, and determine a true semantic confidence level of a text recognition result corresponding to the sample speech; The training unit is used to train the initial semantic confidence prediction model according to the sample feature vector, the text recognition result corresponding to the sample speech and its corresponding true semantic confidence, so as to generate the semantic confidence prediction model.

8. A speech recognition device, characterized in that: include: Processor, memory, system bus; The processor and the memory are connected via the system bus; The memory is configured to store one or more programs, wherein the one or more programs include instructions, and when the instructions are executed by the processor, the processor is enabled to perform the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores instructions, and when the instructions are executed on a terminal device, the terminal device executes the method according to any one of claims 1 to 6.

Images