Speech recognition method and device, speech recognition equipment and storage medium

Abstract

The invention relates to a voice recognition method and device, voice recognition equipment and a storage medium, and the method comprises the steps: obtaining a voice signal inputted by a user, and determining the features of the voice signal based on the voice signal; acquiring voice recognition influence data corresponding to a plurality of preset voice recognition mode influence factors; determining a target voice recognition mode in a plurality of preset candidate voice modes according to the plurality of pieces of voice recognition influence data; and performing voice recognition on the voice signal features by using the target voice recognition mode to obtain a voice recognition result. According to the embodiment of the invention, the multiple voice recognition mode influence factors are comprehensively considered to adapt to variable application scenes, and the target voice recognition modes matched with the different application scenes are determined for the different application scenes to perform voice recognition, so that quick and accurate voice response is provided, delay and misrecognition are reduced, a satisfactory voice recognition result is provided for a user, and the user experience is improved. And the user satisfaction of the smart home equipment is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a speech recognition method, apparatus, speech recognition device and storage medium. Background Art

[0002] With the rapid development of the Internet of Things and artificial intelligence technologies, smart homes have become an important part of modern family life. Smart air conditioners, smart lighting, smart security and other devices provide users with convenient control and services through voice interaction.

[0003] However, existing speech recognition systems cannot adapt to changing home environments and user needs, and cannot provide users with satisfactory speech recognition results. Summary of the invention

[0004] In order to solve the above technical problems or at least partially solve the above technical problems, the present application provides a speech recognition method, apparatus, speech recognition device and storage medium.

[0005] In a first aspect, the present application provides a speech recognition method, comprising:

[0006] Acquire a voice signal input by a user, and determine a voice signal feature based on the voice signal;

[0007] Acquire speech recognition influence data corresponding to a plurality of preset speech recognition mode influence factors;

[0008] Determining a target speech recognition mode from a plurality of preset candidate speech modes according to the plurality of speech recognition influencing data;

[0009] The target speech recognition mode is used to perform speech recognition on the speech signal feature to obtain a speech recognition result.

[0010] Optionally, determining a target speech recognition mode from a plurality of preset candidate speech modes according to the plurality of speech recognition influencing data comprises:

[0011] Determining an optimal mode selection variable according to the plurality of speech recognition influence data and a comprehensive utility function of each speech recognition mode influence factor;

[0012] The target speech recognition mode is determined according to the optimal mode selection variable.

[0013] Optionally, determining the optimal mode selection variable according to the plurality of speech recognition influence data and the comprehensive utility function of each speech recognition mode influence factor comprises:

[0014] Inputting each of the speech recognition impact data into the comprehensive utility function;

[0015] The optimal mode selection variable that maximizes the comprehensive utility function is calculated.

[0016] Optionally, determining the target speech recognition mode according to the optimal mode selection variable includes:

[0017] Obtain constraints considered when selecting a speech recognition mode;

[0018] The target speech recognition mode is determined according to the optimal mode selection variables and the constraint conditions.

[0019] Optionally, determining the target speech recognition mode according to the optimal mode selection variable and the constraint condition includes:

[0020] If the optimal mode selection variable corresponds to the offline mode and the computing power of the device is greater than or equal to the minimum computing power required for the offline mode, determining that the target speech recognition mode is the offline mode;

[0021] If the optimal mode selection variable corresponds to the online mode and the current network bandwidth is greater than or equal to the minimum network bandwidth required for the online mode, determining that the target speech recognition mode is the online mode;

[0022] Otherwise, it is determined that the target speech recognition mode is a mixed mode.

[0023] Optionally, when the target speech recognition mode is a mixed mode, performing speech recognition on the speech signal feature using the target speech recognition mode to obtain a speech recognition result includes:

[0024] Performing offline speech recognition and online speech recognition on the speech signal features respectively to obtain offline recognition results, offline recognition confidence, online recognition results and online recognition confidence;

[0025] Normalizing the offline recognition confidence and the online recognition confidence to obtain an offline recognition confidence weight and an online recognition confidence weight;

[0026] Obtaining a first offline prediction probability of each instruction feature in the speech signal feature predicted by an offline speech recognition model; and a first online prediction probability of each instruction feature in the speech signal feature predicted by an online speech recognition model;

[0027] For each instruction feature in the speech signal feature, calculating a fusion probability based on the first offline prediction probability, the first online prediction probability, the offline recognition confidence weight, and the online recognition confidence weight;

[0028] If the offline recognition result and the online recognition result partially overlap, the word sequence with the maximum fusion probability is selected from the offline recognition result and the online recognition result as the speech recognition result.

[0029] Optionally, when the target speech recognition mode is a mixed mode, performing speech recognition on the speech signal feature using the target speech recognition mode to obtain a speech recognition result further includes:

[0030] If there is no partial overlap between the offline recognition result and the online recognition result, obtaining a second offline prediction probability of the offline speech recognition model predicting each instruction feature in the speech signal feature; and a second online prediction probability of the online speech recognition model predicting each instruction feature in the speech signal feature;

[0031] Calculate a joint posterior probability based on the second offline prediction probability and the second online prediction probability;

[0032] calculating a log probability based on the joint posterior probability;

[0033] Among the offline recognition results and the online recognition results, a word sequence that maximizes the sum of logarithmic probabilities is selected as the final result.

[0034] In a second aspect, the present application provides a speech recognition device, comprising:

[0035] A first acquisition module, used to acquire a voice signal input by a user, and determine a voice signal feature based on the voice signal;

[0036] A second acquisition module is used to acquire speech recognition influence data corresponding to a plurality of preset speech recognition mode influence factors;

[0037] A first determination module, configured to determine a target speech recognition mode from a plurality of preset candidate speech modes according to a plurality of the speech recognition influencing data;

[0038] The speech recognition module is used to perform speech recognition on the speech signal features using the target speech recognition mode to obtain a speech recognition result.

[0039] Optionally, the first determining module includes:

[0040] A first determination submodule, configured to determine an optimal mode selection variable according to a plurality of the speech recognition influencing data and a comprehensive utility function of each of the speech recognition mode influencing factors;

[0041] The second determination submodule is used to determine the target speech recognition mode according to the optimal mode selection variable.

[0042] Optionally, the first determining submodule includes:

[0043] An input unit, used for inputting each of the speech recognition impact data into the comprehensive utility function;

[0044] A calculation unit is used to calculate the optimal mode selection variable that maximizes the comprehensive utility function.

[0045] Optionally, the second determining submodule includes:

[0046] An acquisition unit, used for acquiring constraint conditions considered when selecting a speech recognition mode;

[0047] A determination unit is used to determine the target speech recognition mode according to the optimal mode selection variable and the constraint condition.

[0048] Optionally, the determining unit includes:

[0049] A first determination subunit, configured to determine that the target speech recognition mode is the offline mode if the optimal mode selection variable corresponds to the offline mode and the computing power of the device is greater than or equal to the minimum computing power required for the offline mode;

[0050] A second determining subunit, configured to determine that the target speech recognition mode is the online mode if the optimal mode selection variable corresponds to the online mode and the current network bandwidth is greater than or equal to the minimum network bandwidth required for the online mode;

[0051] The third determination subunit is used to determine that the target speech recognition mode is a mixed mode if the optimal mode selection variable corresponds to an offline mode and the computing power of the device is less than the minimum computing power required for the offline mode, and / or if the optimal mode selection variable corresponds to an online mode and the current network bandwidth is less than the minimum network bandwidth required for the online mode.

[0052] Optionally, when the target speech recognition mode is a mixed mode, the speech recognition module includes:

[0053] The recognition submodule is used to perform offline speech recognition and online speech recognition on the speech signal features to obtain offline recognition results, offline recognition confidence, online recognition results and online recognition confidence;

[0054] A normalization processing submodule, used to perform normalization processing on the offline recognition confidence and the online recognition confidence to obtain an offline recognition confidence weight and an online recognition confidence weight;

[0055] A first acquisition submodule is used to acquire a first offline prediction probability of each instruction feature in the speech signal feature predicted by the offline speech recognition model; and a first online prediction probability of each instruction feature in the speech signal feature predicted by the online speech recognition model;

[0056] A first calculation submodule, configured to calculate a fusion probability for each instruction feature in the speech signal feature based on the first offline prediction probability, the first online prediction probability, the offline recognition confidence weight, and the online recognition confidence weight;

[0057] The first selection submodule is used to select a word sequence with the maximum fusion probability as the speech recognition result from the offline recognition result and the online recognition result if there is a partial overlap between the offline recognition result and the online recognition result.

[0058] Optionally, when the target speech recognition mode is a mixed mode, the speech recognition module further includes:

[0059] A second acquisition submodule is used to obtain, if there is no partial overlap between the offline recognition result and the online recognition result, a second offline prediction probability predicted by the offline speech recognition model for each instruction feature in the speech signal feature; and a second online prediction probability predicted by the online speech recognition model for each instruction feature in the speech signal feature;

[0060] A second calculation submodule, configured to calculate a joint posterior probability based on the second offline prediction probability and the second online prediction probability;

[0061] A third calculation submodule, configured to calculate a logarithmic probability based on the joint posterior probability;

[0062] The second selection submodule is used to select a word sequence that maximizes the sum of logarithmic probabilities from the offline recognition results and the online recognition results as a final result.

[0063] In a third aspect, the present application provides a speech recognition device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0064] Memory, used to store computer programs;

[0065] The processor is used to implement any speech recognition method described in the first aspect when executing the program stored in the memory.

[0066] In a fourth aspect, the present application provides a computer-readable storage medium, on which a program of a speech recognition method is stored. When the program of the speech recognition method is executed by a processor, the steps of the speech recognition method described in any one of the first aspects are implemented.

[0067] The above technical solution provided by the embodiment of the present application has the following advantages compared with the prior art:

[0068] The embodiments of the present application can determine a target speech recognition mode from multiple preset candidate speech modes based on speech recognition influence data corresponding to multiple preset speech recognition mode influence factors, and use the target speech recognition mode to perform speech recognition on speech signal features. It can adapt to changing application scenarios by comprehensively considering multiple speech recognition mode influence factors, determine target speech recognition modes that match different application scenarios for speech recognition, and thus provide fast and accurate speech response, reduce delays and misrecognition, provide user-satisfied speech recognition results, and improve user satisfaction with smart home devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0070] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0071] Figure 1 A flow chart of a speech recognition method provided in an embodiment of the present application;

[0072] Figure 2 An overall flow chart of a speech recognition method provided in an embodiment of the present application;

[0073] Figure 3 A structural diagram of a speech recognition device provided in an embodiment of the present application;

[0074] Figure 4 A structural diagram of a speech recognition device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0075] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0076] Since the existing speech recognition system cannot adapt to the changing home environment and user needs, it cannot provide users with satisfactory speech recognition results. To this end, the embodiments of the present application provide a speech recognition method, apparatus, speech recognition device and storage medium suitable for smart home, which can intelligently switch between offline, online and hybrid modes according to the real-time environment and user needs, and improve the accuracy and response speed of speech recognition through the result fusion algorithm, so as to meet the high requirements of smart home devices for speech interaction.

[0077] The present application embodiment provides a speech recognition method, which can be applied to a speech recognition device. The speech recognition device can be applied to a smart home scenario, and mainly includes:

[0078] 1. Voice input module: installed on various smart home devices, receives user voice commands, and performs preliminary signal processing, such as noise reduction and preprocessing.

[0079] 2. Offline speech recognition module:

[0080] Local recognition model: For commonly used home control commands, a lightweight offline speech recognition model is pre-set, which can quickly process basic commands such as turning on and off lights and adjusting the temperature.

[0081] Feature extraction and semantic understanding: Extract features from speech signals locally and perform preliminary semantic understanding to support simple natural language instructions.

[0082] 3. Online speech recognition module:

[0083] Cloud recognition service: For complex voice commands and advanced function requests (such as playing specific music and checking the weather), voice data is uploaded to the cloud and recognized and processed using powerful computing resources.

[0084] Model update and learning: Cloud models continuously learn and update to support the latest speech recognition technology and user habits.

[0085] 4. Dynamic switching module:

[0086] Environmental monitoring unit: real-time monitoring of network status (bandwidth B, latency), equipment performance (processing capacity C), power and other system parameters.

[0087] Strategy decision unit: Based on the comprehensive utility function U(m), calculate the optimal mode selection variable m * , determines whether the system operates in offline, online or hybrid mode.

[0088] 5. Result fusion module:

[0089] Confidence evaluation unit: evaluates the confidence C of offline and online recognition results offline and C online .

[0090] Fusion algorithm unit: uses a fusion algorithm based on confidence weighting and Bayesian theory to fuse offline and online recognition results and generate the final execution instructions.

[0091] Device collaboration module: Smart devices share recognition results and environmental information through the home LAN to improve overall recognition accuracy and response speed.

[0092] like Figure 1 As shown, the speech recognition method provided in the embodiment of the present application may include the following steps:

[0093] Step S101, obtaining a voice signal input by a user, and determining a voice signal feature based on the voice signal;

[0094] The user inputs a voice signal S(t) through the microphone of the smart home device, and the device performs pre-processing on the voice signal such as noise reduction and removal of silent segments to obtain the processed voice signal S ′ (t), for S ′ (t) Perform feature extraction to obtain speech signal features X, such as Mel-frequency cepstral coefficients (MFCC), spectrogram, etc.

[0095] Step S102, obtaining speech recognition influence data corresponding to a plurality of preset speech recognition mode influence factors;

[0096] In a smart home environment, the system needs to strike a balance between recognition accuracy, response delay, energy consumption, and privacy protection. In the embodiment of the present application, the speech recognition mode influencing factor may refer to a factor that affects the selection of the speech recognition mode, and the multiple speech recognition mode influencing factors may include: recognition accuracy, response delay, energy consumption, and privacy protection.

[0097] Identification accuracy (A(m)): depends on network bandwidth (B) and device performance (C); response delay (D(m)): affected by network delay and device processing capability; energy consumption (E(m)): mainly determined by device performance (C) and power status; privacy protection level (P(m)): affected by user privacy preference and network status.

[0098] Network status (network bandwidth B, response delay): can be obtained by real-time monitoring of network bandwidth and delay; device performance (C): obtained by detecting the processing power of the device; power: can be obtained through the built-in power sensor of the device; user privacy preference: selected by the user in the settings.

[0099] Step S103, determining a target speech recognition mode from a plurality of preset candidate speech modes according to the plurality of speech recognition influencing data;

[0100] In the embodiment of the present application, the multiple preset candidate voice modes include: an online voice recognition mode, an offline voice recognition mode and a hybrid voice recognition mode.

[0101] In this step, the speech recognition influence data corresponding to multiple preset speech recognition mode influence factors can be comprehensively considered to determine the target speech recognition mode from the preset candidate speech modes.

[0102] Step S104: performing speech recognition on the speech signal features using the target speech recognition mode to obtain a speech recognition result.

[0103] In this step, the target speech recognition mode (such as online speech recognition mode, offline speech recognition mode or hybrid speech recognition mode) determined in step S103 may be used to perform speech recognition on the speech signal features to obtain a speech recognition result.

[0104] The embodiments of the present application can determine a target speech recognition mode from multiple preset candidate speech modes based on speech recognition influence data corresponding to multiple preset speech recognition mode influence factors, and use the target speech recognition mode to perform speech recognition on speech signal features. It can adapt to changing application scenarios by comprehensively considering multiple speech recognition mode influence factors, determine target speech recognition modes that match different application scenarios for speech recognition, and thus provide fast and accurate speech response, reduce delays and misrecognition, provide user-satisfied speech recognition results, and improve user satisfaction with smart home devices.

[0105] In yet another embodiment of the present application, determining a target speech recognition mode from a plurality of preset candidate speech modes according to a plurality of the speech recognition influencing data includes:

[0106] Step S201, determining an optimal mode selection variable according to a plurality of the speech recognition influencing data and a comprehensive utility function of the influencing factors of each speech recognition mode;

[0107] In an implementation manner of the present application, determining an optimal mode selection variable according to the multiple voice recognition impact data and a comprehensive utility function regarding each voice recognition mode impact factor includes: inputting each piece of the voice recognition impact data into the comprehensive utility function; calculating the optimal mode selection variable that maximizes the comprehensive utility function.

[0108] In the embodiment of the present application, the comprehensive utility function U(m):

[0109] U(m) = w1A(m) - w2D(m) - w3E(m) + w4P(m)

[0110] Where: A(m): Recognition accuracy rate, defined as:

[0111] A(m) = (1 - m)A offline + mA online

[0112] D(m): Response delay, defined as:

[0113] D(m) = (1 - m)D offline + mD online

[0114] E(m): Energy consumption, defined as:

[0115] E(m) = (1 - m)E offline + mE online

[0116] P(m): Degree of privacy protection, defined as:

[0117] P(m) = (1 - m)P offline + mP online

[0118] m ∈ [0,1]: Mode selection variable, m = 0 represents the offline mode, m = 1 represents the online mode, and 0 < m < 1 represents the hybrid mode.

[0119] w1, w2, w3, w4: Weight coefficients, set according to the smart home application scenario and user preferences.

[0120] The goal is to solve the optimal mode selection variable m * , to maximize the comprehensive utility function U(m):

[0121]

[0122] In practical applications, due to the limited computing resources of smart home devices, m can be discretized into a finite number of values (for example, m = 0, 0.5, 1), corresponding to the offline, hybrid, and online modes.

[0123] Step S202: Determine the target speech recognition mode according to the optimal mode selection variable.

[0124] In one embodiment of the present application, determining the target speech recognition mode according to the optimal mode selection variable includes: obtaining constraints considered when selecting the speech recognition mode; and determining the target speech recognition mode according to the optimal mode selection variable and the constraints.

[0125] In the present embodiment, the following constraints are considered:

[0126] Network bandwidth constraint: B≥B min , where B is the current network bandwidth, B min The minimum bandwidth required for online mode.

[0127] Equipment processing capacity constraint: C ≥ C min , where C is the computing power of the device, C min Minimum computing power required for offline mode.

[0128] User privacy preferences: Users can set privacy preferences user , weight w4 that affects the privacy protection level P(m).

[0129] Wherein, determining the target speech recognition mode according to the optimal mode selection variable and the constraint condition includes:

[0130] If the optimal mode selection variable corresponds to the offline mode and the computing power of the device is greater than or equal to the minimum computing power required for the offline mode, determining that the target speech recognition mode is the offline mode;

[0131] If the optimal mode selection variable corresponds to the online mode and the current network bandwidth is greater than or equal to the minimum network bandwidth required for the online mode, determining that the target speech recognition mode is the online mode;

[0132] Otherwise, it is determined that the target speech recognition mode is a mixed mode.

[0133] That is, if m * = 0 and satisfies C ≥ C min , then select offline mode; if m * =1 and B≥B min , select online mode; otherwise, select hybrid mode.

[0134] The embodiment of the present application realizes the collaborative work between multiple smart home devices, and improves the intelligence level of the overall system by sharing recognition results and environmental information; constructs a comprehensive utility function U(m), quantifies performance indicators such as recognition accuracy, response delay, energy consumption and privacy protection, and dynamically solves the optimal mode selection variable according to the real-time monitored network status, device performance and user preferences, and realizes intelligent switching between offline, online and hybrid modes; it can intelligently select offline, online or hybrid modes according to the home network status, device performance and user preferences to ensure the stability and reliability of voice recognition. In the home scenario, protecting user privacy is crucial. When the network is unstable or the user emphasizes privacy, the system gives priority to using offline mode to prevent voice data leakage.

[0135] In another embodiment of the present application, when the target speech recognition mode is a hybrid mode, the system needs to fuse offline and online recognition results to improve recognition accuracy. Step S104 uses the target speech recognition mode to perform speech recognition on the speech signal features to obtain a speech recognition result, including:

[0136] Step S301, performing offline speech recognition and online speech recognition on the speech signal features respectively, and obtaining an offline recognition result, an offline recognition confidence, an online recognition result and an online recognition confidence;

[0137] In the embodiment of the present application, the offline recognition confidence is: C offline , provided by the offline model, reflecting the reliability of the local recognition results.

[0138] Online recognition confidence: C online , provided by the online model, reflecting the reliability of the cloud recognition results.

[0139] Step S302, normalizing the offline recognition confidence and the online recognition confidence to obtain an offline recognition confidence weight and an online recognition confidence weight;

[0140] Offline recognition confidence weight:

[0141]

[0142] Online recognition confidence weight:

[0143]

[0144] Step S303, obtaining a first offline prediction probability of each instruction feature in the speech signal feature predicted by the offline speech recognition model; and a first online prediction probability of each instruction feature in the speech signal feature predicted by the online speech recognition model;

[0145] Step S304, for each instruction feature in the speech signal feature, calculating a fusion probability based on the first offline prediction probability, the first online prediction probability, the offline recognition confidence weight, and the online recognition confidence weight;

[0146] For each word or command w i , calculate the fusion probability:

[0147]

[0148] Where: P offline (w i ): Offline model for w i The predicted probability, P onlune (w i ): Online model for w i The predicted probability of .

[0149] Step S305: If the offline recognition result and the online recognition result partially overlap, a word sequence with the maximum fusion probability is selected from the offline recognition result and the online recognition result as the speech recognition result.

[0150] Select the word sequence that maximizes the fusion probability as the final result:

[0151]

[0152] The embodiment of the present application proposes a confidence-based result fusion algorithm to perform weighted fusion of offline and online recognition results, integrate the offline and online recognition results, consider the semantic characteristics of smart home commands, improve the recognition accuracy of voice commands, and support more complex home control operations.

[0153] In another embodiment of the present application, when the target speech recognition mode is a mixed mode, step S104 uses the target speech recognition mode to perform speech recognition on the speech signal feature to obtain a speech recognition result, and further includes:

[0154] Step S401, if there is no partial overlap between the offline recognition result and the online recognition result, obtaining a second offline prediction probability of the offline speech recognition model predicting each instruction feature in the speech signal feature; and a second online prediction probability of the online speech recognition model predicting each instruction feature in the speech signal feature;

[0155] Step S402, calculating a joint posterior probability based on the second offline prediction probability and the second online prediction probability;

[0156] Assuming that the offline and online recognition results are independent, the joint posterior probability is:

[0157] P(w i |S)=P off1ine (w i |S)×P online (w i |S)

[0158] Step S403, calculating the logarithmic probability based on the joint posterior probability;

[0159] To avoid numerical underflow, logarithmic probability accumulation is calculated using logarithmic form:

[0160] log P(w i |S)=log P offline (w i |S)+log P online (w i |S)

[0161] Step S404, selecting a word sequence that maximizes the sum of logarithmic probabilities from the offline recognition results and the online recognition results as the final result.

[0162] Select the word sequence that maximizes the logarithmic probability as the final result:

[0163]

[0164] The embodiment of the present application proposes a result fusion algorithm based on Bayesian theory to perform weighted fusion of offline and online recognition results, consider the semantic features of smart home commands, improve the recognition accuracy of voice commands, and support more complex home control operations.

[0165] For ease of understanding, the present application also provides an overall embodiment, such as Figure 2 As shown, the workflow is described as follows:

[0166] Step 1: Speech Input and Preprocessing

[0167] The user inputs the voice signal S(t) through the microphone of the smart home device, and the device performs pre-processing on the voice signal such as noise reduction and removal of silent segments to obtain the processed voice signal S′(t).

[0168] Step 2: Feature Extraction

[0169] Perform feature extraction on S′(t) to obtain acoustic feature parameters X, such as Mel-frequency cepstral coefficients (MFCC), spectrogram, etc.

[0170] Step 3: Mode Selection

[0171] The dynamic switching module is based on the real-time monitoring of network bandwidth B, device computing power C and user privacy preference P.user , using the comprehensive utility function U(m) to calculate the optimal mode selection variable m * , decide to adopt offline, online or hybrid mode.

[0172] Step 4: Speech Recognition Processing

[0173] Offline mode: The offline speech recognition module processes the feature parameter X and generates an offline recognition result R offline and its confidence C offline .

[0174] Online mode: The online speech recognition module uploads X to the cloud, and the cloud model generates the online recognition result R online and its confidence C online .

[0175] Hybrid mode: Offline and online recognition modules process X simultaneously and generate R respectively. offline , C offline and R online , C online .

[0176] Step 5: Result Fusion

[0177] The result fusion module uses the above fusion algorithm to fusion R offline and R online Fusion is performed to obtain the final recognition result R final .

[0178] Step 6: Instruction execution and feedback

[0179] Instruction execution: According to R final , the system controls the corresponding smart home devices to perform operations, such as turning on and off lights, adjusting the temperature, etc.

[0180] User feedback: Feedback execution results to users through voice, display or indicator light.

[0181] Step 7: Device Collaboration and Learning

[0182] Information sharing: Each smart device shares recognition results and environmental information to improve the overall performance of the system.

[0183] Model update: Update offline model parameters based on user feedback and usage habits to improve recognition accuracy.

[0184] For ease of understanding, this application also provides the following two application examples:

[0185] Example 1: Intelligent lighting control

[0186] The user said: "Turn on the living room lights."

[0187] Mode selection: sufficient network bandwidth, good device computing power, and the system calculates m * =0.5, select the blending mode.

[0188] Voice Recognition:

[0189] Offline recognition results: R offline = "Turn on the lights in the living room", C offline =0.8.

[0190] Online recognition result: R online = "Turn on the living room lights", C online =0.9.

[0191] Through the fusion algorithm, the final recognition result is "turn on the lights in the living room".

[0192] Command execution: The system turns on the lights in the living room and provides feedback to the user that the operation was successful.

[0193] Example 2: Playing music

[0194] User said: "Play my favorite music."

[0195] Mode selection: The command is complex (semantically unclear (no specific songs or playlists), and requires more background information (such as user music preferences) to correctly identify and execute), and the system calculates m * =1, select the online mode, which can provide higher accuracy because it can access cloud data and perform deeper semantic understanding.

[0196] Speech recognition: Online recognition module generates results R online and its confidence level C online .

[0197] Command execution: The system plays the user's favorite music based on the recognition results and provides feedback on the operation results.

[0198] In another embodiment of the present application, a speech recognition device is also provided, such as Figure 3 As shown, including:

[0199] A first acquisition module 11 is used to acquire a voice signal input by a user and determine a voice signal feature based on the voice signal;

[0200] A second acquisition module 12 is used to acquire speech recognition influence data corresponding to a plurality of preset speech recognition mode influence factors;

[0201] A first determination module 13, configured to determine a target speech recognition mode from a plurality of preset candidate speech modes according to the plurality of speech recognition influencing data;

[0202] The speech recognition module 14 is used to perform speech recognition on the speech signal features using the target speech recognition mode to obtain a speech recognition result.

[0203] Optionally, the first determining module includes:

[0204] A first determination submodule, configured to determine an optimal mode selection variable according to a plurality of the speech recognition influencing data and a comprehensive utility function of each of the speech recognition mode influencing factors;

[0205] The second determination submodule is used to determine the target speech recognition mode according to the optimal mode selection variable.

[0206] Optionally, the first determining submodule includes:

[0207] An input unit, used for inputting each of the speech recognition impact data into the comprehensive utility function;

[0208] A calculation unit is used to calculate the optimal mode selection variable that maximizes the comprehensive utility function.

[0209] Optionally, the second determining submodule includes:

[0210] An acquisition unit, used for acquiring constraint conditions considered when selecting a speech recognition mode;

[0211] A determination unit is used to determine the target speech recognition mode according to the optimal mode selection variable and the constraint condition.

[0212] Optionally, the determining unit includes:

[0213] A first determination subunit, configured to determine that the target speech recognition mode is the offline mode if the optimal mode selection variable corresponds to the offline mode and the computing power of the device is greater than or equal to the minimum computing power required for the offline mode;

[0214] A second determining subunit, configured to determine that the target speech recognition mode is the online mode if the optimal mode selection variable corresponds to the online mode and the current network bandwidth is greater than or equal to the minimum network bandwidth required for the online mode;

[0215] The third determination subunit is used to determine that the target speech recognition mode is a mixed mode if the optimal mode selection variable corresponds to an offline mode and the computing power of the device is less than the minimum computing power required for the offline mode, and / or if the optimal mode selection variable corresponds to an online mode and the current network bandwidth is less than the minimum network bandwidth required for the online mode.

[0216] Optionally, when the target speech recognition mode is a mixed mode, the speech recognition module includes:

[0217] The recognition submodule is used to perform offline speech recognition and online speech recognition on the speech signal features to obtain offline recognition results, offline recognition confidence, online recognition results and online recognition confidence;

[0218] A normalization processing submodule, used to perform normalization processing on the offline recognition confidence and the online recognition confidence to obtain an offline recognition confidence weight and an online recognition confidence weight;

[0219] A first acquisition submodule is used to acquire a first offline prediction probability of each instruction feature in the speech signal feature predicted by the offline speech recognition model; and a first online prediction probability of each instruction feature in the speech signal feature predicted by the online speech recognition model;

[0220] A first calculation submodule, configured to calculate a fusion probability for each instruction feature in the speech signal feature based on the first offline prediction probability, the first online prediction probability, the offline recognition confidence weight, and the online recognition confidence weight;

[0221] The first selection submodule is used to select a word sequence with the maximum fusion probability as the speech recognition result from the offline recognition result and the online recognition result if there is a partial overlap between the offline recognition result and the online recognition result.

[0222] Optionally, when the target speech recognition mode is a mixed mode, the speech recognition module further includes:

[0223] A second acquisition submodule is used to obtain, if there is no partial overlap between the offline recognition result and the online recognition result, a second offline prediction probability predicted by the offline speech recognition model for each instruction feature in the speech signal feature; and a second online prediction probability predicted by the online speech recognition model for each instruction feature in the speech signal feature;

[0224] A second calculation submodule, configured to calculate a joint posterior probability based on the second offline prediction probability and the second online prediction probability;

[0225] A third calculation submodule, configured to calculate a logarithmic probability based on the joint posterior probability;

[0226] The second selection submodule is used to select a word sequence that maximizes the sum of logarithmic probabilities from the offline recognition results and the online recognition results as a final result.

[0227] In another embodiment of the present application, a speech recognition device is provided, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0228] Memory, used to store computer programs;

[0229] The processor is used to implement the speech recognition method described in any of the above method embodiments when executing the program stored in the memory.

[0230] In the speech recognition device provided by the embodiment of the present invention, the processor determines the target speech recognition mode from multiple preset candidate speech modes according to the speech recognition influence data corresponding to multiple preset speech recognition mode influence factors by executing the program stored in the memory, and uses the target speech recognition mode to perform speech recognition on the speech signal features. It can adapt to the changing application scenarios by comprehensively considering multiple speech recognition mode influence factors, determine the target speech recognition mode that matches the different application scenarios for speech recognition, and then provide fast and accurate speech response, reduce delays and misrecognition, provide users with satisfactory speech recognition results, and improve user satisfaction with smart home devices.

[0231] The communication bus 1140 mentioned in the above-mentioned speech recognition device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The communication bus 1140 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0232] The communication interface 1120 is used for communication between the above-mentioned speech recognition device and other devices.

[0233] The memory 1130 may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0234] The above-mentioned processor 1110 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0235] In another embodiment of the present application, a computer-readable storage medium is provided, on which a program of a speech recognition method is stored. When the program of the speech recognition method is executed by a processor, the steps of the speech recognition method described in any of the aforementioned method embodiments are implemented.

[0236] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0237] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A speech recognition method, characterized in that: include: Acquire a voice signal input by a user, and determine a voice signal feature based on the voice signal; Acquire speech recognition influence data corresponding to a plurality of preset speech recognition mode influence factors; Determining a target speech recognition mode from a plurality of preset candidate speech modes according to the plurality of speech recognition influencing data; The target speech recognition mode is used to perform speech recognition on the speech signal feature to obtain a speech recognition result.

2. The speech recognition method according to claim 1, characterized in that: Determining a target speech recognition mode from a plurality of preset candidate speech modes according to the plurality of speech recognition influencing data comprises: Determining an optimal mode selection variable according to the plurality of speech recognition influence data and a comprehensive utility function of each speech recognition mode influence factor; The target speech recognition mode is determined according to the optimal mode selection variable.

3. The speech recognition method according to claim 2, characterized in that: Determining the optimal mode selection variable according to the plurality of speech recognition impact data and the comprehensive utility function of the impact factors of each speech recognition mode includes: Inputting each of the speech recognition impact data into the comprehensive utility function; The optimal mode selection variable that maximizes the comprehensive utility function is calculated.

4. The speech recognition method according to claim 2, characterized in that: Determining the target speech recognition mode according to the optimal mode selection variable includes: Obtain constraints considered when selecting a speech recognition mode; The target speech recognition mode is determined according to the optimal mode selection variables and the constraint conditions.

5. The speech recognition method according to claim 4, characterized in that: Determining the target speech recognition mode according to the optimal mode selection variable and the constraint condition includes: If the optimal mode selection variable corresponds to the offline mode and the computing power of the device is greater than or equal to the minimum computing power required for the offline mode, determining that the target speech recognition mode is the offline mode; If the optimal mode selection variable corresponds to the online mode and the current network bandwidth is greater than or equal to the minimum network bandwidth required for the online mode, determining that the target speech recognition mode is the online mode; Otherwise, it is determined that the target speech recognition mode is a mixed mode.

6. The speech recognition method according to claim 1, characterized in that: When the target speech recognition mode is a mixed mode, speech recognition is performed on the speech signal feature using the target speech recognition mode to obtain a speech recognition result, including: Performing offline speech recognition and online speech recognition on the speech signal features respectively to obtain offline recognition results, offline recognition confidence, online recognition results and online recognition confidence; Normalizing the offline recognition confidence and the online recognition confidence to obtain an offline recognition confidence weight and an online recognition confidence weight; Obtaining a first offline prediction probability of each instruction feature in the speech signal feature predicted by an offline speech recognition model; and a first online prediction probability of each instruction feature in the speech signal feature predicted by an online speech recognition model; For each instruction feature in the speech signal feature, calculating a fusion probability based on the first offline prediction probability, the first online prediction probability, the offline recognition confidence weight, and the online recognition confidence weight; If the offline recognition result and the online recognition result partially overlap, the word sequence with the maximum fusion probability is selected from the offline recognition result and the online recognition result as the speech recognition result.

7. The speech recognition method according to claim 6, characterized in that: In the case where the target speech recognition mode is a mixed mode, performing speech recognition on the speech signal feature using the target speech recognition mode to obtain a speech recognition result, further comprising: If there is no partial overlap between the offline recognition result and the online recognition result, obtaining a second offline prediction probability of the offline speech recognition model predicting each instruction feature in the speech signal feature; and a second online prediction probability of the online speech recognition model predicting each instruction feature in the speech signal feature; Calculate a joint posterior probability based on the second offline prediction probability and the second online prediction probability; calculating a log probability based on the joint posterior probability; Among the offline recognition results and the online recognition results, a word sequence that maximizes the sum of logarithmic probabilities is selected as the final result.

8. A speech recognition device, characterized in that: include: A first acquisition module, used to acquire a voice signal input by a user, and determine a voice signal feature based on the voice signal; A second acquisition module is used to acquire speech recognition influence data corresponding to a plurality of preset speech recognition mode influence factors; A first determination module, configured to determine a target speech recognition mode from a plurality of preset candidate speech modes according to a plurality of the speech recognition influencing data; The speech recognition module is used to perform speech recognition on the speech signal features using the target speech recognition mode to obtain a speech recognition result.

9. A speech recognition device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; The processor is used to implement the speech recognition method described in any one of claims 1 to 7 when executing the program stored in the memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program of the speech recognition method, and when the program of the speech recognition method is executed by the processor, the steps of the speech recognition method described in any one of claims 1 to 7 are implemented.

Images