Speech recognition method, device, equipment and computer program product

Abstract

The invention provides a voice recognition method, device and equipment and a computer program product, and the method can extract image features from an image of a voice recognition object and extract voice features from audio of the voice recognition object, and the image features comprise lip motion features, gesture features and facial expression features. Respectively determining the weights of the image features and the voice features according to the light condition, the acoustic condition and the like of the voice recognition object; and finally, according to the weights of the image features and the voice features, performing weighted fusion on the image features and the voice features to obtain fusion features, and determining a voice recognition result through the fusion features. Through the arrangement, the voice can be recognized from multiple dimensions such as the voice, the lip action, the gesture action and the facial expression action of the recognized object, meanwhile, the weights corresponding to the different dimensions are adjusted according to the environmental condition adaptability, then the influence of the environment on the voice recognition result is avoided, and the voice recognition accuracy is improved.

Description

Technical Field

[0001] The present application relates to the field of speech recognition technology, and in particular to a speech recognition method, apparatus, device and computer program product. Background Art

[0002] With the rapid development of artificial intelligence and the Internet of Things, speech recognition, as a key entry technology for human-computer interaction, has been increasingly used in a wide range of scenarios, from smart homes and smart cars to industrial manufacturing and smart security. However, noise problems are often difficult to avoid in actual application scenarios, which affects the accuracy of speech recognition. Summary of the invention

[0003] In view of this, the present application proposes a speech recognition method, apparatus, device and computer program product to solve the problem that the accuracy of speech recognition is affected in the prior art.

[0004] The technical solutions proposed in this application are as follows: In a first aspect, an embodiment of the present application provides a speech recognition method, comprising: Extracting image features from an image of a speech recognition object, and extracting speech features from the audio of the speech recognition object; the image and audio of the speech recognition object are both collected when the speech recognition object is speaking, and the image features include lip movement features, gesture features, and facial expression features; Determining the weights of the image features and the voice features respectively according to the environmental conditions in which the voice recognition object is located; the environmental conditions include light conditions and acoustic conditions; According to the weights of the image features and the voice features, the image features and the voice features are weightedly fused to obtain fused features, and the voice recognition result is determined by the fused features.

[0005] Furthermore, in the above method, determining the weights of the image feature and the voice feature respectively according to the environmental conditions of the voice recognition object includes: If it is detected that the lighting condition does not meet the image shooting requirements, it is determined that the weight of the voice feature is greater than the weight of the image feature; or, if it is detected that the acoustic condition does not meet the audio collection requirements, it is determined that the weight of the image feature is greater than the weight of the voice feature.

[0006] Furthermore, in the above method, before extracting image features from the image of the speech recognition object and extracting speech features from the audio of the speech recognition object, the method comprises: When the wake-up information is detected, the image and audio of the speech recognition object when speaking are collected; the wake-up information includes at least one of voice wake-up information, face wake-up information, gesture wake-up information, gaze wake-up information and near-field wake-up information.

[0007] Further, in the above method, the wake-up information includes gesture wake-up information, and the method further includes: In the case where the gesture action of the voice recognition object includes setting a wake-up action, determining the intention of the voice recognition object by analyzing the sight line and / or whole body action of the voice recognition object; If the intention of the voice recognition object includes gesture wake-up, it means that the gesture action of the voice recognition object includes the gesture wake-up information.

[0008] Furthermore, in the above method, the wake-up information includes voice wake-up information, and the method further includes: In a case where the speech content of the speech recognition object includes a set wake-up speech, determining the intention of the speech recognition object by analyzing the speech content of the speech recognition object; If the intention of the voice recognition object includes voice wake-up, it means that the voice content of the voice recognition object includes the voice wake-up information.

[0009] Furthermore, in the above method, the step of performing weighted fusion of the image feature and the speech feature to obtain a fusion feature according to the weights of the image feature and the speech feature comprises: Normalizing the image features and the speech features to the same scale, and performing temporal alignment processing to obtain processed image features and speech features; The processed image features and speech features are weighted and fused to obtain fused features.

[0010] In a second aspect, an embodiment of the present application provides a speech recognition device, comprising: An extraction unit, configured to extract image features from an image of a speech recognition object, and extract speech features from an audio of the speech recognition object; the image and audio of the speech recognition object are both collected when the speech recognition object is speaking, and the image features include lip movement features, gesture features, and facial expression features; A determination unit, configured to determine the weights of the image feature and the speech feature respectively according to the environmental conditions in which the speech recognition object is located; the environmental conditions include light conditions and acoustic conditions; The fusion unit is used to perform weighted fusion on the image feature and the voice feature according to the weights of the image feature and the voice feature to obtain a fusion feature, and determine the voice recognition result through the fusion feature.

[0011] In a third aspect, an embodiment of the present application provides an electronic device, including: A memory and a processor; wherein the memory is used to store programs; and the processor is used to implement any of the methods described above by running the programs in the memory.

[0012] In a fourth aspect, an embodiment of the present application provides a speech recognition device, including: a camera, a microphone array, and a processing device; The camera is used to collect an image of the speech recognition object when the speech recognition object speaks, and the microphone array is used to collect the audio of the speech recognition object when the speech recognition object speaks; The processing device is used to extract image features from the image of the speech recognition object and extract speech features from the audio of the speech recognition object, the image features including lip movement features, gesture features and facial expression features; determine the weights of the image features and the speech features respectively according to the environmental conditions in which the speech recognition object is located, the environmental conditions including light conditions and sound conditions; and perform weighted fusion of the image features and the speech features according to the weights of the image features and the speech features to obtain fused features, and determine the speech recognition result through the fused features.

[0013] In a fifth aspect, an embodiment of the present application provides a computer program product, the computer program product comprising a computer program, and when the computer program is executed by a processor, the computer program implements any of the above methods. Optionally, the computer program can be stored in a readable storage medium of a computer device or in the cloud; the processor of the computer device reads the computer program from the readable storage medium or the cloud.

[0014] The speech recognition method proposed in the present application can extract image features from the image of the speech recognition object and extract speech features from the audio of the speech recognition object, wherein the image and audio of the speech recognition object are collected when the speech recognition object speaks, and the image features include lip movement features, gesture features and facial expression features. Then, according to the light conditions and acoustic conditions of the speech recognition object, the weights of the image features and speech features are determined respectively. Finally, according to the weights of the image features and speech features, the image features and speech features are weightedly fused to obtain fused features, and the speech recognition results are determined by the fused features. In this way, the speech can be recognized from multiple dimensions such as the speech, lip movements, gesture movements, facial expressions and movements of the recognition object, and the weights corresponding to different dimensions are adjusted according to the adaptability of environmental conditions, thereby avoiding the influence of the environment on the speech recognition results and improving the accuracy of speech recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0016] Figure 1 This is a circuit diagram of a speech recognition device provided in an embodiment of the present application.

[0017] Figure 2 It is a structural diagram of a speech recognition device provided in an embodiment of the present application.

[0018] Figure 3 It is a flow chart of a speech recognition method provided in an embodiment of the present application.

[0019] Figure 4 It is a structural schematic diagram of a speech recognition device provided in an embodiment of the present application.

[0020] Figure 5 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0022] With the rapid development of artificial intelligence and Internet of Things technologies, speech recognition, as a key entry technology for human-computer interaction, has been increasingly applied in a wide range of scenarios, from smart homes and smart cars to industrial manufacturing and smart security. However, in actual application scenarios, complex noise environments have always been an important factor affecting the accuracy of speech recognition. In particular, in scenarios with low signal-to-noise ratios, such as noisy factory workshops and busy public places, traditional speech recognition technology often finds it difficult to achieve ideal recognition results.

[0023] Based on this, the present application proposes a speech recognition method, apparatus, device and computer program product. The technical solution recognizes speech from multiple dimensions such as the speech, lip movements, gestures, facial expressions, etc. of the recognition object, and adaptively adjusts the weights corresponding to different dimensions according to environmental conditions, thereby avoiding the influence of the environment on the speech recognition results and improving the accuracy of speech recognition.

[0024] like Figure 1 and Figure 2 As shown, the speech recognition device includes a camera 10, a microphone array 11 and a processing device 12. The camera 10 and the microphone array 11 are electrically connected to the processing device 12 respectively.

[0025] The camera 10 is used to collect images of the speech recognition object when the speech recognition object speaks, including facial images, lip detail images, and gesture action images of the speech recognition object. The microphone array 11 is used to collect audio of the speech recognition object when the speech recognition object speaks. The processing device 12 obtains the image and audio of the speech recognition object, extracts image features from the image of the speech recognition object, and extracts speech features from the audio of the speech recognition object, wherein the image features include lip action features, gesture features, and facial expression features. The processing device 12 first determines the weights of the image features and the speech features respectively according to the environmental conditions in which the speech recognition object is located, wherein the environmental conditions include light conditions and sound conditions, and then performs weighted fusion of the image features and the speech features according to the weights of the image features and the speech features to obtain fusion features, and finally determines the speech recognition result through the fusion features.

[0026] The camera 10 includes at least one high-resolution camera with a high frame rate and good low-light performance to ensure that clear and accurate image information can be obtained under different environmental conditions. The microphone array 11 includes an array composed of multiple high-sensitivity microphones, which are used to collect sound signals from different directions and positions. At the same time, the time delay difference of arrival (TDOA) positioning technology is used to enhance the positioning and collection capabilities of the sound source, improve the quality of the sound signal, and use beamforming technology to suppress noise interference from other directions to a certain extent, and improve the signal-to-noise ratio. It should be noted that Figure 2 The microphone array 11 in the illustrated embodiment is provided with 8 microphones for illustration purposes only and is not intended to limit the number of microphones in the microphone array 11 , and the specific number may be set according to actual conditions.

[0027] The camera 10 can be arranged facing the face of the speech recognition object so as to obtain the complete facial image, lip detail image and gesture image of the speech recognition object. The sound receiving holes of each microphone in the microphone array 11 can be arranged facing the speech recognition object so as to collect clearer audio.

[0028] like Figure 2 As shown, the speech recognition device may further include an audio broadcast device 13. The audio broadcast device 13 is electrically connected to the processing device 12, and the audio broadcast device 13 is used to broadcast the speech content output by the speech recognition device.

[0029] Furthermore, if Figure 1 As shown, the camera 10 is electrically connected to the processing device 12 via a USB camera harness. The microphone array 11 includes a microphone board and a sound card mainboard, the microphone board and the sound card mainboard are electrically connected via an audio line, and the sound card mainboard is electrically connected to the processing device 12 via a USB audio line.

[0030] In some embodiments, the processing device 12 is connected to an external host computer by wired means, for example, the host computer can transmit serial port information to the host computer through an RS235 serial port to USB cable, and the host computer can connect the acquisition to the host computer through a 3.5mm headphone cable. When the speech recognition result is finally obtained, the processing device 12 can send the result to the host computer through the serial port, and the host computer can perform logical judgment, business operation, speech synthesis and other instructions according to the received message.

[0031] In some embodiments, the processing device 12 uses the RK3588 chip, which has powerful computing power and low power consumption. It also integrates a variety of high-performance processor cores, including a central processing unit (CPU), a graphics processing unit (GPU), a neural network processor (NPU), etc., providing sufficient computing power support for processing in multiple dimensions such as voice, lip movements, gestures, and facial expressions. Its advanced architecture design enables it to efficiently run complex deep learning algorithms and image processing algorithms, ensuring the real-time performance and response speed of the multi-dimensional interactive system.

[0032] The above-mentioned embodiments can recognize speech from multiple dimensions such as the speech, lip movements, gestures, facial expressions, etc. of the recognition object, and adaptively adjust the weights corresponding to different dimensions according to environmental conditions, thereby avoiding the influence of the environment on the speech recognition results and improving the accuracy of speech recognition.

[0033] Furthermore, the present application embodiment proposes a speech recognition method, which can be executed by an electronic device, which can be any device with data and instruction processing functions, such as a notebook computer, a tablet computer, a desktop computer, a mobile device (such as a mobile phone, a personal digital assistant, a dedicated messaging device) and other types of user terminals, or a combination of any two or more of these electronic devices, or a server, or the speech recognition device in the above embodiment. Figure 3 As shown, the method includes: S101. Extracting image features from an image of a speech recognition object, and extracting speech features from an audio of the speech recognition object.

[0034] The above-mentioned speech recognition object refers to the subject for speech recognition, that is, in this embodiment, the speech of the speech recognition object is recognized.

[0035] In the embodiment of the present application, the image and audio of the speech recognition object when speaking are collected synchronously. The image of the speech recognition object when speaking can be collected by a camera, and the audio of the speech recognition object when speaking can be collected synchronously by a microphone array.

[0036] For the multi-channel audio of the speech recognition object, pre-processing can be performed first. Specifically, it includes filtering, noise reduction, gain control and other operations to improve the quality of the audio. Then, an adaptive filtering algorithm is used to dynamically adjust the filter parameters according to the characteristics of the ambient noise to further suppress the background noise.

[0037] Then, feature extraction is performed on the preprocessed audio to obtain speech features. In some embodiments, a speech feature extraction model can be used to extract features from the preprocessed audio. Specifically, the preprocessed audio is input into a trained speech feature extraction model to obtain speech features output by the speech feature extraction model. The speech feature extraction model can use a convolutional neural network (CNN), a recurrent neural network (RNN), or a transformer, etc., which is not limited in this embodiment.

[0038] For the image of the speech recognition object, deep learning algorithms such as CNN and RNN can also be used to extract image features. Among them, the image features include lip movement features, gesture features, and facial expression features. In some embodiments, by training the CNN model, the lip movement features, gesture features, and facial expression features in the image are automatically learned. The CNN model can effectively capture the local features and spatial relationships in the image and generate a highly distinguishable feature vector.

[0039] The speech features, lip movement features, gesture features and facial expression features extracted in this way can be used to recognize the speech from multiple dimensions such as the speech, lip movement, gesture, facial expression, etc. of the recognition object, thereby improving the accuracy of the recognition results.

[0040] S102: Determine the weights of the image features and the voice features respectively according to the environmental conditions of the voice recognition object.

[0041] Detect the environmental conditions of the speech recognition object, including light conditions and acoustic conditions.

[0042] The light conditions include the light direction, light intensity, light color, and light contrast of the environment where the speech recognition object is located. The shooting requirements can be set according to the camera model and usage scenario, such as setting the allowable angle range of the light direction and the allowable range of the light intensity. When the light conditions meet the shooting requirements, it means that high-quality images can be taken, and the weight of the image features can be appropriately increased; when the light conditions do not meet the shooting requirements, it means that high-quality images may not be taken, and the weight of the image features can be appropriately reduced.

[0043] Acoustic conditions include the noise level of the environment in which the speech recognition object is located, which can be expressed by the signal-to-noise ratio. The recording requirements can be set according to the model of the microphone array and the usage scenario, such as the allowable range of the signal-to-noise ratio. When the actual signal-to-noise ratio meets the recording requirements, it means that high-quality audio can be collected, and the weight of the speech feature can be appropriately increased; when the actual signal-to-noise ratio does not meet the recording requirements, it means that high-quality audio cannot be collected, and the weight of the speech feature can be appropriately reduced.

[0044] Furthermore, the image features include lip movement features, gesture features, and facial expression features. When a clear image of the face of the speech recognition object can be obtained, the weights of the lip movement features and facial expression features can be appropriately increased; when a clear image of the face of the speech recognition object cannot be obtained, the weights of the lip movement features and facial expression features can be appropriately reduced. When the complete gesture of the speech recognition object can be obtained, the weight of the gesture feature can be appropriately increased; when the complete gesture of the speech recognition object cannot be obtained, the weight of the gesture feature can be appropriately reduced.

[0045] That is to say, the weights of speech features and image features can be dynamically adjusted according to the scene where the speech recognition object is located.

[0046] S103. According to the weights of the image features and the voice features, weighted fusion is performed on the image features and the voice features to obtain fusion features, and a voice recognition result is determined by the fusion features.

[0047] According to the weights of the image features and the speech features, the image features and the speech features are weightedly fused. Specifically, the image features and the speech features can be preprocessed first, for example, the image features and the speech features can be standardized by a standardization or normalization method to obtain single-modal features, and then the features of each single modality are fused in the final stage of multimodal fusion, i.e., the decision layer. The image features and the speech features can be weightedly fused by a voting method, a weighted voting method, a Bayesian fusion method, etc., which is not limited in this embodiment.

[0048] After obtaining the fused features, the fused features can be further parsed to obtain a speech recognition result. Specifically, the fused features can be decoded by a decoder to obtain a speech recognition result, or the fused features can be recognized by a speech recognition model to obtain a speech recognition result. Other methods in the prior art can also be used to parse the fused features into speech recognition results, which are not limited in this embodiment.

[0049] In the above embodiment, image features can be extracted from the image of the speech recognition object, and speech features can be extracted from the audio of the speech recognition object, wherein the image and audio of the speech recognition object are collected when the speech recognition object speaks, and the image features include lip movement features, gesture features, and facial expression features. Then, according to the light conditions and acoustic conditions of the speech recognition object, the weights of the image features and the speech features are determined respectively. Finally, according to the weights of the image features and the speech features, the image features and the speech features are weightedly fused to obtain fused features, and the speech recognition results are determined by the fused features. In this way, the speech can be recognized from multiple dimensions such as the speech, lip movements, gesture movements, and facial expressions of the recognition object, and the weights corresponding to different dimensions are adjusted according to the adaptability of environmental conditions, thereby avoiding the influence of the environment on the speech recognition results and improving the accuracy of speech recognition.

[0050] As an optional implementation, another embodiment of the present application discloses that the steps of the above embodiment respectively determine the weights of the image features and the voice features according to the environmental conditions of the voice recognition object, and specifically may include the following steps: If it is detected that the lighting condition does not meet the image shooting requirements, it is determined that the weight of the voice feature is greater than the weight of the image feature; or, if it is detected that the acoustic condition does not meet the audio collection requirements, it is determined that the weight of the image feature is greater than the weight of the voice feature.

[0051] Specifically, the image shooting requirements can be set according to information such as the camera model and usage scenario, such as setting the light intensity range, light angle range, etc. When the actual light conditions are outside the set range, it means that the current light conditions do not meet the image shooting requirements. For example, in scenes such as backlighting and overexposure, the camera imaging quality is poor. At this time, the weight of the voice feature can be set to be greater than the weight of the image feature to reduce the weight of the image feature and avoid the bad light conditions affecting the voice recognition results.

[0052] The audio collection requirements can be set according to information such as the model of the microphone array and the usage scenario, such as setting the signal-to-noise ratio range. When the actual signal-to-noise ratio is outside the set range, it means that the current acoustic conditions do not meet the audio collection requirements. For example, if the recording quality is poor in a noisy scene, the weight of the image feature can be set to be greater than the weight of the speech feature to reduce the weight of the speech feature and avoid external noise affecting the speech recognition results.

[0053] When the current lighting conditions do not meet the image shooting requirements and the acoustic conditions do not meet the audio collection requirements, in some embodiments, the voice recognition object can be prompted that the current environment cannot perform voice recognition, and it is recommended that the voice recognition object move to a place where the lighting conditions and / or acoustic conditions meet the requirements before performing voice input.

[0054] In another embodiment, multiple groups of light condition ranges may be set according to information such as the camera model and usage scenario, and different light condition ranges correspond to different image feature weights, wherein the more suitable the light condition range is for taking pictures, the higher the corresponding image feature weight is. Then, the light condition range into which the current light condition falls is detected, and the weight corresponding to the light condition range is determined as the weight of the image feature.

[0055] According to the model of the microphone array and the usage scenario, multiple groups of acoustic condition ranges are set. Different acoustic condition ranges correspond to different speech feature weights. The higher the signal-to-noise ratio, the higher the corresponding speech feature weight. Then, the acoustic condition range that the current acoustic condition falls into is detected, and the weight corresponding to the acoustic condition range is determined as the weight of the speech feature.

[0056] Then, according to the weights of the image features and the voice features, the image features and the voice features are weightedly fused to obtain fused features, and the voice recognition result is determined by the fused features.

[0057] In the above embodiment, the weights of speech features and image features can be dynamically adjusted according to the scene where the speech recognition object is located, thereby avoiding the influence of the environment on the speech recognition result and improving the accuracy of speech recognition.

[0058] As an optional implementation, another embodiment of the present application discloses that the steps of the above embodiment, before extracting image features from the image of the speech recognition object and extracting speech features from the audio of the speech recognition object, may specifically include the following steps: When the wake-up information is detected, the image and audio of the speech recognition object when speaking are collected; the wake-up information includes at least one of voice wake-up information, face wake-up information, gesture wake-up information, gaze wake-up information and near-field wake-up information.

[0059] After the wake-up information is detected, the image and audio of the speech recognition object when speaking are collected to recognize the speech of the speech recognition object.

[0060] In the embodiments of the present application, multiple wake-up modes are provided, including at least one of voice wake-up, face wake-up, gesture wake-up, gaze wake-up and near-field wake-up.

[0061] Among them, voice wake-up refers to the method of activating the device through specific voice commands, and the specific voice commands are used as voice wake-up information. Face wake-up refers to the method of activating the device by detecting the user's face, and the user's facial features are used as face wake-up information. Gesture wake-up refers to the method of activating the device through specific gestures, and the specific gestures are used as gesture wake-up information. Gaze wake-up refers to the method of activating the device by detecting the user's gaze, and the user's gaze is used as gaze wake-up information. Near-field wake-up refers to the method of activating the device when the user approaches the device, and the distance between the user and the device is less than the set distance as the near-field wake-up information.

[0062] By setting a variety of wake-up methods, users can use it more conveniently, thereby improving the user experience.

[0063] In addition, to meet the business needs of different third-party smart terminals, third parties can flexibly configure the wake-up capabilities required by the business through configuration parameters. When multiple wake-up capabilities are configured, the scheduling engine uses the built-in interaction right transfer service to block complex scheduling logic with simple interfaces and events, thereby reducing the difficulty of access for manufacturers, ensuring the interactive effect of the product, and improving the interactive experience and intelligence level of the device.

[0064] As an optional implementation, another embodiment of the present application discloses that the wake-up information of the above embodiment includes gesture wake-up information, and the method of the above embodiment may specifically include the following steps: In the case where the gesture action of the voice recognition object includes a set wake-up action, the intention of the voice recognition object is determined by analyzing the line of sight and / or whole body movement of the voice recognition object; if the intention of the voice recognition object includes gesture wake-up, it means that the gesture action of the voice recognition object includes the gesture wake-up information.

[0065] Specifically, when the user's gesture is detected to be the same as the set wake-up action, it means that the gesture wake-up information has been received. However, when the user is talking to others or dancing, there will be certain body movements. If the user makes the same gesture as the set wake-up action when talking to others or dancing, it will cause a false wake-up.

[0066] To avoid this situation, in an embodiment of the present application, when it is detected that the gesture action of the voice recognition object includes a set wake-up action, the line of sight and / or whole body movement of the voice recognition object is first analyzed to determine whether the intention of the voice recognition object is gesture wake-up.

[0067] In some embodiments, the first intention analysis model can be used to input an image containing the sight line and / or whole body movement of the speech recognition object into the first intention analysis model to obtain an analysis result output by the first intention analysis model. The analysis result of the first intention analysis model includes whether the speech recognition object has the intention of gesture wake-up.

[0068] Specifically, a large number of images containing sample users' line of sight and / or whole body movements can be obtained as training samples. Among the training samples, some sample users' gesture movements are for gesture wake-up, and the labels of such training samples are marked as having wake-up intention. Some sample users' gesture movements are for speaking, fitness, or dancing, etc., and the labels of such training samples are marked as having no wake-up intention.

[0069] During training, the training samples are input into the first intent analysis model to obtain the analysis results of the first intent analysis model. The loss of the first intent analysis model is determined by comparing the analysis results of the first intent analysis model with the training labels. The parameters of the first intent analysis model are adjusted with the goal of reducing the loss of the first intent analysis model. The above training process is then repeated until the first intent analysis model meets the training requirements. The above-mentioned first intent analysis model can be obtained by training based on any neural network model, or can be obtained by training based on a pre-trained model, which is not limited in this embodiment.

[0070] In general, when the user's line of sight is looking at the device and the action is a single wake-up action, the user's intention is gesture wake-up; when the user's line of sight is not looking at the device, or the action includes other actions besides the wake-up action, the user's intention is not gesture wake-up. If the intention of the voice recognition object includes gesture wake-up, it means that the gesture action of the voice recognition object includes the gesture wake-up information.

[0071] In the above embodiment, by analyzing the intention of the voice recognition object, it is possible to avoid the occurrence of false wake-up.

[0072] As an optional implementation, another embodiment of the present application discloses that the wake-up information of the above embodiment includes voice wake-up information, and the method of the above embodiment may specifically include the following steps: In the case where the voice content of the voice recognition object includes a set wake-up voice, the intention of the voice recognition object is determined by analyzing the voice content of the voice recognition object; if the intention of the voice recognition object includes voice wake-up, it means that the voice content of the voice recognition object includes the voice wake-up information.

[0073] Specifically, generally, when the user's voice content is detected to be the same as the set wake-up voice, it means that the voice wake-up information has been received. However, if the content spoken by the user while talking to others is the same as the set wake-up voice content, a false wake-up situation will occur.

[0074] To avoid this situation, in an embodiment of the present application, when it is detected that the voice content of the voice recognition object includes a set wake-up voice, the voice content of the voice recognition object is first analyzed to determine whether the intention of the voice recognition object is voice wake-up.

[0075] In some embodiments, the second intention analysis model can be used to input the speech content into the second intention analysis model to obtain the analysis result output by the second intention analysis model. The analysis result of the second intention analysis model includes whether the speech recognition object has the intention of speech wake-up.

[0076] Specifically, a large number of sample users' speech contents can be obtained as training samples. Among the training samples, some sample users' purpose is to perform voice wake-up, and the labels of such training samples are marked as having wake-up intention. Some sample users are talking with others, singing or reciting poetry, etc., and the labels of such training samples are marked as having no wake-up intention.

[0077] During training, the training samples are input into the second intent analysis model to obtain the analysis results of the second intent analysis model. The loss of the second intent analysis model is determined by comparing the analysis results of the second intent analysis model with the training labels. The parameters of the second intent analysis model are adjusted with the goal of reducing the loss of the second intent analysis model. The above training process is then repeated until the second intent analysis model meets the training requirements. The above second intent analysis model can be obtained by training based on any neural network model, or can be obtained by training based on a pre-trained model, which is not limited in this embodiment.

[0078] If the intention of the voice recognition object includes voice wake-up, it means that the voice content of the voice recognition object includes the voice wake-up information.

[0079] In the above embodiment, by analyzing the intention of the voice recognition object, it is possible to avoid the occurrence of false wake-up.

[0080] As an optional implementation, another embodiment of the present application discloses that the above embodiment performs weighted fusion of the image feature and the voice feature according to the weights of the image feature and the voice feature to obtain a fusion feature, which may specifically include the following steps: The image features and speech features are normalized to the same scale and aligned in time to obtain processed image features and speech features; the processed image features and speech features are weighted fused to obtain fused features.

[0081] Specifically, when performing weighted fusion of the image features and the speech features, the image features and the speech features need to be preprocessed first, specifically including normalizing the image features and the speech features to the same scale to eliminate scale differences for subsequent weighted fusion. Then ensure that the image features and the speech features are aligned in time to improve fusion accuracy, thereby improving the accuracy of speech recognition.

[0082] The pre-processed image features and speech features are weighted and fused to obtain fused features. Recognition is performed based on the fused features to obtain more accurate speech recognition results.

[0083] Furthermore, in some embodiments, a gesture action library is established to define various common gestures and their corresponding command meanings. After acquiring gesture image features through image preprocessing and feature extraction, the gesture features collected in real time are compared with the templates in the gesture action library using template matching or deep learning algorithms to identify the gestures made by the user and convert them into corresponding command signals to realize gesture interaction functions.

[0084] Corresponding to the above-mentioned speech recognition method, the present application embodiment also discloses a speech recognition device, see Figure 4 As shown, the device comprises: The extraction unit 100 is used to extract image features from the image of the speech recognition object and extract speech features from the audio of the speech recognition object; the image and audio of the speech recognition object are collected when the speech recognition object speaks, and the image features include lip movement features, gesture features, and facial expression features; The determination unit 110 is used to determine the weights of the image feature and the speech feature respectively according to the environmental conditions of the speech recognition object; the environmental conditions include light conditions and acoustic conditions; The fusion unit 120 is used to perform weighted fusion on the image feature and the speech feature according to the weights of the image feature and the speech feature to obtain a fusion feature, and determine the speech recognition result according to the fusion feature.

[0085] As an optional implementation, another embodiment of the present application discloses that the determination unit 110 of the above embodiment, when determining the weights of the image feature and the voice feature respectively according to the environmental conditions of the voice recognition object, is specifically used to: If it is detected that the lighting condition does not meet the image shooting requirements, it is determined that the weight of the voice feature is greater than the weight of the image feature; or, if it is detected that the acoustic condition does not meet the audio collection requirements, it is determined that the weight of the image feature is greater than the weight of the voice feature.

[0086] As an optional implementation, another embodiment of the present application discloses that the device in the above embodiment further includes: A collection unit is used to collect images and audio when the speech recognition object speaks when wake-up information is detected; the wake-up information includes at least one of voice wake-up information, face wake-up information, gesture wake-up information, gaze wake-up information and near-field wake-up information.

[0087] As an optional implementation, another embodiment of the present application discloses that the wake-up information of the above embodiment includes gesture wake-up information, and the determination unit of the above embodiment is further used to: In the case where the gesture action of the voice recognition object includes setting a wake-up action, determining the intention of the voice recognition object by analyzing the sight line and / or whole body action of the voice recognition object; If the intention of the voice recognition object includes gesture wake-up, it means that the gesture action of the voice recognition object includes the gesture wake-up information.

[0088] As an optional implementation, another embodiment of the present application discloses that the wake-up information of the above embodiment includes voice wake-up information, and the determination unit of the above embodiment is further used to: In a case where the speech content of the speech recognition object includes a set wake-up speech, determining the intention of the speech recognition object by analyzing the speech content of the speech recognition object; If the intention of the voice recognition object includes voice wake-up, it means that the voice content of the voice recognition object includes the voice wake-up information.

[0089] As an optional implementation, another embodiment of the present application discloses that the fusion unit 120 of the above embodiment, when performing weighted fusion of the image feature and the speech feature to obtain the fusion feature according to the weights of the image feature and the speech feature, is specifically used to: Normalize the image features and speech features to the same scale and align them in time to obtain processed image features and speech features; The processed image features and speech features are weighted and fused to obtain fused features.

[0090] Specifically, the device provided in this embodiment belongs to the same application concept as the method provided in the above embodiment of this application, can execute the method provided in any of the above embodiments of this application, and has the corresponding functional modules and beneficial effects of the execution method. For the technical details not fully described in this embodiment, please refer to the specific processing content of the method provided in the above embodiment of this application, which will not be repeated here.

[0091] The functions implemented by the above units can be implemented by the same or different processors respectively, and the embodiments of the present application are not limited thereto.

[0092] It should be understood that the units in the above devices can be implemented in the form of a processor calling software. For example, the device includes a processor, the processor is connected to a memory, and instructions are stored in the memory. The processor calls the instructions stored in the memory to implement any of the above methods or realize the functions of each unit of the device, wherein the processor can be a general-purpose processor, such as a CPU or a microprocessor, etc., and the memory can be a memory in the device or a memory outside the device. Alternatively, the units in the device can be implemented in the form of hardware circuits, and the functions of some or all units can be realized by designing the hardware circuits. The hardware circuit can be understood as one or more processors; for example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units are realized by designing the logical relationship of the components in the circuit; for another example, in another implementation, the hardware circuit can be implemented by PLD, taking FPGA as an example, which can include a large number of logic gate circuits, and the connection relationship between the logic gate circuits is configured by the configuration file, so as to realize the functions of some or all of the above units. All units of the above devices can be implemented in the form of a processor calling software, or in the form of hardware circuits, or in part by a processor calling software, and the remaining part is implemented in the form of hardware circuits.

[0093] In an embodiment of the present application, a processor is a circuit with the ability to process signals. In one implementation, the processor may be a circuit with the ability to read and run instructions, such as a CPU, a microprocessor, a GPU, or a DSP; in another implementation, the processor may implement certain functions through the logical relationship of a hardware circuit, and the logical relationship of the hardware circuit is fixed or reconfigurable, such as a hardware circuit implemented by an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document to implement the hardware circuit configuration can be understood as the process of the processor loading instructions to implement the functions of some or all of the above units. In addition, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, DPU, etc.

[0094] It can be seen that each unit in the above device can be one or more processors (or processing circuits) configured to implement the above method, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0095] In addition, all or part of the units in the above device can be integrated together, or can be implemented independently. In one implementation, these units are integrated together and implemented in the form of a SOC. The SOC may include at least one processor for implementing any of the above methods or implementing the functions of each unit of the device. The type of the at least one processor may be different, for example, including a CPU and an FPGA, a CPU and an artificial intelligence processor, a CPU and a GPU, etc.

[0096] An embodiment of the present application further provides a control device, which includes a processor and an interface circuit. The processor in the control device is connected to an input-output component through the interface circuit of the control device.

[0097] The input-output component specifically refers to a hardware component that enables a user to input information and output information to the user, such as a microphone, keyboard, handwriting tablet, touch screen, display, speaker, printer, etc.

[0098] The above-mentioned interface circuit can be any interface circuit that can realize the data communication function, for example, it can be a USB interface circuit, a Type-C interface circuit, a serial port circuit, a PCIE circuit, etc.

[0099] The processor in the control device is a circuit with signal processing capability, which improves the accuracy of speech recognition by executing any of the speech recognition methods introduced in the above embodiments. The specific implementation of the processor can refer to the above processor implementation, and the embodiments of this application are not strictly limited.

[0100] When the control device is applied to a device with a human-computer interaction function, the input and output components of the control device may be input components and output components on the device, such as a microphone, a keyboard, a handwriting tablet, a touch screen, a display, an audio player, etc. At the same time, the processor of the control device may be a CPU or GPU, etc. provided by the device, and the interface circuit of the control device may be an interface circuit between the information input component of the device and a processor such as a CPU or GPU.

[0101] Corresponding to the above-mentioned speech recognition method, the present application embodiment also discloses an electronic device, see Figure 5 As shown, the electronic device includes: Memory 200 and processor 210; The memory 200 is connected to the processor 210 and is used to store programs; The processor 210 is used to implement the speech recognition method disclosed in any of the above embodiments by running the program stored in the memory 200.

[0102] Specifically, the electronic device may further include: a bus, a communication interface 220 , an input device 230 and an output device 240 .

[0103] The processor 210, the memory 200, the communication interface 220, the input device 230 and the output device 240 are connected to each other via a bus. A bus may include a pathway that transfers information between components of a computer system.

[0104] The processor 210 may be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the present application. It may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0105] The processor 210 may include a main processor, and may also include a baseband chip, a modem, and the like.

[0106] The memory 200 stores a program for executing the technical solution of the present application, and may also store an operating system and other key services. Specifically, the program may include a program code, and the program code includes a computer operation instruction. More specifically, the memory 200 may include a read-only memory (ROM), other types of static storage devices that can store static information and instructions, a random access memory (RAM), other types of dynamic storage devices that can store information and instructions, a disk storage, a flash, and the like.

[0107] The input device 230 may include a device for receiving data and information input by a user, such as a keyboard, a mouse, a camera, a scanner, a light pen, a voice input device, a touch screen, a pedometer, or a gravity sensor.

[0108] Output device 240 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0109] The communication interface 220 may include any transceiver or the like to communicate with other devices or communication networks, such as Ethernet, a radio access network (RAN), a wireless local area network (WLAN), etc.

[0110] The processor 210 executes the program stored in the memory 200 and calls other devices, which can be used to implement the various steps of the speech recognition method provided in the above embodiments of the present application.

[0111] In addition to the above methods and devices, the embodiments of the present application may also be a computer program product, which includes a computer program. When the computer program is executed by a processor, the speech recognition method provided by any of the above embodiments of the present application may be executed. Optionally, the computer program may be stored in a readable storage medium or in the cloud of a computer device; the processor of the computer device reads the computer program from the readable storage medium or the cloud.

[0112] The computer program product may be written in any combination of one or more programming languages ​​to write program codes for performing the operations of the embodiments of the present application, including object-oriented programming languages ​​such as Java, C++, etc., and also conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0113] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium. In another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK).

[0114] In addition, the embodiments of the present application may also be a computer-readable storage medium on which computer program instructions are stored. When the computer program instructions are executed by a processor, the processor executes each step of the speech recognition method provided in the above embodiments.

[0115] The computer readable storage medium may adopt any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may include, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0116] Specifically, the specific working contents of each part of the above-mentioned electronic device, computer program product and storage medium, as well as the specific processing contents when the computer program product or the computer program on the above-mentioned storage medium is executed by the processor, can all be found in the contents of the various embodiments of the above-mentioned speech recognition method, and will not be repeated here.

[0117] For the aforementioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for the present application.

[0118] It should be noted that each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0119] The steps in the methods of each embodiment of the present application can be adjusted in order, combined and deleted according to actual needs, and the technical features recorded in each embodiment can be replaced or combined.

[0120] The modules and sub-modules in the devices and terminals in the various embodiments of the present application can be combined, divided and deleted according to actual needs.

[0121] In the several embodiments provided in the present application, it should be understood that the disclosed terminals, devices and methods can be implemented in other ways. For example, the terminal embodiments described above are only schematic, for example, the division of modules or submodules is only a logical function division, and there may be other division methods in actual implementation, such as multiple submodules or modules can be combined or integrated into another module, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0122] The modules or submodules described as separate components may or may not be physically separated, and the components of the modules or submodules may or may not be physical modules or submodules, that is, they may be located in one place, or they may be distributed on multiple network modules or submodules. Some or all of the modules or submodules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0123] In addition, each functional module or submodule in each embodiment of the present application may be integrated into one processing module, or each module or submodule may exist physically separately, or two or more modules or submodules may be integrated into one module. The above-mentioned integrated modules or submodules may be implemented in the form of hardware or in the form of software functional modules or submodules.

[0124] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0125] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly by hardware, software units executed by a processor, or a combination of the two. The software units may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0126] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprises" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including a..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0127] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A speech recognition method, characterized in that: include: Extracting image features from an image of a speech recognition object, and extracting speech features from an audio of the speech recognition object; The image and audio of the speech recognition object are collected when the speech recognition object speaks, and the image features include lip movement features, gesture features, and facial expression features; Determining the weights of the image features and the voice features respectively according to the environmental conditions in which the voice recognition object is located; the environmental conditions include light conditions and acoustic conditions; According to the weights of the image features and the voice features, the image features and the voice features are weightedly fused to obtain fused features, and the voice recognition result is determined by the fused features.

2. The method according to claim 1, characterized in that The step of determining the weights of the image feature and the voice feature respectively according to the environmental conditions of the voice recognition object comprises: If it is detected that the lighting condition does not meet the image shooting requirements, it is determined that the weight of the voice feature is greater than the weight of the image feature; or, if it is detected that the acoustic condition does not meet the audio collection requirements, it is determined that the weight of the image feature is greater than the weight of the voice feature.

3. The method according to claim 1, characterized in that Before extracting image features from the image of the speech recognition object and extracting speech features from the audio of the speech recognition object, the method includes: When the wake-up information is detected, the image and audio of the speech recognition object when speaking are collected; the wake-up information includes at least one of voice wake-up information, face wake-up information, gesture wake-up information, gaze wake-up information and near-field wake-up information.

4. The method according to claim 3, characterized in that The wake-up information includes gesture wake-up information, and the method further includes: In the case where the gesture action of the voice recognition object includes setting a wake-up action, determining the intention of the voice recognition object by analyzing the sight line and / or whole body action of the voice recognition object; If the intention of the voice recognition object includes gesture wake-up, it means that the gesture action of the voice recognition object includes the gesture wake-up information.

5. The method according to claim 3, characterized in that: The wake-up information includes voice wake-up information, and the method further includes: In a case where the speech content of the speech recognition object includes a set wake-up speech, determining the intention of the speech recognition object by analyzing the speech content of the speech recognition object; If the intention of the voice recognition object includes voice wake-up, it means that the voice content of the voice recognition object includes the voice wake-up information.

6. The method according to claim 1, characterized in that The step of performing weighted fusion on the image feature and the voice feature to obtain a fusion feature according to the weights of the image feature and the voice feature includes: Normalizing the image features and the speech features to the same scale, and performing temporal alignment processing to obtain processed image features and speech features; The processed image features and speech features are weighted and fused to obtain fused features.

7. A speech recognition device, characterized in that: include: An extraction unit, used to extract image features from the image of the speech recognition object, and extract speech features from the audio of the speech recognition object; The image and audio of the speech recognition object are collected when the speech recognition object speaks, and the image features include lip movement features, gesture features, and facial expression features; A determination unit, used to determine the weights of the image feature and the voice feature respectively according to the environmental conditions of the voice recognition object; The environmental conditions include light conditions and acoustic conditions; The fusion unit is used to perform weighted fusion on the image feature and the voice feature according to the weights of the image feature and the voice feature to obtain a fusion feature, and determine the voice recognition result through the fusion feature.

8. An electronic device, characterized in that: include: Memory and processor; Wherein, the memory is used to store programs; The processor is used to implement the method according to any one of claims 1 to 6 by running the program in the memory.

9. A speech recognition device, characterized in that: include: Cameras, microphone arrays, and processing equipment; The camera is used to collect an image of the speech recognition object when the speech recognition object speaks, and the microphone array is used to collect the audio of the speech recognition object when the speech recognition object speaks; The processing device is used to extract image features from the image of the speech recognition object and extract speech features from the audio of the speech recognition object, wherein the image features include lip movement features, gesture features, and facial expression features; According to the environmental conditions of the speech recognition object, the weights of the image features and the speech features are determined respectively, and the environmental conditions include light conditions and sound conditions; according to the weights of the image features and the speech features, the image features and the speech features are weightedly fused to obtain fused features, and the speech recognition result is determined by the fused features.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images