Vehicle-mounted voice recognition method and device, electronic equipment and storage medium

Abstract

The invention provides a vehicle-mounted voice recognition method and device, electronic equipment and a storage medium, and belongs to the technical field of voice recognition, and the method comprises the steps: carrying out the fusion of a voice recognition result and a lip language recognition result of a cab, obtaining a multi-modal recognition result, enabling the voice recognition result to be obtained through the recognition of a voice signal collected in real time, and enabling the lip language recognition result to be a lip language recognition result; the lip language recognition result is obtained by recognizing a mouth shape image collected in real time; analyzing the multi-modal recognition result to obtain a voice source of each voice and an intention corresponding to each voice; and based on the voice source and intention of each voice, respectively generating a vehicle control instruction. According to the invention, the voice recognition result and the lip language recognition result are combined to distinguish the voice from different sources, so that the main driver and the co-driver can respond to the voice instructions when sending the voice instructions at the same time, and the requirements of the main driver and the co-driver can be met at the same time.

Description

Technical Field

[0001] The present invention relates to the field of speech recognition technology, and in particular to a vehicle-mounted speech recognition method, device, electronic equipment and storage medium. Background Art

[0002] With the evolution of automotive intelligence, voice recognition technology has indeed become an indispensable part of modern vehicles and is widely used in many fields such as vehicle control, navigation guidance, and entertainment system control.

[0003] Currently, in-vehicle voice control systems can only recognize and respond to one voice command at a time, usually issued by the driver. When the driver and the co-pilot issue commands at the same time, the system often has difficulty distinguishing the source of the command, resulting in erroneous responses or ignored commands. At the same time, traditional in-vehicle voice systems lack the ability to distinguish between passengers in multiple seats, and cannot meet the needs of the driver and co-pilot to control different systems at the same time. For example, the driver wants to set the navigation route, while the co-pilot wants to adjust the entertainment system. Traditional systems find it difficult to meet these two needs at the same time.

[0004] Therefore, there is an urgent need for an in-vehicle speech recognition method to solve the problem that existing in-vehicle speech recognition solutions can only recognize a single command. Summary of the invention

[0005] The present invention provides a vehicle-mounted speech recognition method, device, electronic equipment and storage medium, which are used to solve the defect that the vehicle-mounted speech recognition solution in the prior art can only recognize a single instruction.

[0006] The present invention provides a vehicle-mounted speech recognition method, comprising the following steps: The speech recognition result and the lip reading recognition result of the cab are integrated to obtain a multimodal recognition result, wherein the speech recognition result is obtained by recognizing the speech signal collected in real time, and the lip reading recognition result is obtained by recognizing the lip shape image collected in real time; Analyze the multimodal recognition results to obtain the speech source of each speech and the intention corresponding to each speech; Based on the speech source and intention of each of the speech, vehicle control instructions are generated respectively.

[0007] According to a vehicle-mounted speech recognition method provided by the present invention, the speech recognition result is obtained based on the following steps: Performing noise elimination processing on the speech signal collected in real time to obtain a denoised speech signal; The denoised speech signal is analyzed using a preset speech recognition model to obtain a speech recognition result, wherein the speech recognition result includes a speech signal recognition result of each speaker.

[0008] According to a vehicle-mounted speech recognition method provided by the present invention, the process of performing noise elimination processing on the speech signal acquired in real time to obtain a denoised speech signal includes: Based on the preset filter, the environmental noise is removed from the speech signal acquired in real time by the microphone array to obtain a noise-reduced speech signal; Determining the position information of each sound source in the speech signal based on the time difference between each microphone in the microphone array receiving the sound; Recognize the cab video stream acquired in real time by the camera to obtain visual recognition results; Based on the position information of each sound source and the visual recognition result, the external voice signal is filtered to obtain a denoised voice signal.

[0009] According to a vehicle-mounted speech recognition method provided by the present invention, the lip reading recognition result is obtained based on the following steps: Perform facial region recognition on the cab video stream acquired in real time to obtain the facial region in each frame image; Extracting the mouth of the face in each frame of the image to obtain a mouth region of interest; The oral region of interest is input into a pre-trained lip reading recognition model to obtain a lip reading recognition result output by the lip reading recognition model, wherein the lip reading recognition model includes a convolutional neural network and a long short-term memory network.

[0010] According to a vehicle-mounted speech recognition method provided by the present invention, the vehicle control instructions are generated based on the speech source and intention of each speech, including: If the intentions of the voices do not interfere with each other, corresponding vehicle control instructions are generated based on the intentions of the voices.

[0011] According to a vehicle-mounted speech recognition method provided by the present invention, the vehicle control instructions are generated based on the speech source and intention of each speech, including: If the intentions of the voices interfere with each other, all the intentions are separated to obtain interfering intentions and non-interfering intentions, wherein the interfering intentions are intentions with an interfering relationship, and the non-interfering intentions are intentions without an interfering relationship; For the interference intention, generating corresponding vehicle control instructions based on the priority of the corresponding voice source; For the non-interference intentions, corresponding vehicle control instructions are generated respectively.

[0012] The present invention also provides a vehicle-mounted speech recognition device, comprising the following modules: The multimodal recognition module is used to: fuse the speech recognition result and the lip reading recognition result of the cab to obtain a multimodal recognition result, wherein the speech recognition result is obtained by recognizing the speech signal collected in real time, and the lip reading recognition result is obtained by recognizing the lip shape image collected in real time; A speech analysis module, used to: analyze the multimodal recognition results to obtain the speech source of each speech and the intention corresponding to each speech; The instruction generation module is used to generate vehicle control instructions based on the speech source and intention of each of the speech.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the vehicle-mounted speech recognition method as described above is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the vehicle-mounted speech recognition method as described in any one of the above is implemented.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the in-vehicle speech recognition method described above is implemented.

[0016] The in-vehicle speech recognition method, device, electronic device and storage medium provided by the present invention fuse the speech recognition result and lip reading recognition result of the cab to obtain a multimodal recognition result, wherein the speech recognition result is obtained by recognizing the speech signal collected in real time, and the lip reading recognition result is obtained by recognizing the lip shape image collected in real time; the multimodal recognition result is parsed to obtain the speech source of each speech and the intention corresponding to each speech; based on the speech source and intention of each speech, vehicle control instructions are generated respectively. The present invention combines the speech recognition result and the lip reading recognition result to distinguish between speech from different sources, so that when the main driver and the co-driver issue voice commands at the same time, they can respond to both, meet the needs of both, and improve the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1This is one of the flow charts of the vehicle-mounted speech recognition method provided by the present invention; Figure 2 This is the second flow chart of the vehicle-mounted speech recognition method provided by the present invention; Figure 3 It is a schematic diagram of the flow of the driver and co-driver voice response provided by the present invention; Figure 4 It is a structural schematic diagram of the vehicle-mounted speech recognition device provided by the present invention; Figure 5 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] It should be noted that in the description of the embodiments of the present invention, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "include one..." do not exclude the existence of other identical elements in the process, method, article or device including the elements. The orientation or position relationship indicated by the terms "upper", "lower" and the like is based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. Unless otherwise clearly specified and limited, the terms "installed", "connected" and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, or it can be a connection between two elements. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0021] The terms "first", "second", etc. in the present invention are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present invention can be implemented in an order other than those illustrated or described herein, and the objects distinguished by "first", "second", etc. are generally of the same type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" means at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0022] Combine the following Figure 1-Figure 5 The in-vehicle speech recognition method, device, electronic device and storage medium provided by the embodiments of the present invention are described.

[0023] Figure 1 This is one of the flow charts of the vehicle-mounted speech recognition method provided by the present invention; Figure 2 This is the second flow chart of the vehicle-mounted speech recognition method provided by the present invention; like Figure 1-Figure 2 As shown, the method includes the following: S110, fusing the speech recognition result and the lip reading recognition result of the cab to obtain a multimodal recognition result; S120, parsing the multimodal recognition result to obtain the speech source of each speech and the intention corresponding to each speech; S130, generating vehicle control instructions respectively based on the speech sources and intentions of the respective speech.

[0024] It should be noted that the execution subject of the vehicle-mounted speech recognition method provided by the embodiment of the present invention may be a vehicle-mounted electronic device. The execution subject of the vehicle-mounted speech recognition method may also be a vehicle-mounted speech control system, which belongs to the vehicle-mounted electronic device.

[0025] In S110, the speech recognition result is obtained by recognizing the speech signal collected in real time, and the lip reading recognition result is obtained by recognizing the lip shape image collected in real time.

[0026] In the specific implementation process, a voice acquisition module is installed in the car to collect voice signals in real time, and the real-time collected voice signals are recognized to obtain voice recognition results; a high-definition camera is installed in the cab to capture the lip images of people in the cab in real time, and the real-time collected images are recognized to obtain lip reading recognition results.

[0027] In S110, the speech recognition result and the lip reading recognition result are fused based on a preset data fusion algorithm, and a final multimodal recognition result is comprehensively output.

[0028] Optionally, based on a preset weight coefficient, the speech recognition result and the lip reading recognition result are combined to reduce the influence of external environmental noise or accent, thereby ensuring the accuracy of the final recognition result.

[0029] Traditional in-vehicle voice recognition mainly relies on single audio signal processing, which has high requirements on voice signal quality. In a noisy environment, single modality processing is difficult to effectively improve the recognition effect. Compared with relying on single audio signal processing, the embodiment of the present invention combines vision (lip reading) and audio signals to jointly recognize voice commands, effectively improving the accuracy of voice recognition and the user's interactive experience.

[0030] The in-vehicle speech recognition method provided by the embodiment of the present invention fuses the speech recognition result and the lip reading recognition result of the cab to obtain a multimodal recognition result, wherein the speech recognition result is obtained by recognizing the speech signal collected in real time, and the lip reading recognition result is obtained by recognizing the lip shape image collected in real time; the multimodal recognition result is parsed to obtain the speech source of each speech and the intention corresponding to each speech; based on the speech source and intention of each speech, vehicle control instructions are generated respectively. The present invention combines the speech recognition result and the lip reading recognition result to distinguish speech from different sources, so that when the main driver and the co-driver issue voice commands at the same time, they can respond to both, meet the needs of both, and improve the user experience. In addition, existing in-vehicle voice recognition systems often rely on the quality of voice signals, and the noisy environment makes the voice signals susceptible to interference, resulting in recognition errors or failures, affecting the system's interactive experience, and the differences in accents and speaking speeds of different drivers also increase the difficulty of recognition. The present invention combines the voice recognition results with the lip reading recognition results, and reduces the impact of the vehicle's external environmental noise and the driver's accent and speaking speed on voice recognition by adjusting the weight ratio of the voice recognition results and the lip reading recognition results, thereby improving the accuracy of voice recognition.

[0031] In an optional embodiment, the speech recognition result is obtained based on the following steps: Performing noise elimination processing on the speech signal collected in real time to obtain a denoised speech signal; The denoised speech signal is analyzed using a preset speech recognition model to obtain a speech recognition result, wherein the speech recognition result includes a speech signal recognition result of each speaker.

[0032] Specifically, the voice acquisition module installed in the cab collects voice signals in real time, removes environmental noise interference, and obtains denoised voice signals; the denoised voice signals are analyzed using a preset voice recognition algorithm to obtain voice recognition results.

[0033] The vehicle-mounted speech recognition method provided by the embodiment of the present invention performs noise elimination processing on the speech signal collected in real time, thereby reducing the interference of environmental noise and improving the accuracy of speech recognition.

[0034] In an optional embodiment, performing noise elimination processing on the speech signal acquired in real time to obtain a denoised speech signal includes: Based on the preset filter, the environmental noise is removed from the speech signal acquired in real time by the microphone array to obtain a noise-reduced speech signal; Determining the position information of each sound source in the speech signal based on the time difference between each microphone in the microphone array receiving the sound; Recognize the cab video stream acquired in real time by the camera to obtain visual recognition results; Based on the position information of each sound source and the visual recognition result, the external voice signal is filtered to obtain a denoised voice signal.

[0035] In an embodiment of the present invention, a microphone array is a system composed of multiple microphones, which are placed in a certain geometric arrangement and collect sound signals from different positions at the same time. After the voice signal is collected in real time by the microphone array, a filter is used to remove the environmental noise interference in the real-time collected voice signal to improve the clarity of the target sound source. After ensuring the time synchronization of all microphones, the location information of the sound source is determined based on the time difference of the sound received by each microphone, so as to distinguish different sound sources.

[0036] In an embodiment of the present invention, the cab video stream acquired in real time is identified and tracked, one or more sound-emitting objects are determined, multiple sound sources determined based on the voice signal are matched with multiple sound-emitting objects determined based on the video data, and it is further determined whether each sound source is a person making a sound in the cab, thereby retaining only the sound signal in the cab and excluding the noise outside the vehicle.

[0037] Optionally, the camera is installed on the upper part of the windshield of the cab, thereby ensuring that the lip-shaped image of the occupants in the cab can be captured.

[0038] Preferably, the camera adopts a high frame rate camera to adapt to lip recognition in a dynamic environment.

[0039] In the prior art, commercial vehicles usually have a large cockpit space, and the engine noise, wind noise and road noise generated during driving are large, which directly affects the accuracy of voice recognition. The recognition accuracy of traditional single voice recognition systems is greatly reduced in these environments, especially when driving at high speeds or under special road conditions; in addition, the current in-vehicle voice recognition system cannot effectively distinguish the sound sources inside and outside the vehicle when receiving instructions, resulting in the risk of misoperation or other dangers. The in-vehicle voice recognition method provided by the embodiment of the present invention removes environmental noise interference through a built-in microphone array, a high-definition camera and a signal processing algorithm, retains only the sound signal in the cab, excludes the noise outside the vehicle, provides a reliable voice signal for subsequent voice recognition, improves the accuracy of subsequent voice recognition, and avoids the risk of misoperation.

[0040] In an optional embodiment, the lip reading recognition result is obtained based on the following steps: Perform facial region recognition on the cab video stream acquired in real time to obtain the facial region in each frame image; Extracting the mouth of the face in each frame of the image to obtain a mouth region of interest; The oral region of interest is input into a pre-trained lip reading recognition model to obtain a lip reading recognition result output by the lip reading recognition model, wherein the lip reading recognition model includes a convolutional neural network and a long short-term memory network.

[0041] In the embodiment of the present invention, face detection and tracking are performed on the real-time collected cab video stream to ensure that the mouth can be stably captured even if the head moves. The specific detection and tracking algorithm used is not limited here.

[0042] Furthermore, after the facial region is detected, the region of interest (ROI) around the mouth is segmented from each frame, namely the mouth ROI, including the upper and lower lips and part of the visible teeth.

[0043] Optionally, after the mouth region of interest of each frame is separated, all frames are resized and normalized so that the images input into the lip reading recognition model are consistent.

[0044] In an embodiment of the present invention, the lip reading recognition model is a model for identifying the lip shape state of the oral area of ​​interest. Here, a convolutional neural network (CNN) is used to extract oral features. The specific CNN architecture used is not limited here. For example, VGG, ResNet, or a small CNN designed specifically for lip reading can be used. After several layers of convolution and pooling operations, a series of high-dimensional feature maps are output to represent the lip shape state at each time point. Afterwards, the feature maps extracted by CNN are arranged into a sequence in chronological order as the input of the Long Short-Term Memory (LSTM) network. A fully connected layer or softmax layer is added after the LSTM to output the final category probability distribution, thereby determining the lip shape at each time point.

[0045] The in-vehicle speech recognition method provided by the embodiment of the present invention combines the convolutional neural network CNN and the long short-term memory network LSTM, which can effectively parse the speech content through the lip image, thereby improving the accuracy and adaptability of the speech recognition system.

[0046] In an optional embodiment, the generating of vehicle control instructions based on the speech source and intention of each of the speech respectively includes: If the intentions of the voices do not interfere with each other, corresponding vehicle control instructions are generated based on the intentions of the voices.

[0047] Figure 3 : is a flow chart of the driver and co-pilot voice response provided by the present invention, such as Figure 3 As shown, taking the example of the sound source including the main driver's voice and the co-driver's voice, when the contents of the main driver's voice and the co-driver's voice do not interfere with each other, that is, when the intention of the main driver's voice does not contradict the intention of the co-driver's voice, corresponding vehicle control instructions are generated based on the intention of the main driver's voice and the intention of the co-driver's voice, respectively, and the vehicle control instructions of both are output at the same time to respectively realize the control of corresponding functions. For example, while the main driver sets the navigation route by voice, the co-driver adjusts the in-vehicle entertainment system. The intention of the main driver's voice and the intention of the co-driver's voice do not interfere with each other, then the corresponding vehicle control instruction for setting the navigation route is generated based on the main driver's voice, and the corresponding vehicle control instruction for adjusting the entertainment system is generated based on the co-driver's voice, and the two vehicle control instructions are output at the same time to control the navigation system and the entertainment system at the same time.

[0048] The in-vehicle voice recognition method provided by the embodiment of the present invention generates vehicle control instructions for voices that do not interfere with each other, meets the use requirements of multiple people issuing voice commands at the same time, and improves the user experience.

[0049] In an optional embodiment, the generating of vehicle control instructions based on the speech source and intention of each of the speech respectively includes: If the intentions of the voices interfere with each other, all the intentions are separated to obtain interfering intentions and non-interfering intentions, wherein the interfering intentions are intentions with an interfering relationship, and the non-interfering intentions are intentions without an interfering relationship; For the interference intention, generating corresponding vehicle control instructions based on the priority of the corresponding voice source; For the non-interference intentions, corresponding vehicle control instructions are generated respectively.

[0050] like Figure 3 As shown, still taking the example of the sound source including the main driver's voice and the co-driver's voice, when the contents of the main driver's voice and the co-driver's voice interfere with each other, the intention is divided into an interfering part and a non-interfering part. For the non-interfering part, the vehicle control instructions are output at the same time, and for the interfering part, the vehicle control instructions of the party with higher priority are output. For example, when the main driver sets the navigation path and the air-conditioning system by voice, and the co-driver adjusts the navigation path and the entertainment system by voice, there is interference between the intention of the main driver's voice and the intention of the co-driver's voice. The intentions of the two are separated to obtain the interfering intention (navigation path setting) and the non-interfering intention (setting the air-conditioning system and adjusting the entertainment system): for the non-interfering intention, the corresponding vehicle control instructions are generated respectively, that is, the vehicle control instructions for setting the air-conditioning system and the vehicle control instructions for adjusting the entertainment system are generated respectively; for the interfering intention, assuming that the priority of the main driver is higher than that of the co-driver, the corresponding vehicle control instructions are generated based on the intention of the main driver's voice.

[0051] It is understandable that the priority of the voice source is pre-set. Specifically, the same priority can be set for all control instructions, for example, all control instructions are set to have a higher priority for the main driver; or different priorities can be set for different control instructions, for example, for the control of a part of the system, the main driver is set to have a higher priority, and for the control of another part of the system, the co-driver is set to have a higher priority. Preferably, the user can set the control priority of each system according to actual usage needs.

[0052] The in-vehicle voice recognition method provided by the embodiment of the present invention divides the intention into an interfering part and a non-interfering part, and outputs control statements for the non-interfering part at the same time to meet the user's needs of controlling different systems at the same time. For the interfering part, control statements are output according to priority to avoid erroneous responses to voice commands.

[0053] The vehicle-mounted speech recognition device provided by an embodiment of the present invention is described below. The vehicle-mounted speech recognition device described below and the vehicle-mounted speech recognition method described above can be referenced to each other.

[0054] Figure 4 : is a schematic diagram of the structure of the vehicle-mounted speech recognition device provided by the present invention, such as Figure 4 As shown, the vehicle-mounted voice recognition device may include but is not limited to; The multimodal recognition module 410 is used to: fuse the speech recognition result and the lip reading recognition result of the cab to obtain a multimodal recognition result, wherein the speech recognition result is obtained by recognizing the speech signal collected in real time, and the lip reading recognition result is obtained by recognizing the lip shape image collected in real time; The speech analysis module 420 is used to analyze the multimodal recognition result to obtain the speech source of each speech and the intention corresponding to each speech; The instruction generation module 430 is used to generate vehicle control instructions based on the speech source and intention of each of the speech.

[0055] It should be noted that the vehicle-mounted speech recognition device provided in the embodiment of the present invention can execute the vehicle-mounted speech recognition method described in any of the above embodiments during specific operation, which will not be elaborated in this embodiment.

[0056] Figure 5 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 5 As shown, the electronic device may include: a processor 510, a communications interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communications interface 520 and the memory 530 communicate with each other through the communication bus 540. The processor 510 may call the logic instructions in the memory 530 to execute the vehicle-mounted speech recognition method, which includes: fusing the speech recognition result and the lip reading recognition result of the cab to obtain a multimodal recognition result, wherein the speech recognition result is obtained by recognizing the speech signal collected in real time, and the lip reading recognition result is obtained by recognizing the lip shape image collected in real time; Analyze the multimodal recognition results to obtain the speech source of each speech and the intention corresponding to each speech; Based on the speech source and intention of each of the speech, vehicle control instructions are generated respectively.

[0057] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0058] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored in a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the vehicle-mounted speech recognition method provided by the above methods, the method comprising: fusing the speech recognition result and the lip reading recognition result of the cab to obtain a multimodal recognition result, the speech recognition result is obtained by recognizing the real-time collected speech signal, and the lip reading recognition result is obtained by recognizing the real-time collected lip shape image; Analyze the multimodal recognition results to obtain the speech source of each speech and the intention corresponding to each speech; Based on the speech source and intention of each of the speech, vehicle control instructions are generated respectively.

[0059] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the vehicle-mounted speech recognition method provided by the above-mentioned methods, the method comprising: fusing the speech recognition result and the lip reading recognition result of the cab to obtain a multimodal recognition result, wherein the speech recognition result is obtained by recognizing a speech signal collected in real time, and the lip reading recognition result is obtained by recognizing a lip shape image collected in real time; Analyze the multimodal recognition results to obtain the speech source of each speech and the intention corresponding to each speech; Based on the speech source and intention of each of the speech, vehicle control instructions are generated respectively.

[0060] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0061] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0062] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A vehicle-mounted speech recognition method, characterized in that: include: The speech recognition result and the lip reading recognition result of the cab are integrated to obtain a multimodal recognition result, wherein the speech recognition result is obtained by recognizing the speech signal collected in real time, and the lip reading recognition result is obtained by recognizing the lip shape image collected in real time; Analyze the multimodal recognition results to obtain the speech source of each speech and the intention corresponding to each speech; Based on the speech source and intention of each of the speech, vehicle control instructions are generated respectively.

2. The vehicle-mounted speech recognition method according to claim 1, characterized in that: The speech recognition result is obtained based on the following steps: Performing noise elimination processing on the speech signal collected in real time to obtain a denoised speech signal; The denoised speech signal is analyzed using a preset speech recognition model to obtain a speech recognition result, wherein the speech recognition result includes a speech signal recognition result of each speaker.

3. The vehicle-mounted speech recognition method according to claim 2, characterized in that: The method of performing noise elimination processing on the speech signal collected in real time to obtain a denoised speech signal includes: Based on the preset filter, the environmental noise is removed from the speech signal acquired in real time by the microphone array to obtain a noise-reduced speech signal; Determining the position information of each sound source in the speech signal based on the time difference between each microphone in the microphone array receiving the sound; Recognize the cab video stream acquired in real time by the camera to obtain visual recognition results; Based on the position information of each sound source and the visual recognition result, the external voice signal is filtered to obtain a denoised voice signal.

4. The vehicle-mounted speech recognition method according to claim 1, characterized in that: The lip reading recognition result is obtained based on the following steps: Perform facial region recognition on the cab video stream acquired in real time to obtain the facial region in each frame image; Extracting the mouth of the face in each frame of the image to obtain a mouth region of interest; The oral region of interest is input into a pre-trained lip reading recognition model to obtain a lip reading recognition result output by the lip reading recognition model, wherein the lip reading recognition model includes a convolutional neural network and a long short-term memory network.

5. The vehicle-mounted speech recognition method according to claim 1, characterized in that: The generating of vehicle control instructions based on the speech source and intention of each of the speech respectively includes: If the intentions of the voices do not interfere with each other, corresponding vehicle control instructions are generated based on the intentions of the voices.

6. The vehicle-mounted speech recognition method according to claim 1, characterized in that: The generating of vehicle control instructions based on the speech source and intention of each of the speech respectively includes: If the intentions of the voices interfere with each other, all the intentions are separated to obtain interfering intentions and non-interfering intentions, wherein the interfering intentions are intentions with an interfering relationship, and the non-interfering intentions are intentions without an interfering relationship; For the interference intention, generating corresponding vehicle control instructions based on the priority of the corresponding voice source; For the non-interference intentions, corresponding vehicle control instructions are generated respectively.

7. A vehicle-mounted speech recognition device, characterized in that: include: The multimodal recognition module is used to: fuse the speech recognition result and the lip reading recognition result of the cab to obtain a multimodal recognition result, wherein the speech recognition result is obtained by recognizing the speech signal collected in real time, and the lip reading recognition result is obtained by recognizing the lip shape image collected in real time; A speech analysis module is used to: analyze the multimodal recognition results to obtain the speech source of each speech and the intention corresponding to each speech; The instruction generation module is used to generate vehicle control instructions based on the speech source and intention of each of the speech.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the in-vehicle speech recognition method according to any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the in-vehicle speech recognition method according to any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the in-vehicle speech recognition method according to any one of claims 1 to 6 is implemented.

Images