Multimodal interaction method, storage medium and electronic device for vehicle

Abstract

The invention provides a multi-modal interaction method, storage medium and electronic equipment for vehicles. In the multi-modal interaction method, natural language processing and recognition can be performed on voice signals in the vehicle to obtain text-corresponding intention descriptions and text At the same time, the action recognition can be performed on the in-vehicle image to obtain the action type, and then based on the intention description corresponding to the text and the intention description corresponding to the action type, fusion processing is performed to obtain the fused intention. Using the present invention can judge the user's intention through the action type after the action recognition of the in-vehicle image on the basis of voice interaction, so that the vehicle-machine voice interaction system can integrate the multi-modality of voice and action to understand the user's intention, and then Understand user intent more accurately to better interact with users and improve user experience.

Description

technical field [0001] The invention relates to the technical field of artificial intelligence algorithms, in particular to a multi-modal interaction method for vehicles, a storage medium and electronic equipment. Background technique [0002] The existing vehicle-machine voice interaction system generally interacts with users through voice, which is mainly composed of a voice recognition module, a natural language processing module, a dialogue management module, a command intent processing module, a reply generation module, and a text-to-speech module. [0003] When the speech recognition module performs speech recognition processing on the speech signal in the car, since there are usually various noises in the car, such as road noise, wind noise, engine and other mechanical component noise, etc., these noises will be superimposed on the voice commands issued by the user. It is easy to cause the recognition error of the voice recognition module, which will cause the in-vehi...

AI Technical Summary

Problems solved by technology

[0003] When the speech recognition module performs speech recognition processing on the speech signal in the car, since there are usually various noises in the car, such as road noise, wind noise, engine and other mechanical component noise, etc., these noises will be superimposed on the voice commands issued by the user. It is easy to cause the recognition error of the speech recognition module, which will cause the in-vehicle voice interaction system to fail to understand the user's intention correctly, and the user experience is poor

In addition, in addition to noise, there are other sounds in the car, such as conversation sounds, music played by mobile phones, and other background voices that have nothing to do with user command voices. In this way, the car-machine voice interaction system communicates with users During voice interaction, the user's command voice and background voice cannot be accurately distinguished, which will also cause the vehicle-machine voice interaction system to fail to understand the user's intentions correctly, reducing the user experience

Images