Vehicle voice assistant wake-up method and apparatus, and electronic device

Abstract

Disclosed in the present application are a vehicle voice assistant wake-up method and apparatus, and an electronic device. The method comprises: acquiring image information of a user; recognizing a wake-up gesture of the user from the acquired image information on the basis of a first machine learning model, and recognizing a gaze of the user on an effective wake-up area from the acquired image information on the basis of a second machine learning model; and waking up a vehicle voice assistant in response to the recognized wake-up gesture and gaze. A wake-up gesture and a gaze of a user are recognized from image information respectively by means of two trained machine learning models, it can be determined in response to the recognized wake-up gesture and gaze that the user is required to wake up a vehicle voice assistant, and the vehicle voice assistant is then woken up. By means of the wake-up method of the present solution, a vehicle voice assistant is not required to be woken up by means of a manual button, thereby improving the convenience of waking up the vehicle voice assistant, and the accuracy of waking up the vehicle voice assistant can be improved by means of a wake-up method involving two indexes, i.e., a wake-up gesture and a gaze.

Description

Method, device and electronic device for waking up a vehicle voice assistant

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to the Chinese patent application filed with the China Patent Office on August 3, 2023, with application number 202310973114.3 and application name “A method, device and electronic device for waking up a vehicle voice assistant”, the entire contents of which are incorporated by reference into this application. Technical Field

[0003] The present application relates to the field of vehicle technology, and in particular to a method, device, electronic device, and computer storage medium for waking up a vehicle voice assistant. Background Art

[0004] The vehicle voice assistant can control various vehicle functions such as navigation, music, and phone calls through voice commands. This technology allows drivers to complete various operations through voice commands without manual operation during driving, improving driving safety and convenience.

[0005] The vehicle's voice assistant is usually in sleep mode and needs to be woken up before voice control. The usual wake-up operations are divided into voice wake-up and manual wake-up.

[0006] Manually waking up the vehicle's voice assistant requires the owner to manually press a specific voice assistant button to start it, and then control various vehicle functions through voice commands. This technology requires the owner to manually press a specific button, which is not convenient enough.

[0007] Voice-activated vehicle voice assistants are systems in which the vehicle's built-in voice recognition device can be activated through voice commands, without the driver having to manually operate the device, to control various vehicle functions. For example, a driver can activate the vehicle's voice assistant by saying a voice command like "Hey, car," and then control the vehicle's navigation, music, phone, and other functions through voice commands. The disadvantage of vehicle voice-activated voice assistants is that the vehicle's built-in voice recognition device is prone to false activation (caused by voice recognition errors) when used alone to activate the vehicle, which impacts the user experience.

[0008] Summary of the Invention

[0009] The purpose of this application is to solve one of the above technical problems at least to a certain extent.

[0010] To this end, the first purpose of this application is to propose a method for waking up a vehicle voice assistant. This method uses two trained machine learning models to recognize the user's wake-up gestures and gazes, and responds to wake up the vehicle voice assistant, which can improve the convenience and accuracy of waking up the vehicle voice assistant.

[0011] To achieve the above-mentioned objectives, the first aspect of the present application proposes a method for waking up a vehicle voice assistant, which includes: obtaining image information of the user; identifying the user's wake-up gesture from the obtained image information based on a first machine learning model, and identifying the user's gaze on the effective wake-up area from the obtained image information based on a second machine learning model; and waking up the vehicle voice assistant in response to the identified wake-up gesture and the gaze.

[0012] According to the wake-up method of the vehicle voice assistant of the embodiment of the present application, two trained machine learning models are used to respectively identify the user's wake-up gesture and gaze from the image information. In response to the identified wake-up gesture and gaze, it can be determined that the user needs to wake up the vehicle voice assistant, and then the vehicle voice assistant is woken up. The wake-up method of this scheme does not require manual button pressing to wake up the vehicle voice assistant, which can improve the convenience of waking up the vehicle voice assistant. On the other hand, through the wake-up method of the dual indicators of wake-up gesture and gaze, the accuracy of waking up the vehicle voice assistant can be improved.

[0013] According to one embodiment of the present application, the method further includes:

[0014] Get the user's audio information;

[0015] Extracting a user's gesture representation from the acquired image information based on the first machine learning model, extracting a user's gaze representation from the acquired image information based on the second machine learning model, and extracting a user's voice representation from the acquired audio information based on the third machine learning model;

[0016] If the vehicle voice assistant is not awakened, the method further includes:

[0017] Concatenating the gesture representation, the gaze representation, and the speech representation to form an embedded representation time series of multimodal heterogeneous sources;

[0018] Using the embedded representation time series as an input parameter of a fourth trained machine learning model to predict whether the user wakes up the vehicle voice assistant;

[0019] If the fourth trained machine learning model predicts that the user wakes up the vehicle voice assistant, wake up the vehicle voice assistant.

[0020] According to one embodiment of the present application,

[0021] The embedded representation time series of the multimodal heterogeneous source includes embedded representations of multiple time steps. The gesture representation, the gaze representation, and the voice representation are concatenated to form the embedded representation time series of the multimodal heterogeneous source, including:

[0022] The gesture representation, the gaze representation, and the voice representation of each time step are concatenated in chronological order to form a vector as the embedded representation of the time step.

[0023] According to one embodiment of the present application, if the fourth trained machine learning model predicts that the user wakes up the vehicle voice assistant, the method further includes:

[0024] Execute the instruction action corresponding to the audio information.

[0025] According to one embodiment of the present application, the effective awakening area includes the front area of ​​the vehicle and the center control area, and the identifying the user's gaze on the effective awakening area from the acquired image information based on the second machine learning model includes:

[0026] Identify the area where the user is located based on the acquired image information, wherein the area includes a driver area and a passenger area;

[0027] If the area where the user is located is identified as the driver area, identifying, based on the second machine learning model, from the acquired image information, the user's gaze on the area in front of the vehicle or the central control area as the gaze of the effective awakening area;

[0028] If the area where the user is located is identified as the passenger area, based on the second machine learning model, the user's gaze on the central control area is identified from the acquired image information as the gaze of the effective awakening area.

[0029] According to one embodiment of the present application, the first machine learning model is a machine learning model based on a spatiotemporal convolution transformer.

[0030] According to one embodiment of the present application, the second machine learning model is a machine learning model based on a deep convolutional neural network.

[0031] According to one embodiment of the present application, the third machine learning model is a machine learning model based on an automatic speech recognition acoustic model;

[0032] The fourth trained machine learning model is a machine learning model based on a long short-term memory network.

[0033] To achieve the above-mentioned purpose, the second embodiment of the present application proposes a vehicle voice assistant wake-up device, the device comprising:

[0034] An image acquisition unit, used to acquire image information of a user;

[0035] a recognition unit, configured to recognize a user's wake-up gesture from the acquired image information based on a first machine learning model, and recognize a user's gaze on an effective wake-up area from the acquired image information based on a second machine learning model;

[0036] A first wake-up unit is configured to wake up the vehicle voice assistant in response to the recognized wake-up gesture and the gaze.

[0037] According to one embodiment of the present application, the device further includes:

[0038] An audio acquisition unit, used to acquire audio information of the user;

[0039] a representation extraction unit, configured to extract a user's gesture representation from the acquired image information based on a first machine learning model, extract a user's gaze representation from the acquired image information based on a second machine learning model, and extract a user's voice representation from the acquired audio information based on a third machine learning model;

[0040] If the vehicle voice assistant is not awakened, the method further includes:

[0041] a representation concatenation unit, configured to concatenate the gesture representation, the gaze representation, and the speech representation to form an embedded representation time sequence of a multimodal heterogeneous source;

[0042] a prediction unit, configured to use the embedded representation time series as an input parameter of a fourth trained machine learning model to predict whether the user wakes up the vehicle voice assistant;

[0043] The second wake-up unit is used to wake up the vehicle voice assistant if the fourth trained machine learning model predicts that the user wakes up the vehicle voice assistant.

[0044] According to one embodiment of the present application, the embedded representation time series of a multimodal heterogeneous source includes embedded representations of multiple time steps, and the representation concatenation unit includes: a representation concatenation module, which is used to concatenate the gesture representation, the gaze representation, and the voice representation of each time step in chronological order to form a vector as the embedded representation of the time step.

[0045] According to one embodiment of the present application, if the fourth trained machine learning model predicts that the user wakes up the vehicle voice assistant, the device further includes: an execution unit for executing the instruction action corresponding to the audio information.

[0046] According to one embodiment of the present application, the effective wake-up area includes the front area and the central control area of ​​the vehicle, and the recognition unit includes:

[0047] An identification module is used to identify the area where the user is located based on the acquired image information, wherein the area includes a driver area and a passenger area;

[0048] If the area where the user is located is identified as the driver area, identifying, based on the second machine learning model, from the acquired image information, the user's gaze on the area in front of the vehicle or the central control area as the gaze of the effective awakening area;

[0049] If the area where the user is located is identified as the passenger area, based on the second machine learning model, the user's gaze on the central control area is identified from the acquired image information as the gaze of the effective awakening area.

[0050] According to one embodiment of the present application, the first machine learning model is a machine learning model based on a spatiotemporal convolution transformer.

[0051] According to one embodiment of the present application, the second machine learning model is a machine learning model based on a deep convolutional neural network.

[0052] According to one embodiment of the present application, the third machine learning model is a machine learning model based on an automatic speech recognition acoustic model; the fourth trained machine learning model is a machine learning model based on a long short-term memory network.

[0053] To achieve the above-mentioned purpose, the electronic device proposed in the third embodiment of the present application includes: a memory, a processor, and a computer program stored on the memory and runnable on the processor. When the processor executes the computer program, the vehicle voice assistant wake-up method described in the first embodiment of the present application is implemented.

[0054] In order to achieve the above-mentioned purpose, the computer-readable storage medium proposed in the fourth embodiment of the present application, when the computer program is executed by the processor, implements the vehicle voice assistant wake-up method described in the first embodiment of the present application.

[0055] Additional aspects and advantages of the present application will be given in part in the description below, and in part will become apparent from the description below, or will be learned through practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0057] FIG1 is a flow chart of a method for waking up a vehicle voice assistant according to one embodiment of the present application;

[0058] FIG2 is a flow chart of a first decision-making system in a method for waking up a vehicle voice assistant according to another embodiment of the present application;

[0059] FIG3 is a flow chart of a second decision-making system in a method for waking up a vehicle voice assistant according to another embodiment of the present application;

[0060] FIG4 is a schematic diagram of a wake-up device for a vehicle voice assistant according to one embodiment of the present application;

[0061] FIG5 is a schematic diagram of a vehicle voice assistant wake-up device according to another embodiment of the present application;

[0062] FIG6 is a schematic structural diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0063] The following describes in detail embodiments of the present application, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are intended to be used to explain the present application, and should not be construed as limiting the present application.

[0064] To this end, the present application proposes a method, device, electronic device and storage medium for waking up a vehicle voice assistant.

[0065] Specifically, the following describes a method, device, electronic device, and storage medium for waking up a vehicle voice assistant according to an embodiment of the present application with reference to the accompanying drawings.

[0066] FIG1 is a flow chart of a method for waking up a vehicle voice assistant according to an embodiment of the present application. It should be noted that the method for waking up a vehicle voice assistant according to an embodiment of the present application can be applied to a device for waking up a vehicle voice assistant according to an embodiment of the present application. The device can be configured on an electronic device or on a server. The electronic device can be a vehicle, an in-vehicle terminal, or a mobile terminal. This embodiment of the present application does not limit this.

[0067] As shown in Figure 1, the vehicle voice assistant wake-up method includes:

[0068] S110, obtaining user image information;

[0069] Taking the example of a car camera collecting user image information, the car camera can capture a video sequence of the user's gestures and transmit it to the first machine learning model. The car camera can also capture a video sequence of the user's face and transmit it to the second machine learning model.

[0070] S120: Identify a user's wake-up gesture from the acquired image information based on the first machine learning model, and identify the user's gaze on the effective wake-up area from the acquired image information based on the second machine learning model;

[0071] In wake-up gesture recognition: This solution adopts a method based on deep learning. In actual specific applications, the first machine learning model can be a machine learning model based on the Space Time Convolutional Transformer (ST-CvT) for dynamic gesture recognition. The ST-CvT model is a new end-to-end deep architecture that unifies 3D convolutional neural networks and transformers. It can simultaneously extract spatial features and temporal features in video sequences, using convolutional layers and transformer layers to capture the shape and motion changes of gestures, and then using fully connected layers and softmax layers to output classification results. Transformer is a neural network that can process sequential data. It can effectively capture long-distance dependencies and self-attention mechanisms in sequences. ST-CvT takes advantage of the advantages of transformers, treating the input in each time window in the video sequence as a token, and then uses transformer layers to update the feature representation of the token, thereby enhancing the fusion and expression capabilities of spatiotemporal features.

[0072] Before the first machine learning model recognizes, the image and video sequence of the user's gestures can be preprocessed: the image and video sequence of the user's gestures can be preprocessed through image processing technology, including cropping, scaling, grayscale, etc., to reduce the computational complexity of the first machine learning model and improve recognition efficiency.

[0073] To improve recognition accuracy and robustness, this solution employs data augmentation in the processing of training data for the spatiotemporal convolutional transformer (SCT). Furthermore, multi-task learning techniques are employed during the training phase of the SCT model, enabling the SCT model to better focus on key aspects of gestures and adapt to different scenarios. After inference using the SCT model, the output classification results represent either a detected wake-up gesture or no wake-up gesture.

[0074] It can be understood that the specific gesture method of the wake-up gesture can be defined arbitrarily. For ease of understanding, in the embodiment of the present application, the wake-up gesture is a waving gesture, which is specifically defined as the user's palm facing forward and waving left and right two or more times. The recognition success rate and accuracy of this method are relatively high.

[0075] In the identification of gaze on the effective awakening area: This method uses an artificial neural network, and the second machine learning model can be a machine learning model based on a deep convolutional neural network (DCNN). The DCNN model extracts features from the image video sequence of the user's face and uses a regression layer to predict the gaze direction angle, thereby identifying the user's gaze on the effective awakening area. The deep convolutional neural network structure includes a convolution layer, a pooling layer, an activation layer, and a fully connected layer. The model used in this step was previously trained using a gradient descent optimization method on a large amount of labeled data through supervised learning. DCNN model reasoning: This method does not require the step of facial key point detection, and can directly estimate the gaze direction from the original image and output the angle. DCNN model post-processing: Combined with the previous calibration results, the user's current gaze area is determined based on the gaze angle. A simple threshold judgment method is used in this method to divide the boundary value of the area to be judged through the calibration data, and then the current gaze area is obtained based on the gaze angle output in the previous step, thereby identifying the user's gaze on the effective awakening area.

[0076] Before the first machine learning model recognizes, the image video sequence of the user's face can be preprocessed: the image video sequence of the input user's face can be preprocessed, including cropping the image to adapt to the model input size, and normalizing and standardizing the image to reduce the computational complexity of the second machine learning model and improve recognition efficiency.

[0077] It can be understood that the gaze recognition method of the effective wake-up area can adopt different recognition technologies. For ease of understanding, in the embodiment of the present application, the gaze of the effective wake-up area refers to judging the user's gaze direction by the angle of the user's head.

[0078] S130: In response to the recognized wake-up gesture and the gaze, wake up the vehicle voice assistant.

[0079] That is, if the user is recognized to have a wake-up gesture and gaze at the effective wake-up area at the same time, the vehicle voice assistant will be woken up.

[0080] During implementation, the following steps can be repeated cyclically:

[0081] a. Obtain the wake-up gesture recognized at the current moment and the gaze result of the valid wake-up area.

[0082] b. Update the latest recognized wake-up gesture and gaze result of the valid wake-up area into two queues of length x respectively. For example, x can be 10.

[0083] c. Traverse the two queues. If there is both a wake-up gesture and gaze on the valid wake-up area in the queue, it is considered a valid wake-up signal, triggering the voice assistant to enter the wake-up state and wait for the user's instructions.

[0084] d. If there is no valid wake-up signal, continue to wait for the next moment's information.

[0085] e. If the voice assistant has entered the awake state, this algorithm will not be executed until the voice assistant returns to the awakening state.

[0086] According to the wake-up method of the vehicle voice assistant of the embodiment of the present application, two trained machine learning models are used to respectively identify the user's wake-up gesture and gaze from the image information. In response to the identified wake-up gesture and gaze, it can be determined that the user needs to wake up the vehicle voice assistant, and then the vehicle voice assistant is woken up. The wake-up method of this scheme does not require manual button pressing to wake up the vehicle voice assistant, which can improve the convenience of waking up the vehicle voice assistant. On the other hand, through the wake-up method of the dual indicators of wake-up gesture and gaze, the accuracy of waking up the vehicle voice assistant can be improved.

[0087] When training machine learning models, some categories have very few samples, yet these few samples make up a significant portion of the entire dataset. These small numbers of samples are often referred to as "long tails," and low-frequency long tails (LFELTs) appear because they represent a very small proportion of the entire dataset, yet their presence has a significant impact on model training and performance. Furthermore, because they represent special cases within the dataset, they are difficult for the model to accurately identify and classify. In practical applications, low-frequency long tail events often lead to decreased model performance, as the model struggles to learn effective features and patterns from these small numbers of samples. This can result in poor wake-up performance for vehicle voice assistants, so methods for addressing low-frequency long tail events are crucial.

[0088] To solve the above problems, FIG2 is a flow chart of a first decision-making system in a method for waking up a vehicle voice assistant according to another embodiment of the present application; FIG3 is a flow chart of a second decision-making system in a method for waking up a vehicle voice assistant according to another embodiment of the present application; as shown in FIG2 and FIG3, the method for waking up a vehicle voice assistant includes:

[0089] Get the user's image information;

[0090] Recognize the user's wake-up gesture from the acquired image information based on the first machine learning model, and recognize the user's gaze on the effective wake-up area from the acquired image information based on the second machine learning model;

[0091] Get the user's audio information;

[0092] Based on the first machine learning model, the user's gesture representation is extracted from the acquired image information, based on the second machine learning model, the user's gaze representation is extracted from the acquired image information, and based on the third machine learning model, the user's voice representation is extracted from the acquired audio information; specifically, the third machine learning model is a machine learning model based on the Automatic Speech Recognition Acoustic Model (ASR).

[0093] In response to the recognized wake-up gesture and the gaze, a vehicle voice assistant is woken up.

[0094] If the vehicle voice assistant is not awakened, the method further includes:

[0095] The gesture representation, the gaze representation, and the voice representation are concatenated to form a time series of embedded representations of a multimodal heterogeneous source. Specifically, the time series of embedded representations of a multimodal heterogeneous source includes embedded representations of multiple time steps. During the concatenation, the gesture representation, the gaze representation, and the voice representation of each time step are concatenated in chronological order to form a vector as the embedded representation of the time step.

[0096] Using the embedded representation time series as an input parameter of a fourth trained machine learning model to predict whether the user wakes up the vehicle voice assistant;

[0097] If the fourth trained machine learning model predicts that the user wakes up the vehicle voice assistant, wake up the vehicle voice assistant.

[0098] According to the wake-up method of the vehicle voice assistant of the embodiment of the present application, two decision-making systems are used. The first decision-making system responds to the recognized wake-up gesture and the gaze at the effective wake-up area to wake up the vehicle voice assistant. The logic of this decision-making system is clear and the response is fast. The main function of the second decision-making system is to process low-frequency long-tail events that the first decision-making system cannot handle, so as to further improve the effect of the overall system. When the first decision-making system encounters a low-frequency long-tail event and fails to wake up the vehicle voice assistant, when the user subsequently makes a voice request, the second decision-making system will input the user gesture representation (embedding) in the past period of time, the gaze representation, and the voice representation into the fourth trained machine learning model for decision-making. This dual decision-making method can effectively handle complex situations that the first decision-making system cannot handle, and improve the overall efficiency and accuracy of the vehicle voice assistant.

[0099] Specifically, in the vehicle voice assistant wake-up method of the embodiment of the present application, the message middleware obtains wake-up gestures from users in different seats in the vehicle and the result of the users' gaze on the effective wake-up area at a certain frequency, such as 10 frames per second, and stores this information in memory for invocation by the first decision-making system. Simultaneously, the message middleware obtains the extracted gesture representations, gaze representations, and voice representations at a certain frequency, such as 10 frames per second, and stores this information in memory for invocation by the second decision-making system.

[0100] The fourth trained machine learning model can be a machine learning model based on the Long Short-Term Memory Model (LSTM). The LSTM model is used to integrate multimodal time series representation input to make decisions on waking up the vehicle voice assistant. That is, through training, it is possible to predict whether the user is waking up the assistant with a gesture based on multimodal time series input. In the training data, the positive and negative samples are constructed as follows: the positive samples include the positive samples of the first decision system that successfully wake up the vehicle voice assistant and the positive samples that are not successfully woken up by the first decision system through offline mining and manual labeling, so as to improve the generalization effect of the second decision system relative to the first decision system; the negative samples are randomly sampled from the massive samples corresponding to all invalid voice requests generated online. This construction method can effectively improve the accuracy and generalization ability of the model.

[0101] Furthermore, if the fourth trained machine learning model predicts that the user has activated the vehicle voice assistant, the method further includes executing the command action corresponding to the audio information. Upon deciding to activate the vehicle voice assistant, the second decision-making system may directly execute the command action corresponding to the audio information, without the user having to further voice-control the vehicle voice assistant.

[0102] In this solution, the vehicle voice assistant wake-up method can be applied to occupants in different seats in the car, such as the driver in the driver's area and the passengers in the passenger area. The applicant found that for the driver, if the driver's gaze is taken away from the effective wake-up area of ​​the road and one hand is taken away from the steering wheel to perform a wake-up gesture at the same time, a sudden accident may occur on the road, which may easily lead to a traffic accident. In this regard, based on the vehicle voice assistant wake-up method described in any of the above embodiments, the effective wake-up area includes the front area of ​​the vehicle and the central control area. The front area of ​​the vehicle can be the area in the direction of the front windshield of the vehicle, and the central control area is usually the center position in front of the vehicle, on the lower side of the front windshield. The method of identifying the user's gaze on the effective wake-up area from the acquired image information based on the second machine learning model includes:

[0103] Identify the area where the user is located based on the acquired image information, wherein the area includes a driver area and a passenger area;

[0104] The user area is the area inside the vehicle where the user is located, the driver area is the area where the vehicle is driven, and the area outside the vehicle driving area is the passenger area.

[0105] If the area where the user is located is identified as the driver area, the second machine learning model is used to identify from the acquired image information that the user's gaze on the area in front of the vehicle or the central control area is a gaze in the effective wake-up area; if the area where the user is located is identified as the passenger area, the second machine learning model is used to identify from the acquired image information that the user's gaze on the central control area is a gaze in the effective wake-up area.

[0106] According to the vehicle voice assistant wake-up method of the embodiment of the present application, when the driver wakes up the vehicle voice assistant, he can wake up the vehicle voice assistant by looking at the central control area away from the road and making a wake-up gesture when parking is safe. When driving, in unsafe situations, he can also stay on the road, look at the area in front of the vehicle and make a wake-up gesture to wake up the vehicle voice assistant, thereby improving driving safety.

[0107] Figure 4 is a schematic diagram of a vehicle voice assistant wake-up device according to an embodiment of the present application; as shown in Figure 4, the vehicle voice assistant wake-up device includes: an image acquisition unit 10, a recognition unit 20 and a first wake-up unit 30.

[0108] An image acquisition unit 10 is used to acquire image information of a user;

[0109] A recognition unit 20 is configured to recognize a user's wake-up gesture from the acquired image information based on a first machine learning model, and recognize a user's gaze on an effective wake-up area from the acquired image information based on a second machine learning model;

[0110] The first awakening unit 30 is configured to awaken the vehicle voice assistant in response to the recognized awakening gesture and the gaze.

[0111] FIG5 is a schematic diagram of a vehicle voice assistant wake-up device according to another embodiment of the present application. As shown in FIG5 , according to one embodiment of the present application, the device further includes:

[0112] The audio acquisition unit 40 is used to acquire the user's audio information;

[0113] a representation extraction unit 50 for extracting a user's gesture representation from the acquired image information based on a first machine learning model, extracting a user's gaze representation from the acquired image information based on a second machine learning model, and extracting a user's voice representation from the acquired audio information based on a third machine learning model;

[0114] If the vehicle voice assistant is not awakened, the method further includes:

[0115] a representation concatenation unit 60, configured to concatenate the gesture representation, the gaze representation, and the speech representation to form an embedded representation time sequence of a multimodal heterogeneous source;

[0116] a prediction unit 70 for using the embedded representation time series as an input parameter of a fourth trained machine learning model to predict whether the user wakes up the vehicle voice assistant;

[0117] The second awakening unit 80 is used to wake up the vehicle voice assistant if the fourth trained machine learning model predicts that the user wakes up the vehicle voice assistant.

[0118] According to one embodiment of the present application, the embedded representation time series of the multimodal heterogeneous source includes embedded representations of multiple time steps, and the representation concatenation unit 50 includes: a representation concatenation module, which is used to concatenate the gesture representation, the gaze representation, and the voice representation of each time step in chronological order to form a vector as the embedded representation of the time step.

[0119] According to one embodiment of the present application, if the fourth trained machine learning model predicts that the user wakes up the vehicle voice assistant, the device also includes: an execution unit 90, which is used to execute the instruction action corresponding to the audio information.

[0120] According to one embodiment of the present application, the effective wake-up area includes the front area and the central control area of ​​the vehicle, and the recognition unit 20 includes:

[0121] An identification module is used to identify the area where the user is located based on the acquired image information, wherein the area includes a driver area and a passenger area;

[0122] If the area where the user is located is identified as the driver area, identifying, based on the second machine learning model, from the acquired image information, the user's gaze on the area in front of the vehicle or the central control area as the gaze of the effective awakening area;

[0123] If the area where the user is located is identified as the passenger area, based on the second machine learning model, the user's gaze on the central control area is identified from the acquired image information as the gaze of the effective awakening area.

[0124] According to one embodiment of the present application, the first machine learning model is a machine learning model based on a spatiotemporal convolution transformer.

[0125] According to one embodiment of the present application, the second machine learning model is a machine learning model based on a deep convolutional neural network.

[0126] According to one embodiment of the present application, the third machine learning model is a machine learning model based on an automatic speech recognition acoustic model; the fourth trained machine learning model is a machine learning model based on a long short-term memory network.

[0127] According to the apparatus of the embodiment of the present application, reference is made to FIG6 , which shows a schematic diagram of the structure of an electronic device 500 suitable for implementing the embodiment of the present application. The electronic devices in the embodiment of the present application may include, but are not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., as well as fixed terminals such as digital TVs, desktop computers, etc. The electronic device shown in FIG6 is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0128] As shown in FIG6 , the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. Various programs and data required for the operation of the electronic device 500 are also stored in the RAM 503. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0129] Typically, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although FIG. 6 illustrates the electronic device 500 with various devices, it should be understood that not all of the illustrated devices are required to be implemented or present. More or fewer devices may alternatively be implemented or present.

[0130] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network via the communication device 509, or installed from the storage device 508, or installed from the ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of the embodiment of the present application are performed.

[0131] It should be noted that the computer-readable medium mentioned above in this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor device, apparatus, or component, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this application, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution device, apparatus, or component. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. This propagated data signal can take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or convey a program for use by or in conjunction with an instruction execution apparatus, device, or device. The program code contained on the computer-readable medium may be conveyed using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0132] In some embodiments, the client and server can communicate using any currently known or later developed network protocol, such as HTTP (HyperText Transfer Protocol), and can be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or later developed network.

[0133] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0134] The above-mentioned computer-readable medium carries one or more programs. When the above-mentioned one or more programs are executed by the electronic device, the electronic device: obtains the user's image information; recognizes the user's wake-up gesture from the obtained image information based on the first machine learning model, and recognizes the user's gaze on the effective wake-up area from the obtained image information based on the second machine learning model; wakes up the vehicle voice assistant in response to the recognized wake-up gesture and the gaze.

[0135] Alternatively, the computer-readable medium carries one or more programs, which, when executed by the electronic device, enables the electronic device to: obtain image information of the user; identify the user's wake-up gesture from the obtained image information based on a first machine learning model, and identify the user's gaze on the effective wake-up area from the obtained image information based on a second machine learning model; and wake up the vehicle voice assistant in response to the identified wake-up gesture and the gaze.

[0136] Computer program code for performing the operations of the present application can be written in one or more programming languages, or a combination thereof, including, but not limited to, object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0137] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the devices, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based device that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0138] The units involved in the embodiments described in this application may be implemented in software or hardware. The name of a unit does not necessarily limit the unit itself. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses."

[0139] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip devices (SOCs), complex programmable logic devices (CPLDs), and the like.

[0140] In the context of the present application, a machine-readable medium can be a tangible medium that can contain or store a program for use with an instruction execution device, device or equipment or used in combination with an instruction execution device, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared or semiconductor devices, devices or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0141] The above description is merely a preferred embodiment of the present application and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of application involved in this application is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but also encompasses other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned application concept. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions applied for in this application.

[0142] In addition, although adopting specific order to describe each operation, this should not be interpreted as requiring these operations to be executed in the specific order shown or in sequential order.Under certain environment, multitasking and parallel processing may be advantageous.Similarly, although comprising some specific implementation details in the above discussion, these should not be interpreted as limiting the scope of the application.Some features described in the context of separate embodiment can also be implemented in a single embodiment in combination.On the contrary, the various features described in the context of a single embodiment also can be implemented in multiple embodiments individually or in the mode of any suitable subcombination.

[0143] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. A method for waking up a vehicle voice assistant, characterized in that: include: Get the user's image information; Recognize the user's wake-up gesture from the acquired image information based on the first machine learning model, and recognize the user's gaze on the effective wake-up area from the acquired image information based on the second machine learning model; In response to the recognized wake-up gesture and the gaze, a vehicle voice assistant is woken up.

2. The vehicle voice assistant wake-up method according to claim 1, further comprising: Get the user's audio information; Extracting a user's gesture representation from the acquired image information based on the first machine learning model, extracting a user's gaze representation from the acquired image information based on the second machine learning model, and extracting a user's voice representation from the acquired audio information based on the third machine learning model; If the vehicle voice assistant is not awakened, the method further includes: The gesture representation, the gaze representation, and the voice representation are concatenated to form an embedded representation time series of multimodal heterogeneous sources; Using the embedded representation time series as an input parameter of a fourth trained machine learning model to predict whether the user wakes up the vehicle voice assistant; If the fourth trained machine learning model predicts that the user will wake up the vehicle voice assistant, wake up the vehicle voice assistant.

3. The vehicle voice assistant wake-up method according to claim 2, wherein: The embedded representation time series of the multimodal heterogeneous source includes embedded representations of multiple time steps, and the gesture representation, the gaze representation, and the voice representation are concatenated to form the embedded representation time series of the multimodal heterogeneous source, including: The gesture representation, the gaze representation, and the voice representation of each time step are concatenated in chronological order to form a vector as the embedded representation of the time step.

4. The vehicle voice assistant wake-up method according to claim 2, wherein: If the fourth trained machine learning model predicts that the user wakes up the vehicle voice assistant, the method further includes: Execute the instruction action corresponding to the audio information.

5. The vehicle voice assistant wake-up method according to claim 1, wherein: The effective awakening area includes the front area of ​​the vehicle and the central control area, and the identifying the user's gaze on the effective awakening area from the acquired image information based on the second machine learning model includes: Identify the area where the user is located according to the acquired image information, wherein the area includes a driver area and a passenger area; If the area where the user is located is identified as the driver area, based on the second machine learning model from the acquired image The information identifies that the user's gaze on the area in front of the vehicle or the central control area is the gaze on the effective awakening area; If the area where the user is located is identified as the passenger area, based on the second machine learning model, it is identified from the acquired image information that the user's gaze on the central control area is a gaze on the effective wake-up area.

6. The vehicle voice assistant wake-up method according to any one of claims 1 to 5, wherein: The first machine learning model is a machine learning model based on a spatiotemporal convolutional transformer.

7. The vehicle voice assistant wake-up method according to any one of claims 1 to 5, wherein: The second machine learning model is a machine learning model based on deep convolutional neural network.

8. The vehicle voice assistant wake-up method according to any one of claims 1 to 5, wherein: The third machine learning model is a machine learning model based on an automatic speech recognition acoustic model; The fourth trained machine learning model is a machine learning model based on a long short-term memory network.

9. A vehicle voice assistant wake-up device, characterized in that: The device comprises: An image acquisition unit, used to acquire image information of a user; a recognition unit, configured to recognize a user's wake-up gesture from the acquired image information based on a first machine learning model, and recognize a user's gaze on an effective wake-up area from the acquired image information based on a second machine learning model; The first wake-up unit is used to wake up the vehicle voice assistant in response to the recognized wake-up gesture and the gaze.

10. The vehicle voice assistant wake-up device according to claim 9, characterized in that: The device also includes: An audio acquisition unit, used to acquire audio information of a user; a representation extraction unit, configured to extract a user's gesture representation from the acquired image information based on a first machine learning model, extract a user's gaze representation from the acquired image information based on a second machine learning model, and extract a user's voice representation from the acquired audio information based on a third machine learning model; If the vehicle voice assistant is not awakened, the method further includes: A representation concatenation unit, used for concatenating the gesture representation, the gaze representation, and the voice representation to form an embedded representation time series of a multimodal heterogeneous source; A prediction unit, configured to use the embedded representation time series as an input parameter of a fourth trained machine learning model to predict whether the user wakes up the vehicle voice assistant; The second wake-up unit is used to wake up the vehicle voice assistant if the fourth trained machine learning model predicts that the user wakes up the vehicle voice assistant.

11. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the wake-up method of the vehicle voice assistant according to any one of claims 1-8.

12. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the vehicle voice assistant wake-up method according to any one of claims 1-8.

Images