Method and system for guiding vehicle operation through multi-mode interaction and vehicle

Through multimodal interactive guidance of vehicle operation methods, combining user behavior and vehicle status, a personalized guidance strategy is generated, which solves the problem of single interaction mode and intuition in the existing technology, and achieves a more efficient user experience and interaction mode Diversity.

CN119089243APending Publication Date: 2024-12-06CHINA FAW CO LTD +1

Patent Information

Application Number
CN202411138041.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-19
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The existing vehicle assistant has a single interaction method and cannot meet the diverse interaction needs in different scenarios. The guidance is not intuitive and accurate enough, and lacks perception of users and vehicle status, so it cannot provide personalized and adaptable guidance.

Method used

Using multimodal interactive guidance vehicle operation method, through data acquisition and preparation, model training, multimodal perception and guidance strategy output steps, combined with user behavior, vehicle status and voice command data, a multimodal perception model and guidance strategy generation are constructed. The model perceives the user and vehicle status in real time, generates guidance strategies that meet user needs, and outputs them through voice broadcasts, dynamic image displays or light prompts.

Benefits of technology

It realizes personalized guidance based on the actual situation of users and vehicles, improves the user experience, and enhances the diversity of interaction methods and the intuitiveness and accuracy of guidance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119089243A_ABST
    Figure CN119089243A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of vehicles, and discloses a method and system for guiding vehicle operation through multi-mode interaction, electronic equipment, a storage medium and a vehicle, and the method comprises the steps: carrying out the marking in response to collected original data, distributing a corresponding sample label, carrying out the training of the original data in response to the marked original data, and carrying out the recognition of the original data. Constructing a multi-modal perception model and a guidance strategy generation model to identify user behaviors and vehicle states, and generating a guidance strategy; in response to a vehicle use process, analyzing vehicle use data based on the trained multi-mode perception model so as to perceive states of a user and a vehicle; and in response to the perceived vehicle use data, generating and outputting a guide strategy meeting the current demand of the user based on the trained guide strategy generation model. The system can sense the states of the user and the vehicle in real time, provides a corresponding guide strategy according to the actual situation of the user, provides a corresponding interaction mode, and improves the user experience.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of vehicle technology, and in particular to a method, system, electronic equipment, storage medium and vehicle for multi-modal interactive guidance of vehicle operation. Background Art

[0002] At present, car assistants mainly rely on a single interaction method, such as voice interaction or simple touch operation. When providing vehicle operation guidance, they usually adopt a fixed mode, such as displaying text instructions or static pictures and general videos on the car screen.

[0003] The prior art has the following problems:

[0004] 1. Limited interaction methods: It can only receive user instructions through a single input method, which cannot meet the diverse interaction needs of users in different scenarios. For example, when interacting through voice, users are required to accurately describe the question in voice. If the user does not know how to accurately describe what he wants to ask, or does not know the name of a certain component, it is difficult to successfully get an answer.

[0005] 2. Guidance is not intuitive and accurate enough: Fixed text instructions and static pictures are not effective in guiding complex vehicle operations, and users may find it difficult to understand and accurately perform operations.

[0006] 3. Lack of awareness of user and vehicle status: Failure to provide personalized and adaptive guidance based on the user’s actual location, posture, and real-time status of the vehicle.

[0007] Therefore, the present application provides a method for multi-modal interactive guidance of vehicle operation to solve the above technical problems. Summary of the invention

[0008] The purpose of the present invention is to provide a method, system, electronic device, storage medium and vehicle for guiding vehicle operation through multimodal interaction, so as to solve the technical problem in the prior art that the interaction modes are limited when guiding vehicle operation and the guidance cannot be performed according to the actual situation of the user.

[0009] In order to solve the above technical problems, the present invention provides a method for multi-modal interactive guidance of vehicle operation, comprising:

[0010] A data collection and preparation step, including, in response to the collected raw data, labeling the raw data and assigning corresponding sample labels, wherein the raw data includes user behavior data, vehicle status data, and voice command data;

[0011] A model training step includes training the raw data in response to the labeled raw data, constructing a multimodal perception model and a guidance strategy generation model to identify user behavior and vehicle status, and generate a guidance strategy;

[0012] The multimodal sensing step includes analyzing the vehicle usage data based on the trained multimodal sensing model in response to the vehicle usage process to sense the status of the user and the vehicle;

[0013] The guidance strategy output step includes generating and outputting a guidance strategy that meets the user's current needs based on the trained guidance strategy generation model in response to the perceived vehicle usage data.

[0014] In some specific embodiments, the data collection and preparation step includes, in response to the collected raw data, labeling the raw data and assigning corresponding sample labels, wherein the raw data includes user behavior data, vehicle status data and voice command data, and further includes:

[0015] Collecting the user behavior data, including the user's body movements, eye direction, and facial expression data;

[0016] Collecting the vehicle status data, including the vehicle's speed, posture, and component status data;

[0017] The collected user behavior, vehicle status and voice command data are labeled, a corresponding user behavior label and vehicle status label are assigned to each sample, and the labeled data are cleaned.

[0018] In some specific embodiments, the model training step includes training the raw data in response to the labeled raw data, constructing a multimodal perception model and a guidance strategy generation model to identify user behavior and vehicle status, and generate a guidance strategy, and further includes:

[0019] Based on a deep learning algorithm, the labeled user behavior data, vehicle status data, and voice command data are analyzed to construct and optimize the multimodal perception model so that the multimodal perception model can recognize the user's position, posture, eye direction, and vehicle status information;

[0020] Based on the big model framework, combined with the vehicle operation knowledge base and user habits, the guidance strategy generation model is constructed and optimized so that the guidance strategy generation model generates the guidance strategy that meets the user's actual situation.

[0021] In some specific embodiments, the multimodal perception step includes analyzing the vehicle usage data based on the trained multimodal perception model in response to the vehicle usage process to perceive the status of the user and the vehicle, and further includes:

[0022] Receiving the vehicle usage data in real time, including real-time status information reflecting the user and the vehicle;

[0023] Analyze the vehicle usage data based on the trained multimodal perception model to extract the user's position, posture, eye direction and vehicle status information;

[0024] Fuse data from different sources to form a comprehensive perception of user and vehicle status.

[0025] In some specific embodiments, the guidance strategy output step includes generating and outputting a guidance strategy that meets the current needs of the user based on the trained guidance strategy generation model in response to the perceived vehicle usage data, and further includes:

[0026] Analyze the user's current needs and predict the user's operation intention based on the real-time perceived vehicle usage data;

[0027] Based on the analysis result of the demand, generate the guidance strategy that meets the actual situation of the user, wherein the guidance strategy includes operation steps, precautions and risk warnings;

[0028] In response to user feedback, continuously optimizing the guidance strategy to improve the accuracy of the guidance strategy;

[0029] Output the guidance strategy, including voice broadcast or dynamic image display or light prompt and combined output.

[0030] In some specific embodiments, a feedback mechanism is also included to collect users' evaluation and feedback on the guidance strategy, and further adjust and optimize the multimodal perception model, guidance strategy generation model and output method based on the feedback.

[0031] Based on the same concept, the present invention also provides a system for guiding vehicle operation by multi-modal interaction, comprising:

[0032] A data collection and preparation module, configured to annotate the collected raw data and assign corresponding sample labels in response to the collected raw data, wherein the raw data includes user behavior data, vehicle status data, and voice command data;

[0033] A model training module is configured to train the raw data in response to the annotated raw data, construct a multimodal perception model and a guidance strategy generation model to identify user behavior and vehicle status, and generate a guidance strategy;

[0034] a multimodal perception module configured to analyze vehicle usage data based on the trained multimodal perception model in response to the vehicle usage process to perceive the status of the user and the vehicle;

[0035] The guidance strategy output module is configured to generate and output a guidance strategy that meets the user's current needs based on the trained guidance strategy generation model in response to the perceived vehicle usage data.

[0036] Based on the same concept, the present invention also provides an electronic device, comprising: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the method for multimodal interactive guidance of vehicle operation.

[0037] Based on the same concept, the present invention also provides a computer-readable storage medium storing a computer program executable by an electronic device. When the computer program runs on the electronic device, the electronic device executes the steps of the method for multimodal interactive guidance of vehicle operations.

[0038] Based on the same concept, the present invention also provides a vehicle, wherein the vehicle is provided with the multi-modal interactive system for guiding vehicle operation as described above.

[0039] Compared with the prior art, the beneficial effects are:

[0040] The present invention discloses a method, system, electronic device, storage medium and vehicle for multi-modal interactive guidance of vehicle operation, which can sense the status of a user and a vehicle in real time, provide corresponding guidance strategies according to the actual situation of the user, and provide corresponding interaction methods to enhance the user's experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a flowchart of a method for multi-modal interactive guidance of vehicle operation in some specific embodiments of the present invention;

[0042] Figure 2 It is a schematic diagram of the process architecture of a method for multi-modal interactive guidance of vehicle operation in some applications of the present invention;

[0043] Figure 3 It is a schematic diagram of the structure of a system for multi-modal interactive guidance of vehicle operation in some specific embodiments of the present invention;

[0044] Figure 4 It is a schematic diagram of the structure of an electronic device in some specific embodiments of the present invention. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0046] The terms used in the embodiments of the present application are only for the purpose of describing specific embodiments, and are not intended to limit the present application. The singular forms "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings, and "multiple" generally includes at least two.

[0047] It should be understood that the term "and / or" used in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship.

[0048] It should be understood that although the terms first, second, third, etc. may be used to describe in the embodiments of the present application, these descriptions should not be limited to these terms. These terms are only used to distinguish the descriptions. For example, without departing from the scope of the embodiments of the present application, the first may also be referred to as the second, and similarly, the second may also be referred to as the first.

[0049] As used herein, the words "if" and "if" may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to determining" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)", depending on the context.

[0050] It should also be noted that the term "includes", "comprising" or any other variation thereof is intended to cover non-exclusive inclusion, so that a commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprising a ..." do not exclude the existence of other identical elements in the commodity or device including the elements.

[0051] It should be particularly noted that any symbols and / or numbers in the specification that are not marked in the accompanying drawings are not drawing marks.

[0052] Reference Figure 1 , a method for multi-modal interactive guidance of vehicle operation, comprising:

[0053] S101, a data collection and preparation step, including labeling the collected raw data and assigning corresponding sample labels, wherein the raw data includes user behavior data, vehicle status data, and voice command data;

[0054] It is understandable that user behavior data collection utilizes devices such as high-definition cameras and posture sensors installed inside the vehicle to capture real-time video data such as the user's body movements (such as gestures, head rotations), eye direction, and facial expressions.

[0055] For example, the camera captures a user waving his hand to the right, while the gesture sensor detects a slight turn of the user's head.

[0056] Vehicle status data collection: The vehicle is equipped with various sensors, such as speed sensors, acceleration sensors, gyroscopes, etc., which collect vehicle status data such as vehicle speed, acceleration, steering angle, braking status, etc. in real time.

[0057] For example, the speed sensor shows that the current vehicle speed is 60km / h, and the gyroscope detects that the vehicle is turning left.

[0058] Voice command data collection: The on-board microphone array records the user's voice commands, such as "turn on the air conditioner", "slow down", etc.

[0059] Formulate data labeling specifications to clarify the labeling standards, label classification and naming rules for various types of data.

[0060] User behavior data labeling: assign corresponding behavior labels based on user behaviors in video data. For example, label a user’s waving gesture to the right as “right turn gesture” and a head turn as “head turn-right”.

[0061] Vehicle status data labeling, assigning corresponding status labels according to the vehicle status in the sensor data. For example, a speed of 60km / h is labeled as "medium speed driving" and a brake activation status is labeled as "braking".

[0062] Voice command data annotation: convert voice commands into text and assign corresponding command labels based on the command content. For example, annotate "turn on the air conditioner" as "air conditioner-on".

[0063] The quality control of annotation is carried out by cross-validation, regular spot checks, etc. to ensure the accuracy and consistency of the annotation results. Data with incorrect or ambiguous annotations will be re-annotated or removed.

[0064] The label system is constructed to cover multiple dimensions such as user behavior, vehicle status, and voice commands, with no overlap or omissions between labels.

[0065] Label assignment: assign the labeled data according to the label system and assign corresponding sample labels to each data sample. For example, a data sample containing a user's right turn gesture, a vehicle driving at medium speed, and a voice command of "navigate to the nearest gas station" is assigned a composite label of "right turn gesture-medium speed driving-navigation-gas station".

[0066] In some of the applications, the data collection and preparation steps include labeling the collected raw data and assigning corresponding sample labels in response to the collected raw data, where the raw data includes user behavior data, vehicle status data and voice command data; collecting user behavior data, including user's body movements, eye direction and facial expression data; collecting vehicle status data, including vehicle speed, posture and component status data; labeling the collected user behavior, vehicle status and voice command data, assigning corresponding user behavior labels and vehicle status labels to each sample, and cleaning the labeled data.

[0067] It is understandable that noise interference in the original data is removed, such as blurred parts in camera images, noise in microphone recordings, etc. Detect and process outliers in the data, such as extreme values ​​in sensor data, unreasonable actions in user behavior data, etc. For outliers, you can choose to delete, correct or interpolate. Remove duplicate, irrelevant or redundant information in the data, reduce the redundancy of the data set, and improve the efficiency of subsequent processing. Organize the cleaned data in a unified format, such as converting image data to a unified resolution and format, and converting text data to a unified encoding format.

[0068] S102, a model training step, comprising, in response to the annotated raw data, training the raw data, building a multimodal perception model and a guidance strategy generation model, so as to identify user behavior and vehicle status, and generate a guidance strategy;

[0069] In some of the applications, the model training step includes training the raw data in response to the annotated raw data, building a multimodal perception model and a guidance strategy generation model to identify user behavior and vehicle status, and generating a guidance strategy. Based on a deep learning algorithm, the annotated user behavior data, vehicle status data, and voice command data are analyzed to build and optimize a multimodal perception model so that the multimodal perception model can identify the user's position, posture, eye direction, and vehicle status information; based on a large model framework, combined with a vehicle operation knowledge base and user habits, a guidance strategy generation model is built and optimized so that the guidance strategy generation model generates a guidance strategy that meets the user's actual situation.

[0070] It is understandable that the model architecture design:

[0071] Designing a neural network architecture that integrates multimodal information includes the following parts:

[0072] Image processing branch: Use CNN to extract features in the image, such as user body movements and eye direction.

[0073] Text processing branch: Use RNN or Transformer structure to process voice command text and extract semantic information.

[0074] Numerical processing branch: Use a fully connected layer or a specially designed network structure to process vehicle status data.

[0075] Fusion layer: Fusion the features from different branches to form a unified feature representation.

[0076] Output layer: Based on the fused features, output the recognition results of user behavior and vehicle status.

[0077] Model training:

[0078] Use the labeled data as the training set and use optimization algorithms such as mini-batch gradient descent for training. During the training process, set the loss function (such as cross entropy loss) to evaluate the difference between the model prediction results and the true labels. Introduce regularization terms (such as L1, L2 regularization) and dropout technology to prevent overfitting. Monitor the loss value and accuracy during the training process in real time, and adjust hyperparameters such as learning rate and batch size to optimize the training effect.

[0079] Model evaluation and optimization:

[0080] Use the validation set to evaluate the trained model and check its generalization ability on unseen data. Tune the model based on the evaluation results, including adjusting the model architecture, adding more training data, improving feature extraction methods, etc. Use visualization tools (such as TensorBoard) to observe gradient changes and feature distribution during model training to better understand model behavior and optimize it.

[0081] Knowledge base construction:

[0082] Organize and encode information such as traffic rules, vehicle performance parameters, and safe driving specifications to form a structured knowledge base. Embed the knowledge base into the model so that it can be queried and referenced in real time when generating guidance strategies.

[0083] User habit model:

[0084] Use the user's historical driving data to train a user habit model. The model can be a sequence model based on LSTM or Transformer to capture the user's driving preferences and habits. The user habit model is updated regularly to reflect changes in user behavior.

[0085] Strategy generation model design:

[0086] Based on the large model framework (such as the GPT series), the guidance strategy generation model is designed in combination with the knowledge base and user habit model. The model receives the output of the multimodal perception model as input, and generates a specific guidance strategy in combination with the knowledge base and user habits.

[0087] Model training and evaluation:

[0088] Use simulated data or actual data containing a variety of driving scenarios and user behaviors for training. Introduce a reinforcement learning mechanism to evaluate the effectiveness of the generated guidance strategy through a simulated driving environment, and adjust the model parameters based on the evaluation results. Collect user feedback on the guidance strategy in actual driving to further optimize the model.

[0089] Strategy Optimization and Personalization:

[0090] Iterate and optimize the guidance strategy based on user feedback and driving performance data. Introduce personalized parameters so that the guidance strategy can better adapt to the driving habits and needs of different users.

[0091] S103, a multimodal perception step, comprising analyzing the vehicle usage data based on the trained multimodal perception model in response to the vehicle usage process to perceive the status of the user and the vehicle;

[0092] In some of these applications, the multimodal perception step includes analyzing vehicle usage data based on a trained multimodal perception model in response to vehicle usage to perceive the status of the user and the vehicle, and receiving vehicle usage data in real time, including real-time status information reflecting the user and the vehicle; analyzing vehicle usage data based on a trained multimodal perception model to extract the user's position, posture, eye direction and vehicle status information; and fusing data from different sources to form a comprehensive perception of the user and vehicle status.

[0093] It is understandable that high-definition cameras, multi-channel microphone arrays, inertial measurement units (IMUs, including accelerometers and gyroscopes), radars, and other sensors (such as lidars, ultrasonic sensors, etc.) are installed and configured on the vehicle, and time synchronization protocols (such as NTP) are used to ensure that the timestamps of all sensors are consistent for subsequent data alignment and fusion.

[0094] Based on the trained multimodal perception model, the image data is analyzed. Image enhancement techniques (such as brightness adjustment, contrast enhancement, noise suppression, etc.) are used to improve the image quality. Then, target detection algorithms (such as YOLO and SSD) are used to extract key features of the user and the surrounding environment (such as faces, limbs, road signs, etc.). For audio data, voice activity detection (VAD) is first performed to distinguish between speech segments and non-speech segments. Then, the speech segments are denoised, endpoint detected, and feature extracted (such as MFCC and i-vector) to prepare for subsequent speech recognition and semantic understanding. For numerical data (such as acceleration and gyroscope data), filtering processing (such as low-pass filtering and Kalman filtering) is performed to remove high-frequency noise and outliers, and converted into a format suitable for model input.

[0095] The preprocessed multimodal data are input into the corresponding sub-models. For example, image data is input into the CNN model for feature extraction and classification; audio data is input into the RNN or Transformer model for speech recognition and semantic understanding; numerical data is input into the fully connected network for state estimation.

[0096] Within the model, attention mechanisms, gating mechanisms and other technologies are used to perform weighted fusion of multimodal features to form a unified feature representation. This feature representation includes both individual features such as the user's position, posture, and eye direction, as well as overall state information such as the vehicle's speed, acceleration, and posture.

[0097] The model outputs the recognition results of the user status and vehicle status, and gives a confidence score for each result. The confidence score reflects the model's confidence in the recognition result, which helps to consider uncertainty factors when making decisions in subsequent steps.

[0098] According to the characteristics and importance of different data sources, fusion strategies (such as weighted average, Bayesian fusion, neural network fusion, etc.) are used to fuse the recognition results output by the multimodal perception model with the direct measurement data. During the fusion process, the timestamp and location information of the data are considered to ensure the spatiotemporal consistency of the fusion results. The fused results are verified to check whether there are logical contradictions or outliers. For example, if the user's location is identified as being outside the vehicle and the vehicle status shows that it is driving, it is an erroneous result. If an abnormal result is found, an anomaly detection algorithm (such as isolation forest, LOF) is used for identification and processing, and the error is corrected by reanalyzing the relevant data and adjusting the fusion strategy.

[0099] The verified and optimized fusion results are constructed into a comprehensive perception result set, including the user's position, posture, eye direction, behavioral intention, and the vehicle's speed, acceleration, posture, component status and other information.

[0100] S104, a guidance strategy output step, includes generating and outputting a guidance strategy that meets the user's current needs based on the trained guidance strategy generation model in response to the perceived vehicle usage data.

[0101] In some of these applications, the guidance strategy output step includes responding to the perceived vehicle usage data, generating and outputting a guidance strategy that meets the user's current needs based on a trained guidance strategy generation model, analyzing the user's current needs and predicting the user's operating intentions based on real-time perceived vehicle usage data; generating a guidance strategy that meets the user's actual situation based on the analysis results of the needs, the guidance strategy including operating steps, precautions and risk warnings; responding to user feedback, continuously optimizing the guidance strategy to improve the accuracy of the guidance strategy; outputting the guidance strategy, including voice broadcast or dynamic image display or light prompts and combined output.

[0102] It is understandable that key features are extracted, such as the user's facial expressions, gestures, voice command content, etc. According to the needs of the guidance strategy generation model, the features that are most helpful in predicting user needs and intentions are selected. The extracted features are input into the trained guidance strategy generation model for reasoning and prediction. The model outputs the user's current needs and predicted operation intentions. Combined with the vehicle's current driving state (such as speed, acceleration, direction), traffic environment (such as road type, traffic flow, weather conditions), and the user's personal habits and historical data, the results output by the model are contextually understood and optimized.

[0103] For example, when driving in the rain or at night, add reminders to use lights; when driving on the highway, emphasize the importance of maintaining a safe distance.

[0104] Based on the results of the needs and intentions analysis, select the template that best suits the current situation from the predefined guidance policy template library. Fine-tune the template according to the actual situation, such as adjusting the order of operation steps, adding or reducing precautions and risk warnings, etc.

[0105] By using user portraits and combining information such as user age, gender, driving experience, preferences, etc., we can inject personalized elements into the guidance strategy.

[0106] For example, for young users, a more concise and clear language style is used; for older users, more explanations and instructions are added.

[0107] Provide multiple user feedback channels, such as the touch screen of the vehicle system, voice assistant, mobile phone APP, etc., so that users can provide feedback anytime and anywhere. Encourage users to actively propose improvement suggestions, such as adding new guidance scenarios, optimizing existing strategies, etc. Sort and analyze the collected user feedback data to identify hot issues and improvement directions that users are concerned about. According to user feedback and data analysis results, iteratively optimize and update the guidance strategy generation model.

[0108] Design clear, accurate, and easy-to-understand voice announcements to ensure that users can quickly capture key information. Adjust announcement volume, speech speed, and intonation based on user voice preferences and vehicle environment.

[0109] Use the display screen of the vehicle system to display dynamic images or animations related to the guidance strategy, such as rearview mirror observation diagrams when changing lanes, flashing turn signal prompts when turning, etc. Ensure that the image content is intuitive and easy to understand, and complements the voice broadcast content.

[0110] When special reminders are needed, use the vehicle's internal lighting system (such as dashboard indicators, steering wheel lights, etc.) to provide reminders. Design light flashing patterns and color combinations to attract users' attention and convey the correct information.

[0111] According to the actual situation and user needs, various output methods such as voice broadcast, dynamic image display and light prompts can be flexibly combined to ensure that various output methods can cooperate and complement each other to form a more comprehensive and effective guidance strategy output effect.

[0112] Some of these applications also include a feedback mechanism for collecting users' evaluations and feedback on the guidance strategies, and further adjusting and optimizing the multimodal perception model, guidance strategy generation model, and output method based on the feedback.

[0113] It is understandable that users can provide feedback through channels such as the touch screen of the vehicle system, voice assistant or mobile phone APP. The collected feedback data is sorted and analyzed to identify hot issues and improvement directions of user concern. Based on the feedback results, the multimodal perception model, guidance strategy generation model and output method are further adjusted and optimized.

[0114] For example, users provided the following feedback through the in-vehicle system’s touchscreen: “The voice announcements were sometimes too frequent, causing distraction; the dynamic graphics were clear but updated slowly.”

[0115] For the problem of too frequent voice announcements, adjust the triggering conditions and frequency of voice announcements based on the user's historical driving behavior and current traffic environment. For example, increase the number of announcements in traffic congestion or complex road conditions, and reduce unnecessary announcements on unobstructed highways. For the problem of slow dynamic image update speed, optimize the image processing algorithm to increase the speed of image rendering and update.

[0116] Combine the following Figure 2 The following are some examples of the method for guiding vehicle operation through multi-modal interaction according to the present invention in some applications:

[0117] like Figure 2 As shown, user behavior data collection:

[0118] Camera data: The in-car camera captures the user’s body movements, eye direction, facial expressions, etc., and is used to analyze the user’s behavioral intentions and emotional state.

[0119] Sensor data: Use various sensors inside the vehicle (such as accelerometers, gyroscopes, pressure sensors, etc.) to collect data such as the vehicle's speed, posture, status of various components, and user operating habits.

[0120] Voice data: Collect user voice commands through the vehicle microphone for training and optimizing voice interaction models.

[0121] Data labeling and cleaning:

[0122] Label the collected data, including user behavior labels, vehicle status labels, etc., to ensure data quality.

[0123] Clean data, remove noise and outliers, and improve data validity and accuracy.

[0124] Multimodal perception model training:

[0125] A multimodal perception model is constructed using deep learning techniques such as convolutional neural networks (CNN) to process image data, recurrent neural networks (RNN) or Transformer to process speech data, and posture estimation algorithms to process body movement data.

[0126] The model is trained with a large amount of labeled data, enabling it to accurately identify the user's position, posture, eye direction and vehicle status.

[0127] Guided strategy generation model training:

[0128] A guided strategy generation model is built based on large models (such as the GPT series, BERT, etc.), which can deeply understand and analyze multimodal perception data.

[0129] Combining the vehicle operation knowledge base and user habit model, the model is trained through reinforcement learning or supervised learning to enable it to generate personalized guidance strategies.

[0130] Output Module:

[0131] A variety of output methods (voice broadcast, dynamic image display, light prompts, etc.) combined with trained models ensure that users can receive guidance information clearly and quickly.

[0132] This embodiment is described below in conjunction with application scenarios:

[0133] The user parks in a basement with poor lighting and wants to ask what an indicator light means that he doesn't know its name. He can point to the indicator light and ask what it means. The user holds the bottom of the steering wheel with both hands and wants to adjust the steering wheel height. He asks the voice assistant where the adjustment button is. The voice assistant feedbacks that the adjustment button is on the upper left of the left hand and below the steering wheel. A graphic and text appear on the screen, and the reading light shines on the position of the adjustment button. The user follows the instructions and gropes upward with his left hand. The user detects that the user has reached too far upward, and the voice reminds the user in real time to move his left hand down a little. After the user finds the button, he adjusts the steering wheel upward. After adjusting it to the highest position, it detects that the user is still pressing, and the voice reminds the user that it has reached the highest position.

[0134] For the method steps disclosed in the above embodiments, the method steps are described as a series of action combinations for the purpose of simple description, but those skilled in the art should know that the embodiments of the present invention are not limited by the order of the actions described, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of the present invention.

[0135] like Figure 3As shown, the present invention also provides a system for multi-modal interactive guidance of vehicle operation, comprising:

[0136] The data collection and preparation module 201 is configured to annotate the collected raw data and assign corresponding sample labels, wherein the raw data includes user behavior data, vehicle status data, and voice command data;

[0137] A model training module 202 is configured to train the raw data in response to the annotated raw data, construct a multimodal perception model and a guidance strategy generation model to identify user behavior and vehicle status, and generate a guidance strategy;

[0138] The multimodal perception module 203 is configured to analyze the vehicle usage data based on the trained multimodal perception model in response to the vehicle usage process to perceive the status of the user and the vehicle;

[0139] The guidance strategy output module 204 is configured to generate and output a guidance strategy that meets the current needs of the user based on the trained guidance strategy generation model in response to the perceived vehicle usage data.

[0140] It is worth noting that although only some basic functional modules are disclosed in the embodiments of the present invention, it does not mean that the composition of the present system is limited to the above basic functional modules. On the contrary, what the present embodiment wants to express is that on the basis of the above basic functional modules, those skilled in the art can arbitrarily add one or more functional modules in combination with the prior art to form an infinite number of embodiments or technical solutions. In other words, the present system is open rather than closed. Just because the present embodiment only discloses individual basic functional modules, it cannot be considered that the protection scope of the claims of the present invention is limited to the disclosed basic functional modules. At the same time, for the convenience of description, the above devices are described in terms of functions and are described separately in various units and modules. Of course, when implementing the present invention, the functions of each unit and module can be implemented in the same or one or more software and / or hardware.

[0141] like Figure 4 As shown, the present invention also provides an electronic device, comprising: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the method for multi-modal interactive guidance of vehicle operation.

[0142] Figure 4 Schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Figure 4The electronic device provided in the embodiment of the present invention includes: one or more processors 710 and a storage device 720; the processor 710 in the electronic device may be one or more, Figure 4 A processor 710 is taken as an example; the storage device 720 is used to store one or more programs; the one or more programs are executed by the one or more processors 710, so that the one or more processors 710 implement the method of multimodal interactive guidance of vehicle operation as described in any one of the embodiments of the present invention.

[0143] The electronic device may further include: an input device 730 and an output device 740 .

[0144] The processor 710, storage device 720, input device 730 and output device 740 in the electronic device can be connected via a bus or other means. Figure 4 The example of connecting through bus is taken in the following.

[0145] The storage device 720 in the electronic device is a computer-readable storage medium that can be used to store one or more programs, which can be software programs, computer executable programs and modules, such as program instructions / modules corresponding to the method for multi-modal interactive guidance of vehicle operation provided in the embodiment of the present invention. The processor 710 executes various functional applications and data processing of the electronic device by running the software programs, instructions and modules stored in the storage device 720, that is, the method for multi-modal interactive guidance of vehicle operation in the above method embodiment is implemented.

[0146] The storage device 720 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the storage device 720 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the storage device 720 may further include a memory remotely arranged relative to the processor 710, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0147] The input device 730 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the electronic device. The output device 740 may include a display device such as a display screen.

[0148] The present invention also provides a computer-readable storage medium storing a computer program executable by an electronic device. When the computer program runs on the electronic device, the electronic device executes the steps of the method for multimodal interactive guidance of vehicle operations.

[0149] Specifically, the computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present embodiment, the computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0150] The present invention also provides a vehicle, wherein the vehicle is provided with the multi-modal interactive system for guiding vehicle operation as described above.

[0151] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for guiding vehicle operation by multimodal interaction, characterized in that: include: A data collection and preparation step, including, in response to the collected raw data, labeling the raw data and assigning corresponding sample labels, wherein the raw data includes user behavior data, vehicle status data, and voice command data; A model training step includes training the raw data in response to the labeled raw data, constructing a multimodal perception model and a guidance strategy generation model to identify user behavior and vehicle status, and generate a guidance strategy; The multimodal sensing step includes analyzing the vehicle usage data based on the trained multimodal sensing model in response to the vehicle usage process to sense the status of the user and the vehicle; The guidance strategy output step includes generating and outputting a guidance strategy that meets the user's current needs based on the trained guidance strategy generation model in response to the perceived vehicle usage data.

2. The method for guiding vehicle operation by multimodal interaction according to claim 1, characterized in that: The data collection and preparation step includes, in response to the collected raw data, labeling the raw data and assigning corresponding sample labels, wherein the raw data includes user behavior data, vehicle status data and voice command data, and further includes: Collecting the user behavior data, including the user's body movements, eye direction, and facial expression data; Collecting the vehicle status data, including the vehicle's speed, posture, and component status data; The collected user behavior, vehicle status and voice command data are labeled, a corresponding user behavior label and vehicle status label are assigned to each sample, and the labeled data are cleaned.

3. The method for guiding vehicle operation by multimodal interaction according to claim 1, characterized in that: The model training step includes training the raw data in response to the labeled raw data, constructing a multimodal perception model and a guidance strategy generation model to identify user behavior and vehicle status, and generate a guidance strategy, further including: Based on a deep learning algorithm, the labeled user behavior data, vehicle status data, and voice command data are analyzed to construct and optimize the multimodal perception model so that the multimodal perception model can recognize the user's position, posture, eye direction, and vehicle status information; Based on the big model framework, combined with the vehicle operation knowledge base and user habits, the guidance strategy generation model is constructed and optimized so that the guidance strategy generation model generates the guidance strategy that meets the user's actual situation.

4. The method for guiding vehicle operation by multimodal interaction according to claim 1, characterized in that: The multimodal perception step includes analyzing the vehicle usage data based on the trained multimodal perception model in response to the vehicle usage process to perceive the status of the user and the vehicle, and further includes: Receiving the vehicle usage data in real time, including real-time status information reflecting the user and the vehicle; Analyze the vehicle usage data based on the trained multimodal perception model to extract the user's position, posture, eye direction and vehicle status information; Fuse data from different sources to form a comprehensive perception of user and vehicle status.

5. The method for guiding vehicle operation by multimodal interaction according to claim 1, characterized in that: The guidance strategy output step includes generating and outputting a guidance strategy that meets the current needs of the user based on the trained guidance strategy generation model in response to the perceived vehicle usage data, and further includes: Analyze the user's current needs and predict the user's operation intention based on the real-time perceived vehicle usage data; Based on the analysis result of the demand, generate the guidance strategy that meets the actual situation of the user, wherein the guidance strategy includes operation steps, precautions and risk warnings; In response to user feedback, continuously optimizing the guidance strategy to improve the accuracy of the guidance strategy; Output the guidance strategy, including voice broadcast or dynamic image display or light prompt and combined output.

6. The method for guiding vehicle operation by multimodal interaction according to claim 1, characterized in that: It also includes a feedback mechanism for collecting users' evaluation and feedback on the guidance strategy, and further adjusting and optimizing the multimodal perception model, the guidance strategy generation model and the output method based on the feedback.

7. A system for multi-modal interactive guidance of vehicle operation, characterized in that: include: A data collection and preparation module, configured to annotate the collected raw data and assign corresponding sample labels in response to the collected raw data, wherein the raw data includes user behavior data, vehicle status data, and voice command data; A model training module is configured to train the raw data in response to the annotated raw data, construct a multimodal perception model and a guidance strategy generation model to identify user behavior and vehicle status, and generate a guidance strategy; a multimodal perception module configured to analyze vehicle usage data based on the trained multimodal perception model in response to the vehicle usage process to perceive the status of the user and the vehicle; The guidance strategy output module is configured to generate and output a guidance strategy that meets the user's current needs based on the trained guidance strategy generation model in response to the perceived vehicle usage data.

8. An electronic device, characterized in that: include: A processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the method described in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: It stores a computer program executable by an electronic device, and when the computer program runs on the electronic device, the electronic device executes the steps of the method described in any one of claims 1 to 6.

10. A vehicle, characterized in that: The vehicle is provided with the system for multimodal interactive guidance of vehicle operation as claimed in claim 7.

Citation Information

Patent Citations

  • Unmanned vehicle and navigation method and device thereof

    CN111026873A

  • Intelligent interaction method and device based on multimode fusion

    CN115610349A

  • Automatic driving obstacle avoidance system and method based on machine vision

    CN117037115A

  • Vehicle interaction method and device, electronic equipment, storage medium and vehicle

    CN117235320A

  • Interaction method and apparatus for intelligent cockpit, device, and medium

    US20220234593A1

Cited By

  • Vehicle-mounted function guiding method and device, vehicle and program product

    CN120986334A

  • Vehicle instrument panel icon state recognition method based on frequency domain dynamic suppression mechanism

    CN122200614A