Vehicle using method and device based on multi-mode interaction

Abstract

The invention provides a vehicle use method and device based on multimode interaction, and the method comprises the steps: responding to question information inputted by a user, and outputting first information; in response to receiving the feedback signal within the preset time, collecting in-vehicle information; and outputting second information in response to the condition that the in-vehicle information comprises the target information. According to the method provided by the invention, for the defects in the aspect of information presentation closed loop at present, the multimode information is introduced in the human-computer interaction process of the automobile specification, and the interaction signal feedback of the user and the automobile is added into the interaction process, so that the user is helped to more practically understand, learn and master related automobile control knowledge, and the user experience is improved. And whether own operation is correct or not is confirmed, so that closed loop of information acquisition is realized. Furthermore, according to the method, whether the operation of the user is correct or not can be recognized by obtaining the operation feedback of the user, and more detailed description and guidance can be provided for incorrect operation according to presetting, so that the user is helped to really master the operation mode of the vehicle.

Description

Technical Field

[0001] The present application relates to the field of automotive technology, and in particular to a vehicle use method and device based on multimodal interaction. Background Art

[0002] In today's society, users primarily learn vehicle operation through training at driving schools, using training and test vehicles to master the specific operations of vehicle operation, control, and driving. However, with the passage of time, evolution in technology and design, and differences in vehicle types, brands, and models, novice drivers often struggle to quickly master all aspects of a new car's operation. They often need to seek help from sales staff, vehicle manuals, or the internet to learn how to operate certain functions.

[0003] With the popularity of large-screen in-car displays, many manufacturers have integrated vehicle manuals into these systems, allowing users to access the manuals through manual browsing, voice inquiries, and other methods. The content is typically presented in the form of text, text + images, or videos, allowing users to more conveniently and quickly access information on vehicle functions.

[0004] Current mainstream interactive solutions for car manuals often utilize a Web 2.0 approach to information exchange, where the car's computer presents information in a one-way manner, and users access information through browsing. This interactive model offers no effective feedback from the user, making it difficult for users to confirm whether they have correctly understood and mastered the relevant information and knowledge.

[0005] Although human-computer interaction through in-vehicle voice assistants has improved the interaction friendliness, for example, users can obtain relevant information by asking questions in natural language without having to locate it themselves based on directories or keyword searches, which improves the efficiency of information acquisition, the situation in which information is output to users still does not get rid of the one-way output mode. Users still find it difficult to determine whether they have obtained the information correctly, making the information acquisition process incomplete. Summary of the Invention

[0006] The present application provides a vehicle usage method and device based on multi-modal interaction to solve the technical problem that when the car manual outputs information, the user cannot correctly understand whether his operation is correct and cannot achieve a complete closed loop.

[0007] The first aspect of the present application provides a vehicle usage method based on multimodal interaction, including: outputting first information in response to question information input by a user, wherein the question information includes an operation question of a target device, and the first information includes a first step of operating the target device; collecting in-vehicle information in response to receiving a feedback signal within a preset time, wherein the feedback signal is generated by the user operating the in-vehicle device, and the in-vehicle information includes working information of multiple devices in the vehicle; outputting second information in response to the in-vehicle information including target information; wherein the target information includes working information of the target device executed according to the first step, and the second information includes a second step of operating the target device, and the second step is the next step of the first step.

[0008] In some feasible implementations, in response to not receiving the feedback signal within the preset time, a timeout prompt message is output, and it is re-determined whether the feedback signal is received within the preset time.

[0009] In some feasible implementations, in response to the in-vehicle information not including the target information and the in-vehicle information being whitelist information, the in-vehicle information is ignored.

[0010] In some feasible implementations, in response to the in-vehicle information not including the target information and the in-vehicle information not being whitelist information, an error prompt message is output and the first information is re-output.

[0011] In some feasible implementations, after responding to the question information input by the user, the vehicle usage method also includes: obtaining operation information, wherein the operation information includes all steps of operating the target device; determining whether the vehicle is configured with interactive content; in response to the vehicle being configured with interactive content, outputting first information; in response to the vehicle not being configured with interactive content, statically displaying the operation information.

[0012] The second aspect of the present application provides a vehicle use device based on multimodal interaction, including: an output module, configured to output first information in response to question information input by a user, wherein the question information includes an operating question of a target device, and the first information includes a first step of operating the target device; a collection module, configured to collect in-vehicle information in response to receiving a feedback signal within a preset time, wherein the feedback signal is generated by the user operating the in-vehicle device, and the in-vehicle information includes working information of multiple devices in the vehicle; the output module is also configured to output second information in response to the in-vehicle information including target information; wherein the target information includes working information of the target device executed according to the first step, and the second information includes a second step of operating the target device, and the second step is the next step of the first step.

[0013] In some feasible implementations, the output module is further configured to, in response to not receiving the feedback signal within the preset time, output a timeout prompt message and re-determine whether the feedback signal is received within the preset time.

[0014] In some feasible implementations, the output module is further configured to ignore the in-vehicle information in response to the in-vehicle information not including the target information and the in-vehicle information being whitelist information.

[0015] In some feasible implementations, the output module is further configured to, in response to the in-vehicle information not including the target information and the in-vehicle information not being whitelist information, output an error prompt message and re-output the first information.

[0016] In some feasible implementations, the vehicle usage device also includes an acquisition module, a determination module and a display module; the acquisition module is configured to acquire operation information, wherein the operation information includes all steps of answering question information; the determination module is configured to determine whether the vehicle is configured with interactive content; the output module is further configured to output first information in response to the vehicle being configured with interactive content; the display module is configured to statically display the operation information in response to the vehicle not being configured with interactive content.

[0017] The present application provides a vehicle usage method and device based on multimodal interaction. The vehicle usage method includes: outputting first information in response to a user input of a question, wherein the question includes an operation question for a target device and the first information includes a first step for operating the target device; collecting in-vehicle information in response to receiving a feedback signal within a preset time, wherein the feedback signal is generated by the user operating an in-vehicle device and the in-vehicle information includes operating information of multiple in-vehicle devices; and outputting second information in response to the in-vehicle information including target information, wherein the target information includes operating information of the target device executed according to the first step and the second information includes a second step for operating the target device, the second step being the next step after the first step. This vehicle usage method addresses the current shortcomings in closed-loop information presentation by introducing multimodal information into the human-computer interaction process of vehicle manuals, incorporating interactive signal feedback between the user and the vehicle into the interaction process, helping users to more effectively understand, learn, and master relevant vehicle operation knowledge and confirm whether their operations are correct, thereby achieving a closed-loop information acquisition system. Furthermore, the method can also obtain user operation feedback to identify whether the user operation is correct or not. For incorrect operations, more detailed instructions and guidance can be provided according to pre-set settings to help users truly master the operation of the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0019] Figure 1 It is a flowchart of a method for using a vehicle provided in an embodiment of the present application;

[0020] Figure 2 This is a schematic diagram of an application flow of a vehicle use method based on multimodal interaction provided in an embodiment of the present application;

[0021] Figure 3 This is a structural block diagram of a vehicle usage device based on multimodal interaction provided in an embodiment of the present application.

[0022] Graphic mark:

[0023] 100 - vehicle use device; 101 - output module; 102 - collection module; 103 - acquisition module; 104 - determination module; 105 - display module. DETAILED DESCRIPTION

[0024] The following will clearly describe the technical solutions in the embodiments of the present application in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments of the present application, other embodiments obtained by ordinary technicians in this field without making any creative work are all within the scope of protection of this application.

[0025] Hereinafter, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature identified with "first," "second," etc., may explicitly or implicitly include one or more of the features. In the description of this application, unless otherwise specified, "plurality" means two or more.

[0026] In addition, in this application, directional terms such as "upper", "lower", "inner" and "outer" are defined relative to the orientation of the components in the drawings. It should be understood that these directional terms are relative concepts. They are used for relative description and clarification, and they can change accordingly according to changes in the orientation of the components in the drawings.

[0027] Vehicles can usually use two major ways to answer questions raised by users. The first is document display, including document analysis and introduction, graphic display and video playback. The second includes in-vehicle voice assistant Q&A, including voice Q&A.

[0028] Currently, car manuals are usually stored in the form of PDF documents or built-in apps on large-screen car systems, mobile phone apps, websites, etc. At the same time, some OEMs disassemble and process the contents of car manuals and configure them to in-vehicle voice assistants in the form of a knowledge base, thereby providing question-and-answer support for the content.

[0029] In the first category of document display, the in-car large screen usually uses the Android system as the operating system. Under this operating system, there are many ways to load and display electronic car manuals, such as:

[0030] PDF Reader: Stores the vehicle manual as a PDF file in the system's storage space. When the user accesses the file, various PDF readers are used to parse and display the file content. Depending on the file permissions and configuration, interactive features such as browsing, specifying pages, zooming, directory navigation, and keyword search are available.

[0031] APP-based: The car manual is made into an application (APP) similar to an e-book, which supports the display of the car manual content and interactive functions such as navigation and search.

[0032] The contents of the vehicle manual can be updated through OTA upgrades or network resource updates.

[0033] Among them, document interpretation and display: the document content can be parsed and loaded and displayed through the PDF Renderer that comes with the Android system or other third-party applications, third-party libraries and other PDF parsers, and relevant interactive functions can be provided according to the capabilities defined by the application.

[0034] Graphical display: On the large screen in the car, the text, pictures and other information that serve as the answer can be displayed according to the display style pre-defined by the program.

[0035] Video playback: Relevant video materials can be displayed on the large screen in the car. According to user operations or pre-defined automatic playback logic, the local video player is used to play the video content, which is generally an operation instruction video.

[0036] In the second type of in-car voice assistant Q&A, from the data end, relevant staff will break down the content of the car manual into multiple sets of question and answer pairs, and enter the data into the voice assistant knowledge base in the form of question and answer pairs, where the questions exist in text form and the answers exist in the form of text, pictures, videos or a combination of multiple elements. On the car side, when the user asks a question, the user's voice question is converted into question text through audio pickup, acoustic front-end processing, speech recognition (speech to text) and other steps, and then the natural language processing module normalizes it and matches it with the questions in the knowledge base. If there is a similar or consistent question in the knowledge base, the answer information corresponding to the question will be returned and displayed and played according to the preset display rules.

[0037] Voice Q&A: Through the in-vehicle voice system and its backend-linked car manual knowledge base, users can ask pre-recorded questions, retrieve relevant answers from the knowledge base, and then display them on the vehicle side in various forms such as text, pictures, and videos.

[0038] However, in the above display method, it is still difficult for users to confirm whether they have obtained the correct information, making the information acquisition process incomplete.

[0039] To address the above technical issues, the embodiments of the present application provide a multi-modal interactive vehicle usage method. This method addresses the current shortcomings in closed-loop information presentation by introducing multi-modal information into the human-computer interaction process of the vehicle manual, incorporating interactive signal feedback between the user and the vehicle into the interaction process to help users more effectively understand, learn, and master relevant vehicle control knowledge and confirm whether their operations are correct. Furthermore, the method can also identify whether the user's operation is correct by obtaining user operation feedback. For incorrect operations, more detailed instructions and guidance can be provided based on pre-set settings to help users truly master the vehicle's operation methods.

[0040] Figure 1 It is a flowchart of the multi-modal interactive vehicle usage method provided in an embodiment of the present application.

[0041] See also Figure 1 The multi-modal interactive vehicle usage method provided in the embodiment of the present application is implemented by the following steps S100 to S300.

[0042] Step S100: Outputting first information in response to question information input by the user.

[0043] The question information includes an operation question of the target device, and the first information includes a first step of operating the target device.

[0044] Specifically, the question information may include how to use an in-car device, and the target device may be one or more in-car devices. For example, the question information may include how to use the in-car air conditioner, how to use the in-car air conditioner, or how to increase the temperature of the in-car air conditioner. In this case, the target device is the in-car air conditioner.

[0045] The first information includes the first step of operating the in-car air conditioner. For example, when the question is: how to increase the temperature of the in-car air conditioner, the first information may be: switch the display screen of the central control screen to the in-car air conditioner control interface.

[0046] The way in which the user inputs question information may include asking questions by voice, or selecting on the central control screen.

[0047] Step S200: In response to receiving a feedback signal within a preset time, collecting in-vehicle information.

[0048] Specifically, after the first information is output, the user may perform operations according to the content of the first information, or the user may not perform operations according to the content of the first information due to other reasons, or the user may perform erroneous operations due to other reasons.

[0049] After the first information is output, a timer is started to determine whether a feedback signal is received within a preset time. The feedback signal is generated by the user operating the in-vehicle device, which may be the target device or another in-vehicle device that is not the target device.

[0050] If a feedback signal is received within the preset time, it indicates that the user has performed an operation within the preset time. In order to further verify the accuracy of the user's operation, the in-vehicle information is collected. The in-vehicle information includes the working information of multiple devices in the vehicle.

[0051] After collecting the in-vehicle information, it is determined whether the in-vehicle information includes target information, wherein the target information includes operation information of the target device executed according to the first step.

[0052] For example, if the question is "How to increase the temperature of the vehicle's air conditioning?", the first information may be "Switch the central control screen display to the vehicle's air conditioning control interface." After collecting operating information of multiple devices in the vehicle, it is determined whether the in-vehicle information includes switching the central control screen display to the vehicle's air conditioning control interface. If it is determined that the in-vehicle information includes that the central control screen display is already in the vehicle's air conditioning control interface, step S300 is executed to output the second information.

[0053] Step S300: In response to the in-vehicle information including target information, output second information.

[0054] The second information includes a second step of the target device, and the second step is the next step of the first step. The second step may be to click on the temperature adjustment module in the control interface of the in-vehicle air conditioner.

[0055] In this way, the vehicle usage method provided in the embodiment of the present application introduces multimodal information into the human-computer interaction process of the car manual, and adds the interactive signal feedback between the user and the vehicle into the interaction process, so as to help the user understand, learn, and master the relevant vehicle control knowledge more effectively, and confirm whether his own operation is correct.

[0056] The interaction between the car and the user is reflected in the process of adding an interactive process after the information reaches the user. By collecting feedback from the user's actual operation, the user's operation is determined to be correct and the corresponding content is executed accordingly. Multimodal information input is reflected in the entire interactive process of the user obtaining the content of the car manual. In addition to the original query information, the input information also uses the user's actual vehicle operation information as a new source of information to determine whether the user's operation is correct, which can help users use the in-car equipment correctly.

[0057] In some feasible implementations, when it is determined that no feedback signal is received within a preset time, step S110 is executed.

[0058] Step S110: In response to not receiving the feedback signal within the preset time, outputting a timeout prompt message, and re-determining whether the feedback signal is received within the preset time.

[0059] If the feedback signal is not received within the preset time, it means that the user has not operated the in-vehicle device after outputting the first information. At this time, a timeout reminder is issued to remind the user to operate according to the content of the first information, and the timing is reset to repeat the operation of determining whether the feedback signal is received within the preset time.

[0060] In this implementation, the user can be reminded to operate the target device in time, so as to avoid the user forgetting the content of the first information due to inactivity for a long time, causing the user to re-enter the question information and avoid wasting the user's time.

[0061] In some feasible implementations, in the step of determining whether the in-vehicle information includes target information, if the in-vehicle information does not include target information, step S210 and step S220 are executed according to the circumstances.

[0062] If the in-car information does not include the target information, it means that the user did not operate the target device at that time. The in-car information may be caused by user error or other reasons. At this time, it is further determined whether the in-car information is whitelisted information. Whitelisted information may include air conditioners and windows. Since the air conditioner may make noise during operation, it may be mistakenly believed that the user has operated a certain in-car device. The sound of the car window falling can also be made, and the sound of strong wind outside the car hitting the car window can also mistakenly be believed that the user has operated a certain in-car device.

[0063] Therefore, it is determined whether the target information is whitelist information, and corresponding reminders are given according to different results to improve the user experience.

[0064] If it is determined that the in-vehicle information is whitelist information, step S210 is executed. If it is determined that the in-vehicle information is not whitelist information, step S220 is executed.

[0065] Step S210: In response to the in-vehicle information not including the target information and the in-vehicle information being whitelist information, the in-vehicle information is ignored.

[0066] In step S210 , if the in-vehicle information is whitelist information, it indicates that it is interference information and is ignored.

[0067] Step S220: In response to the in-vehicle information not including the target information and the in-vehicle information not being whitelist information, outputting an error prompt message and re-outputting the first information.

[0068] In step S220, the in-vehicle information is not whitelist information, indicating that the user has operated a certain in-vehicle device at this time, but the in-vehicle device is not the target device, or the user did not operate the target device according to the first step in the first information, resulting in an operation error. At this time, an error prompt is output to remind the user that an incorrect operation has been performed, and the first information is re-output to prompt the user to perform the correct operation according to the first information.

[0069] In some feasible implementations, step S100 provided in the embodiment of the present application may include the following steps S100-step S100.

[0070] Step S100: Acquire operation information in response to question information input by the user.

[0071] The operation information includes all steps of operating the target device.

[0072] For example, still taking the question information as: How to increase the temperature of the car air conditioner, the operation information is all the steps for operating the car air conditioner, all the steps include the first step, the second step, or also include subsequent steps of the second step, such as the third step, the fourth step, etc.

[0073] Step S102: Determine whether the vehicle is configured with interactive content. If so, proceed to step S103; if not, proceed to step S104.

[0074] In step S102 , it is necessary to determine whether the vehicle is configured with interactive operation information, so as to facilitate subsequent display in a corresponding manner.

[0075] Step S103: In response to the vehicle being configured with interactive content, outputting first information.

[0076] If the vehicle is configured with interactive content, the first information is outputted in an interactive manner, that is, the first step of operating the target device is outputted in an interactive manner.

[0077] When interactive content is not configured, step S104 is executed.

[0078] Step S104: In response to the vehicle not being configured with interactive content, statically displaying operation information.

[0079] If the vehicle is not configured with interactive content, the operation information is displayed statically, that is, all steps for operating the target device are displayed in a static manner.

[0080] It is worth noting that the method provided in the embodiment of the present application outputs the information for the next step only after determining that the user has executed the current step. The operation steps for the target device are not limited to completing the operation within two steps, and may also include third information and fourth information. The third information may include the third step of operating the target device, which is the next step after the second step. The fourth information may include the fourth step of operating the target device, which is the next step after the third step.

[0081] For example, using the air conditioning heating operation as an example, the air conditioning heating operation may include four steps. After outputting the second information, the steps of receiving a feedback signal and timing are repeated to determine whether the feedback signal is received within a preset time. After the feedback signal is received within the preset time, and after collecting in-vehicle information and determining that the in-vehicle information includes the target information, the third information is output. Next, the steps of receiving a feedback signal are repeated, and after determining that the feedback signal is received within the preset time and the in-vehicle information includes the target information, the fourth information is output.

[0082] In the interaction process of each step, the whitelist information can be the same or different. In other words, the scope of the whitelist information can be independently defined, or the universal whitelist information can be referenced.

[0083] Figure 2 This is a schematic diagram of the application flow of a vehicle usage method based on multimodal interaction provided in an embodiment of the present application.

[0084] See also Figure 2 In a specific implementation, it can be achieved through steps S001 to S520.

[0085] Step S001: The user asks / selects a question.

[0086] In step S001, the user can ask questions by voice or select questions on the central control screen.

[0087] Step S401: Obtain the answer configuration corresponding to the question.

[0088] In step S401, the configuration for obtaining the answer here can be understood as the obtaining operation information mentioned above.

[0089] Step S402: Whether interactive content is configured. If it is determined that interactive content is configured, step S403 is executed; if it is determined that interactive content is not configured, step S404 is executed.

[0090] Step S403: Return to the first step and start timing.

[0091] In step S403, returning to the first step and starting the timing can be understood as outputting the first information and starting the timing as mentioned above.

[0092] Step S404: static display.

[0093] What is statically displayed at this time is the operation information, that is, all the steps of operating the target device. After executing step S404, the operation process ends.

[0094] Step S405: Check if the timeout has occurred. If so, go to step S410; if not, go to step S500.

[0095] In step S405, it is determined whether a feedback signal is received within a preset time. If a feedback signal is received, step S500 is executed; if not, step S410 is executed.

[0096] Step S410: Timeout reminder and re-execution of step S405.

[0097] At this time, the user is reminded that the operation has timed out, and step S405 is executed again.

[0098] Step S500: Collect in-vehicle information.

[0099] In step S500, operating information of multiple devices in the vehicle is collected.

[0100] Step S501: Check if it is target information. If so, go to step S502; if not, go to step S503.

[0101] Determine whether the in-vehicle information is target information, thereby facilitating determination of whether the user has operated the target device.

[0102] Step S502: proceed to the next step.

[0103] At this time, it is determined that the user has operated the target device according to the first step in the first information. Step S502 corresponds to the step of outputting the second information in the above embodiment, thereby verifying that the user has correctly executed the first step, prompting the user to execute the second step, and repeating the step of determining whether the feedback signal is received within the preset time.

[0104] Step S503: Check whether it is whitelist information. If yes, go to step S510; if not, go to step S520.

[0105] In step S503, after determining that the in-vehicle information is not the target information, it means that the user has not operated the target device. In order to further determine the content of the in-vehicle information, it is necessary to re-determine whether the in-vehicle information is whitelisted information. Then, the corresponding operation is performed based on the result of the re-confirmation.

[0106] Step S510: Ignore the information.

[0107] The whitelist information may include interference information generated by air conditioners and windows, etc. When it is determined to be whitelist information, the information is ignored.

[0108] Step S520: The error prompt provides detailed guidance according to the configuration.

[0109] In step S520, if the in-vehicle information is determined to be non-target information and not whitelisted, it indicates that the user may have operated another device or that the user's operation on the target device is incorrect. In this case, an error message is output to remind the user of the incorrect operation. The process then returns to step S403, where the first message is re-output to prompt the user to operate the target device according to the first message.

[0110] Step S504: Is this the last step? If so, end the operation process; if not, proceed to step S505.

[0111] Step S505: Repeat the timeout determination until the next step is continued.

[0112] In step S505, if the current step is not the last step, the third information, the fourth information, etc. are continuously output, and the timing operation is repeatedly performed when the third information and the fourth information are output.

[0113] The third information includes a third step of operating the target device, which is the next step of the second step; and the fourth information includes a fourth step of operating the target device, which is the next step of the third step.

[0114] Of course, in other implementations, more or less information such as the fifth information and the sixth information may also be included.

[0115] The present application provides a multi-modal interactive vehicle usage method. This method addresses the current shortcomings in closed-loop information presentation by introducing multi-modal information into the human-machine interaction process of the vehicle manual. This incorporates feedback from the user's interaction with the vehicle into the interaction process, helping users more effectively understand, learn, and master relevant vehicle control knowledge and confirm whether their operations are correct, thereby achieving a closed-loop information acquisition process. Furthermore, the method can also obtain user operation feedback to identify whether the user's operation is correct. For incorrect operations, more detailed instructions and guidance can be provided based on pre-set settings to help users truly master the vehicle's operation.

[0116] Corresponding to the aforementioned embodiment of the vehicle use method, the present application also provides an embodiment of a vehicle use device based on multimodal interaction.

[0117] Figure 3 This is a structural block diagram of a vehicle usage device based on multimodal interaction provided in an embodiment of the present application.

[0118] See also Figure 3 The vehicle use device 100 includes an output module 101 , a collection module 102 , an acquisition module 103 , a determination module 104 and a display module 105 .

[0119] The output module 101 is configured to output first information in response to question information input by a user, wherein the question information includes an operation question of a target device, and the first information includes a first step of operating the target device.

[0120] That is, the output module 101 can be used to execute step S100 in the above-mentioned method embodiment.

[0121] The collection module 102 is used to collect in-vehicle information in response to receiving a feedback signal within a preset time, wherein the feedback signal is generated by a user operating an in-vehicle device, and the in-vehicle information includes working information of multiple devices in the vehicle.

[0122] That is, the collection module 102 can be used to execute step S200 in the above-mentioned method embodiment.

[0123] The output module 101 can also be used to output second information in response to the in-vehicle information including target information; wherein the target information includes work information of the target device being executed according to the first step, and the second information includes the second step of operating the target device, and the second step is the next step of the first step.

[0124] That is, the output module 101 can also be used to execute step S300 in the above-mentioned method embodiment.

[0125] In some feasible implementations, the output module 101 may also be configured to output a timeout prompt message in response to not receiving a feedback signal within a preset time, and re-determine whether a feedback signal is received within the preset time.

[0126] That is, the output module 101 can also be used to execute step S110 in the above-mentioned method embodiment.

[0127] In some feasible implementations, the output module 101 may also be configured to ignore the in-vehicle information in response to the in-vehicle information not including the target information and the in-vehicle information being whitelist information.

[0128] That is, the output module 101 can also be used to execute step S210 in the above-mentioned method embodiment.

[0129] In some feasible implementations, the output module 101 may further output an error prompt message and re-output the first information in response to the in-vehicle information not including the target information and the in-vehicle information not being whitelist information.

[0130] That is, the output module 101 can also be used to execute step S220 in the above-mentioned method embodiment.

[0131] The acquisition module 103 is used to acquire operation information, wherein the operation information includes all steps of answering the question information.

[0132] The determination module 104 is configured to determine whether the vehicle is configured with interactive content.

[0133] That is, the determination module 104 can also be used to execute step S102 in the above-mentioned method embodiment.

[0134] The output module 101 is further configured to output first information in response to the vehicle being configured with interactive content.

[0135] That is, the output module 101 can also be used to execute step S103 in the above-mentioned method embodiment.

[0136] The display module 105 is configured to statically display operation information in response to the vehicle not being configured with interactive content.

[0137] That is, the display module 105 can also be used to execute step S104 in the above-mentioned method embodiment.

[0138] In this way, the vehicle user device 100 can process the original static image or video feedback content. For the operational instructions, it is divided into actual operation steps based on the operational purpose as the ultimate goal, and then presented to the user step by step in a sequential order. After the user issues a request, the user device 100 will determine the answer type based on the actual matched question. If the associated answer is an interactive answer, the first information is returned to the client for display, prompting the user to take action; otherwise, it is displayed as a standard multimedia answer, and the process ends.

[0139] After the user obtains the first information, he operates the target device according to the first step, and after using the device 100 to obtain the corresponding correct signal feedback within the preset time, he continues to the step-by-step display of the next step. If the signal return timeout occurs, it means that the user has not performed any operation, or the signal returns an error, which means that the user has made an error in the operation, then a corresponding prompt will be given to guide the user to perform the correct operation. During the process, in order to ensure that the user can use the functions in the car normally without being affected, whitelist information identification is required. When the vehicle returns such information, it is not considered that the user has made an erroneous operation, but the signal is ignored. The process ends until all the configured operating steps are completed normally. The user can exit the process at any stage on his own, and the user will be asked whether to reset the previous operation before exiting.

[0140] The vehicle use device 100 provided in the embodiment of the present application can address the current defects in the closed loop of information presentation by introducing multimodal information in the human-computer interaction process of the automobile manual, adding the interactive signal feedback between the user and the car to the interactive process, helping the user to more effectively understand, learn, and master the relevant vehicle operation knowledge, and confirm whether their own operation is correct, thereby improving the accuracy and fun of the user's acquisition and mastery of automobile operation knowledge. Furthermore, the device can also identify whether the user's operation is correct by obtaining user operation feedback. For incorrect operations, it can provide more detailed instructions and guidance according to pre-set settings to help users truly master the operation of the vehicle. In addition, automobile manufacturers can also collect relevant error signal data to identify problems or unfriendly aspects in the manual or industrial design, and optimize and iterate in subsequent versions to continuously improve the product experience.

[0141] It should be noted that those skilled in the art will easily conceive of other implementations of the present application after considering the specification and practicing the application disclosed herein.

[0142] This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, and the true scope of this application is indicated by the claims.

[0143] It should be understood that the present application is not limited to the exact structures described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A vehicle use method based on multimodal interaction, characterized in that: include: Outputting first information in response to question information input by a user, wherein the question information includes an operation question of a target device, and the first information includes a first step of operating the target device; In response to receiving a feedback signal within a preset time, collecting in-vehicle information, wherein the feedback signal is generated by a user operating an in-vehicle device, and the in-vehicle information includes operating information of multiple in-vehicle devices; In response to the in-vehicle information including target information, second information is output; wherein, the target information includes work information of the target device being executed according to the first step, and the second information includes a second step of operating the target device, and the second step is the next step of the first step.

2. The vehicle use method based on multimodal interaction according to claim 1, characterized in that: In response to not receiving the feedback signal within the preset time, outputting a timeout prompt message, and re-determining whether the feedback signal is received within the preset time.

3. The vehicle use method based on multimodal interaction according to claim 1, characterized in that: In response to the in-vehicle information not including the target information and the in-vehicle information being whitelist information, the in-vehicle information is ignored.

4. The vehicle use method based on multimodal interaction according to claim 3, characterized in that: In response to the in-vehicle information not including the target information and the in-vehicle information not being the whitelist information, an error prompt message is output, and the first information is re-output.

5. The vehicle use method based on multimodal interaction according to claim 1, characterized in that: After responding to the question information input by the user, the vehicle use method further includes: Acquiring operation information, wherein the operation information includes all steps of operating the target device; determining whether the vehicle is configured with interactive content; In response to the vehicle being configured with interactive content, outputting the first information; In response to the vehicle not being configured with interactive content, the operation information is statically displayed.

6. A vehicle use device based on multimodal interaction, characterized in that: include: an output module configured to output first information in response to question information input by a user, wherein the question information includes an operation question of a target device, and the first information includes a first step of operating the target device; a collecting module configured to collect in-vehicle information in response to receiving a feedback signal within a preset time, wherein the feedback signal is generated by a user operating an in-vehicle device, and the in-vehicle information includes operating information of multiple in-vehicle devices; The output module is also configured to output second information in response to the in-vehicle information including target information; wherein the target information includes work information of the target device executed according to the first step, and the second information includes a second step of operating the target device, and the second step is the next step of the first step.

7. The vehicle use device based on multimodal interaction according to claim 6, characterized in that: The output module is further configured to, in response to not receiving the feedback signal within the preset time, output a timeout prompt message and re-determine whether the feedback signal is received within the preset time.

8. The vehicle use device based on multimodal interaction according to claim 6, characterized in that: The output module is further configured to ignore the in-vehicle information in response to the in-vehicle information not including the target information and the in-vehicle information being whitelist information.

9. The vehicle use device based on multimodal interaction according to claim 8, characterized in that: The output module is further configured to, in response to the in-vehicle information not including the target information and the in-vehicle information not being the whitelist information, output an error prompt message and re-output the first information.

10. The vehicle using device based on multimodal interaction according to claim 6, characterized in that: Also includes: Acquisition module, determination module and display module; The acquisition module is configured to acquire operation information, wherein the operation information includes all steps of answering the question information; The determination module is configured to determine whether the vehicle is configured with interactive content; The output module is further configured to output the first information in response to the vehicle being configured with interactive content; The display module is configured to statically display the operation information in response to the vehicle not being configured with interactive content.

Images