Vehicle interaction system and interaction method

By using virtual models and sensing devices in vehicles, combined with augmented reality guidance and virtual games, the problem of drivers in the prior art is solved, which improves the efficiency and accuracy of troubleshooting and reduces fault handling time and cost.

CN118991807APending Publication Date: 2024-11-22FORD GLOBAL TECH LLC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310557402.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-17
Publication Date
2024-11-22

AI Technical Summary

Technical Problem

Existing vehicle interaction systems fail to effectively teach drivers to properly handle faults when a vehicle failure occurs, resulting in increased troubleshooting time and cost.

Method used

By introducing processors and memory into the vehicle, displaying virtual vehicle models, receiving fault information and displaying prompts on the model, using sensor devices to obtain occupant operation data, providing feedback information and augmented reality guidance, guiding occupants to perform repair operations, and through Virtual game simulation fault repair process.

Benefits of technology

It significantly improves the driver's repair efficiency and accuracy in vehicle failure situations, reduces the time and cost of troubleshooting, and enhances the driver's repair capabilities.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118991807A_ABST
    Figure CN118991807A_ABST
Patent Text Reader

Abstract

According to one aspect of the application, an interactive system for a vehicle and a vehicle using the interactive system are provided, the interactive system comprises a processor and a memory, the memory stores instructions executable by the processor, and when the instructions are executed by the processor, the following steps are implemented: displaying a virtual model of the vehicle; fault information of the vehicle is received, and a first fault prompt is displayed on the virtual model based on the fault information; instructing the occupant to perform a first repair operation based on the first fault cue and at least in part through the virtual model; acquiring real-time data of the first repair operation implemented by the passenger through a sensing device; first feedback information is provided based on the first repair operation. According to the other aspect of the invention, the invention further provides an interaction method for the vehicle.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application generally relates to the field of vehicle technology, and more specifically, to a vehicle interaction system and a vehicle interaction method. Background Art

[0002] In modern vehicles, methods and systems for providing the motor vehicle driver with information about messages from vehicle monitoring systems are generally known. For example, engine temperature, engine speed, external ambient temperature, ABS system (Antilock Brake System), brake system, tire pressure, etc. are monitored and their status is signaled or a fault message is transmitted in the event of a fault.

[0003] The above fault messages provide the driver with information about the vehicle's operating status. Some fault messages require the driver to drive the vehicle to a repair shop or manufacturer for repair, while some fault messages can be eliminated by the driver's manual operation without sending the vehicle to a repair shop, which saves time and cost for vehicle troubleshooting. However, since most drivers lack knowledge and experience in troubleshooting vehicle faults, when a vehicle fault occurs, the driver does not know how to properly handle it to eliminate the vehicle fault.

[0004] That is to say, the current solutions do not focus on the technical problem of how to better teach the vehicle driver to correctly troubleshoot the vehicle fault through the vehicle's interactive system. Therefore, how to provide more intelligent interaction to help the driver troubleshoot the vehicle fault smoothly is a research direction of the applicant of this application.

[0005] Application Contents

[0006] This disclosure summarizes various aspects of the embodiments and should not be used to limit the claims. Other embodiments are conceivable based on the technology described herein, which will be apparent to those of ordinary skill in the art after studying the following drawings and detailed descriptions, and these embodiments are intended to be included within the scope of this application.

[0007] According to one aspect of the present application, an interactive system for a first vehicle is provided, including a processor and a memory, wherein the memory stores processor-executable instructions, and when the instructions are executed by the processor, the following steps are implemented:

[0008] Displaying a virtual model of the vehicle;

[0009] receiving fault information of the vehicle and displaying a first fault prompt on the virtual model based on the fault information;

[0010] Instructing the occupant to perform a first repair operation based on the first fault indication and at least in part through the virtual model;

[0011] Acquiring real-time data of the first repair operation performed by the occupant through a sensor device;

[0012] First feedback information is provided based on the first repair operation.

[0013] In one embodiment, the sensing device includes at least one of a vehicle sensor and a smart device communicatively connected to the vehicle.

[0014] In another embodiment, the step of teaching the occupant to perform the first repair operation further includes simultaneously providing augmented reality guidance.

[0015] In yet another embodiment, the steps include:

[0016] Call the camera device and obtain the real-time data captured by the camera;

[0017] Compare real-time data with virtual models; and

[0018] When it is determined that the camera device is aimed at the vehicle component corresponding to the first repairing operation, the occupant is reminded to perform the first repairing operation through a visual signal and / or an auditory signal.

[0019] In yet another embodiment, the correct position of the vehicle component corresponding to the first repair operation is marked for the user by an indicator and / or a focus frame on the virtual model and / or on a display of the vehicle and / or a display of a camera device.

[0020] In yet another embodiment, when the occupant performs a first repair operation, first feedback information is displayed in real time on the virtual model according to real-time data of the first repair operation, and the first feedback prompt includes a correct step prompt and / or an incorrect step prompt.

[0021] In yet another embodiment, when the occupant performs the first repair operation, the repair progress of the first fault prompt is updated in real time on the virtual model according to the real-time data.

[0022] In yet another embodiment, the first fault prompt is an under-pressure prompt, and the under-pressure prompt is displayed at the position of the corresponding virtual wheel of the virtual model;

[0023] The camera device is called according to the low tire pressure prompt to obtain the real-time data captured by the camera device, and the correct position of the wheel and the inflation port corresponding to the virtual wheel is marked for the user according to the real-time data and the virtual model.

[0024] During the inflation operation, the real-time data of the tire pressure is obtained and the repair progress is updated on the virtual model. When the tire pressure reaches a first tire pressure threshold, the user is reminded to stop inflation through visual and / or auditory signals.

[0025] In yet another embodiment, the steps further include:

[0026] Providing a first virtual game associated with a component of a vehicle, which includes preset operation steps;

[0027] Receive user input;

[0028] Provide feedback based on how well the user input corresponds to the preset operation steps.

[0029] In yet another embodiment, the virtual model of the vehicle is rendered using vehicle modeling data matched to the vehicle.

[0030] According to another aspect of the present application, an interaction method is provided, the method comprising:

[0031] Displaying a virtual model of the vehicle;

[0032] receiving fault information of the vehicle and displaying a first fault prompt on the virtual model based on the fault information;

[0033] Instructing the occupant to perform a first repair operation based on the first fault indication and at least in part through the virtual model;

[0034] Acquire real-time data of the crew's first repair operation;

[0035] First feedback information is provided based on the first repair operation.

[0036] In yet another embodiment, it further comprises acquiring real-time data of the first repair operation of the occupant through a sensing device, wherein the sensing device comprises at least one of a vehicle sensor and an intelligent device communicatively connected to the vehicle.

[0037] In yet another embodiment, the step of teaching the occupant to perform the first repair operation further includes simultaneously providing augmented reality guidance.

[0038] In yet another embodiment, the method further comprises:

[0039] Call the camera device and obtain the real-time data captured by the camera;

[0040] Compare real-time data with virtual model information; and

[0041] When it is determined that the camera device is aimed at the vehicle component corresponding to the first repairing operation, the occupant is reminded to perform the first repairing operation through a visual signal and / or an auditory signal.

[0042] In yet another embodiment, the method further comprises: marking the correct position of the vehicle component corresponding to the first repair operation for the user through an indicator and / or a focus frame on the display of the virtual model and / or the vehicle and / or the display of the camera device.

[0043] In yet another embodiment, it further comprises: when the occupant performs the first repair operation, first feedback information is displayed in real time on the virtual model according to real-time data of the first repair operation, and the first feedback prompt includes a correct step prompt and / or an incorrect step prompt.

[0044] In another embodiment, the method further comprises: when the occupant performs the first repair operation, updating the repair progress of the first fault indication on the virtual model in real time according to the real-time data.

[0045] In yet another embodiment, the first fault prompt is an insufficient tire pressure prompt, and the insufficient tire pressure prompt is displayed on a corresponding virtual wheel of the virtual model;

[0046] The camera device is called according to the low tire pressure prompt to obtain real-time data captured by the camera device, and the correct positions of the wheels and inflation ports corresponding to the virtual wheels are marked for the user according to the real-time data and the virtual model.

[0047] In yet another embodiment, during the inflation operation performed by the user, real-time data of tire pressure of the vehicle tire is obtained and updated on the virtual model, and when the tire pressure reaches a first tire pressure threshold, the user is reminded to stop inflation through visual and / or auditory signals;

[0048] and, when the tire pressure exceeds a second tire pressure threshold, displaying a second fault prompt at the virtual wheel position of the virtual model;

[0049] Among them, the second tire pressure threshold is greater than the first tire pressure threshold.

[0050] According to another aspect of the present application, a vehicle is provided, wherein the vehicle includes an interactive system, the interactive system includes a processor and a memory, the memory stores processor-executable instructions, and when the instructions are executed by the processor, the following steps are implemented:

[0051] Displaying a virtual model of the vehicle;

[0052] receiving fault information of the vehicle and displaying a first fault prompt on the virtual model based on the fault information;

[0053] Instructing the occupant to perform a first repair operation based on the first fault indication and at least in part through the virtual model;

[0054] Acquiring real-time data of the first repair operation of the occupant through a sensor device;

[0055] First feedback information is provided based on the first repair operation. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] For a better understanding of the present application, reference may be made to the embodiments shown in the following drawings. The components in the drawings are not necessarily drawn to scale, and related elements may be omitted, or in some cases the scale may have been exaggerated in order to emphasize and clearly illustrate the novel features described herein. In addition, as is known in the art, the system components may be arranged differently. Moreover, in the drawings, throughout the several views, the same reference numerals represent corresponding parts.

[0057] Figure 1 A vehicle including an interactive system according to the present application is shown;

[0058] Figure 2 An exemplary block topology diagram of an interactive system according to the present application is shown;

[0059] Figure 3 A block diagram showing the steps implemented when the executable instructions included in an embodiment of the interactive system according to the present application are executed;

[0060] Figure 4 An interactive interface diagram according to an embodiment of the interactive system of the present application is shown;

[0061] Figure 5 Another interactive interface diagram according to an embodiment of the interactive system of the present application is shown;

[0062] Figures 6A-6C A schematic diagram showing augmented reality guidance when executing an interactive method according to an embodiment of an interactive system of the present application is shown;

[0063] Figure 7 A schematic diagram showing augmented reality guidance when executing an interaction method according to another embodiment of the interaction system of the present application is shown;

[0064] Figures 8A-8C A schematic diagram of a virtual game operation interface when an interactive method is executed according to an embodiment of an interactive system of the present application is shown. DETAILED DESCRIPTION

[0065] Embodiments of the present disclosure are described below. However, it should be understood that the disclosed embodiments are merely examples, and other embodiments may take various alternative forms. The drawings are not necessarily drawn to scale; certain features may be exaggerated or minimized to show the details of specific components. Therefore, the specific structural and functional details disclosed herein should not be interpreted as limiting, but merely as a representative basis for teaching those skilled in the art to use the present application in various ways. As will be understood by those of ordinary skill in the art, the various features shown and described with reference to any one of the figures may be combined with the features shown in one or more other figures to produce embodiments that are not explicitly shown or described. The combination of features shown provides representative embodiments for typical applications. However, various combinations and modifications of features consistent with the teachings of the present disclosure may be desirable for certain specific applications or embodiments.

[0066] One or more embodiments of the present application will be described below in conjunction with the accompanying drawings. The flowchart illustrates the process performed by the system. It is understood that the execution of the flowchart does not need to be performed in sequence, one or more steps may be omitted, one or more execution steps may be added, and one or more steps may be performed in sequence or in reverse order, and even in some embodiments, one or more steps may be performed simultaneously.

[0067] The following embodiments involve "user", "driver", "virtual three-dimensional model", "other clients of the same user", etc. In one or more embodiments, they are used to illustrate the interaction between the vehicle and the user. In some cases, the roles can be exchanged or other names can be used without departing from the spirit of the present application.

[0068] One or more embodiments of the present application may be applicable to a variety of vehicles. For example, with reference to the vehicle 100 in the accompanying drawings, it may be a standard gasoline-powered vehicle, a hybrid vehicle, an electric vehicle, a fuel cell vehicle, and / or any other type of vehicle, and may also be a bus, a ship, or an aircraft. The vehicle includes components related to mobility, such as an engine, an electric motor, a transmission, a suspension, a drive shaft, and / or wheels, etc. The vehicle may be non-autonomous, semi-autonomous (e.g., some conventional motor functions are controlled by the vehicle) or autonomous (e.g., motor functions are controlled by the vehicle without direct input from the driver).

[0069] Figure 1A vehicle 100 according to the present application is shown, which includes an interactive system 1. In this embodiment, the interactive system 1 is shown as an in-vehicle interactive system 1 located in the vehicle. Those skilled in the art will appreciate that a portion or multiple portions of the interactive system 1 may also be located outside the vehicle, for example, the memory may be located in the cloud, and the processor may also be located in the cloud in some embodiments. In yet other embodiments, the interactive system may also include servers located outside the vehicle that communicate with each other via a communication protocol. It may also be appreciated that in other application scenarios, one or more embodiments of the present application may also include interactive methods and systems utilizing one or more mobile devices. Figure 2 An embodiment of an in-vehicle interactive system is shown. Figure 2 As further shown in the exemplary block topology diagram of the vehicle interaction system, in the context of the in-vehicle interaction system, the vehicle interaction system 1 may include: a processor 3 and a memory 7, the memory 7 stores processor executable instructions, and the instructions are executed by the processor 3 to implement Figures 3 to 8C Follow one or more of the steps shown in .

[0070] Next, combine Figure 2 To illustrate an exemplary hardware environment of a vehicle interaction system (also referred to as a vehicle computing system VCS (vehicle computing system)) 1 for a vehicle 100. An example of an operating system installed in the vehicle interaction system 1 is the SYNC or SYNC+ system manufactured by Ford Motor Company. A vehicle equipped with the vehicle interaction system 1 may include a display 4 located in the vehicle, and the display 4 may be one or more to present an interactive interface alone or in combination. By way of example and not limitation, display types may include CRT (cathode ray tube) displays, LCD (liquid crystal) displays, LED (light emitting diode) displays, PDP (plasma display), laser displays, VR (virtual reality) displays, and the like. In other embodiments, the display 4 may also be a projection display projected onto a windshield or side windows or any suitable surface of the vehicle.

[0071] The processor (CPU) 3 in the vehicle interaction system 1 can control at least part of the operation of the vehicle interaction system 1. In one or more embodiments, the processor 3 is capable of executing on-board processing instructions and programs, for example, the processor can process one or more executable instructions for the processor of the vehicle interaction system 1. The processor 3 can be connected to a non-persistent memory 5 and a persistent memory 7. The memories 5 and 7 can include volatile and non-volatile memories such as read-only memory (ROM), random access memory (RAM) and keep-alive memory (KAM). It can be understood by those skilled in the art that in one or more embodiments of the present application, any number of known storage devices (such as programmable read-only memory (PROM), EPROM (electrically programmable read-only memory), EEPROM (electrically erasable programmable read-only memory), flash memory or any other electronic, magnetic, optical or combined storage device capable of storing data) can be used to implement the memory 5 and / or the memory 7. The memory 5 and the memory 7 can store, for example, processor executable instructions of the vehicle interaction system 1.

[0072] In one or more embodiments, the processor 3 may also be configured to receive and process a plurality of different inputs, thereby allowing a user to interact with the processor. In an illustrative embodiment, the inputs include a microphone 29 configured to receive voice signals, an auxiliary input 25 (e.g., a CD (optical disc), a tape, etc.) for an input 33, a USB (universal serial bus) input 23, a GPS (global positioning system) input 24, and a Bluetooth input 15. An input selector 51 is also provided to allow a user to switch between a plurality of inputs. The inputs from the microphone and the auxiliary connector may be converted from analog signals to digital signals by a converter 27 before being passed to the processor 3. In addition, although not shown, a plurality of vehicle components and auxiliary components that communicate with the vehicle interaction system may use a vehicle network (e.g., but not limited to, a CAN (controller area network) bus) to transmit data to or receive data from the vehicle interaction system 1 (or its components).

[0073] Additionally, the processor 3 may communicate with a plurality of vehicle sensors and actuators via an input / output (I / O) interface, which may be implemented as a single integrated interface that provides a plurality of raw data or signal adjustments, processing and / or conversions, short circuit protection, etc. Further, by way of example and not limitation, the types of sensors that are communicatively connected to the processor 3 may include, for example, cameras, ultrasonic sensors, pressure sensors, fuel level sensors, engine speed sensors, temperature sensors, photoplethysmographic sensors, etc., to identify user interaction information such as button presses, voice, touch, text input, facial expressions or movements, hand gestures or movements, head gestures or movements, and body gestures or movements, as well as to identify vehicle information such as fuel level, powertrain faults, in-vehicle temperature, etc.

[0074] The outputs of the vehicle interaction system 1 may include, but are not limited to, a display 4, a speaker 13, and various actuators. The speaker 13 may be connected to an amplifier 11 and receive its signal from the processor 3 via a digital-to-analog converter 9. The output of the system may also be output to a remote Bluetooth device (e.g., a personal navigation device 54) or a USB device (e.g., a vehicle navigation device 60) along the bidirectional data streams shown at 19 and 21, respectively.

[0075] In one illustrative embodiment, the vehicle interaction system 1 uses the antenna 17 of the Bluetooth transceiver 15 to communicate with a user's nomadic device 53 (e.g., a cell phone, smart phone, personal digital assistant, etc.). The nomadic device 53 can in turn communicate 59 with the cloud 125 outside the vehicle 31 through, for example, communication 55 with a cellular tower 57. In some embodiments, the cellular tower 57 can be a Wi-Fi (wireless local area network) access point. Signal 14 represents an exemplary communication between the nomadic device 53 and the Bluetooth transceiver 15. Pairing of the nomadic device 53 and the Bluetooth transceiver 15 can be indicated by a button 52 or similar input, thereby indicating to the processor 3 that the vehicle-mounted Bluetooth transceiver is to be paired with the Bluetooth transceiver in the nomadic device.

[0076] Data may be communicated between the processor 3 and the cloud 125 using, for example, a data-plan, data over voice, or dual tone multi-frequency (DTMF) tones associated with the nomadic device 53. Alternatively, the vehicle interaction system 1 may include an onboard modem 63 having an antenna 18 to communicate 16 data between the processor 3 and the nomadic device 53 via a voice band. The nomadic device 53 may then be able to communicate 59 with the cloud 125 outside of the vehicle 31, for example, via communication 55 with a cellular tower 57. In some embodiments, the modem 63 may establish communication 20 directly with a cellular tower for further communication with the cloud 125. As a non-limiting example, the modem 63 may be a USB cellular modem and the communication 20 may be a cellular communication.

[0077] In one illustrative embodiment, the processor is provided with an operating system including an API (application programming interface) for communicating with the modem application software. The modem application software can access an embedded module or firmware on the Bluetooth transceiver 15 to complete wireless communication with a remote Bluetooth transceiver (e.g., a Bluetooth transceiver located in a nomadic device). Bluetooth is a subset of the IEEE802PAN (Personal Area Network) protocol. The IEEE 802LAN (Local Area Network) protocol includes Wi-Fi and has considerable cross-functionality with IEEE802PAN. Both are suitable for wireless communication in a vehicle. Other communication methods may include free space optical communication (e.g., infrared data protocol, IrDA) and non-standard consumer infrared (consumer IR) protocols, etc.

[0078] In one embodiment, the nomadic device 53 may be a wireless local area network (LAN) device capable of communicating via, for example, an 802.11 network (e.g., Wi-Fi) or a WiMax (Worldwide Interoperability for Microwave Access) network. Other sources that may interact with the vehicle include a personal navigation device 54 having, for example, a USB connection 56 and / or an antenna 58, or a vehicle navigation device 60 having a USB 62 or other connection, an onboard GPS device 24, or a remote navigation system (not shown) connected to the cloud 125.

[0079] In addition, the processor 3 may communicate with a number of other auxiliary devices 65. These auxiliary devices 65 may be connected via wireless connections 67 or wired connections 69. Similarly or alternatively, the CPU may connect to a vehicle-based wireless router 73 using, for example, a Wi-Fi 71 transceiver. This may allow the CPU to connect to a remote network within range of the local router 73. The auxiliary devices 65 may include, but are not limited to, personal media players, wireless health devices, mobile computers, and the like.

[0080] Specifically, refer to Figure 3 To further illustrate the concept of the present application, there is shown a step process 300 implemented when the executable instructions included in an embodiment of the vehicle interaction system 1 of the present application are executed. In the described embodiment, the steps include: the process 300 starts at box 305, for example but not limited to any time when the vehicle interaction system 1 detects that the user is in the vehicle 100. In one example, the vehicle interaction system 1 can detect that the user is inside the vehicle 100 through sensors such as microphones, cameras, touch sensors, pressure sensors, or pairing of roaming devices. In some usage scenarios, such as a scenario where a family shares one or more vehicles, when the vehicle is used by one of the family members, other family members, that is, authorized users, can also use the non-vehicle client to enter the process. At this time, box 305 can start when the associated authorized user opens the corresponding program interface. For example, a user can open the corresponding program interface through, for example, Ford's Ford Lincoln The corresponding management interface can be entered through the application program or applet, and the status change of the corresponding vehicle can be viewed. It can be understood by those skilled in the art that users with vehicle management use authority can view the vehicle status or management interface through the APP during the parking period.

[0081] In some embodiments, the vehicle interaction system 1 can use, for example, a microphone, a camera, a touch sensor, an ultrasonic sensor, etc. to recognize the user's input, and the user's input can include the user's voice, touch, text input, predetermined gestures or actions, and obtain the user's interaction information therefrom. Combining several interaction methods can significantly improve the interaction efficiency with the user.

[0082] Then, process 300 proceeds from block 305 to block 310, where the Figure 4 It can be seen that in this embodiment, a virtual three-dimensional model 400 of the vehicle 100 is displayed on the display 4 of the interactive system 1 of the vehicle 100. The virtual three-dimensional model 400 can be obtained by three-dimensionally rendering the vehicle model data matching the vehicle 100 and displayed on the display 4. It should be understood that in other embodiments, the virtual model can also be a virtual two-dimensional model based on the vehicle model data.

[0083] Next, in block 315, the interactive system 1 of the vehicle 100 may receive the vehicle status information and / or fault information from the plurality of sensors of the vehicle 100, and display the received status information and / or fault information at the corresponding position of the virtual three-dimensional model 400 of the vehicle 100, for example Figure 4 As shown, information 410 of insufficient tire pressure is displayed at the corresponding wheel position of the virtual three-dimensional model 400. In this embodiment, the wheel with insufficient tire pressure is the right rear wheel of the vehicle, and information 420 of insufficient front windshield washer fluid is displayed at a position above the engine hood of the virtual three-dimensional model 400 corresponding to the liquid filling port.

[0084] In one or more embodiments, examples of status information or fault information may be insufficient tire pressure of the vehicle 100, or, for example, a headlight failure, an electric steering system failure, an overly high coolant temperature, etc. These faults can be displayed as reminders through the virtual three-dimensional model 400. In this embodiment, when the right rear tire of the vehicle is under-inflated, a "low tire pressure" logo and / or text may be displayed near the right rear tire of the virtual three-dimensional model 400. Such information may be selectively displayed in the form of a pop-up window or bubble. It is understood that the attached Figure 4The low tire pressure information 410 and the insufficient washer fluid information 420 are two of the multiple vehicle status or vehicle fault information that can be displayed by the virtual three-dimensional model 400, and do not represent a limitation on the vehicle status or vehicle fault information that can be displayed by the virtual three-dimensional model 400.

[0085] Then in box 320, the user can click on the low tire pressure mark 410 on the display 4. According to the user's selection of the low tire pressure mark 410 and in combination with FIG. 6, it can be seen that the interactive system 1 provides the user with multiple options corresponding to the fault information selected by the user through the display 4. In this embodiment, the multiple options may at least include 1. Navigate to the maintenance site, 2. Enter the fault repair guide, 3. Call customer service, and 4. More options. The sphere 510 on the left side of the display is used to prompt the user that the interactive system 1 is interacting with the user through the state of the sphere 510 (such as vibration, breathing-like expansion and contraction, color change) when the user interacts with the interactive system 1 by voice to listen to the user's voice command. The user can select from the multiple options according to his or her desired troubleshooting method as needed.

[0086] In another embodiment, the vehicle's interactive system 1 may evaluate the severity of the fault based on the received fault information, and compare the evaluation result with a preset threshold. When the evaluation result exceeds the preset threshold, the multiple options provided to the user do not include the option of "entering fault repair guidance". This is because when the evaluation result exceeds the preset fault threshold, it may indicate that the difficulty of repairing the fault is not suitable for handling by non-professional maintenance personnel such as vehicle occupants.

[0087] In one example, when the user selects "Enter fault repair guidance", the interactive system 1 enters the guidance of the vehicle fault repair method and provides augmented reality guidance to the user. When performing augmented reality guidance, the interactive system 1 can call the vehicle-mounted and / or external sensing device, which can be a vehicle sensor and / or an intelligent device connected to the vehicle in communication. In this embodiment, the sensing device is a smart phone with a camera connected to the vehicle in communication. When performing augmented reality guidance, the interactive system 1 can send an instruction to turn on the camera of the user's smart phone, and prompt the user on the screen of the smart phone to point the camera to the vehicle part that needs to be repaired. When the user moves the camera and aligns it with the part that needs to be repaired, the user is prompted that the camera has been aligned with the corresponding part by displaying "Already aligned" on the screen of the smart phone, and subsequent repair operations can be performed. It can be understood that the notification of the corresponding part being aligned can also be prompted to the user in multiple ways such as mobile phone vibration, mobile phone or vehicle 100 speakers.

[0088] Subsequently, in box 325, process 300 obtains the real-time image data captured by the camera device through the interactive system 1, and superimposes the indicator or focus frame on the acquired real-time image data, so as to guide the user to the correct position where the current step needs to be operated in the actual repair operation. When the user begins to follow the image guided by the augmented reality to perform the actual repair operation, in box 330, the interactive system 1 obtains the real-time data captured by the camera device, and provides feedback based on the acquired real-time data. In one embodiment, the user's current operation step can be prompted by feedback information whether it is correct. If the current step is correct, the user is prompted by sound and / or text information, for example, by displaying the correct information of the current step on the screen of a smart phone or by a specific prompt tone. If the current step is incorrect, the user can also be prompted by text and / or sound information, and the correct position and correct operation method to be operated can be guided by enhanced display.

[0089] In one embodiment, when the user performs actual repair operations on the vehicle, the interactive system 1 of the vehicle 100 updates the repair progress of the fault in real time on the virtual three-dimensional model based on the received data information. For example, when inflating the vehicle tires, the interactive system 1 can compare the recommended tire pressure value and the current tire pressure value, and display the current inflation progress next to the tire of the virtual three-dimensional model based on the comparison results.

[0090] exist Figures 6A-6C In one embodiment of the present invention shown, when the user selects the tire pressure insufficient information 410 displayed in the virtual three-dimensional model 400, the interactive system 1 provides the user with multiple options corresponding to eliminating the tire pressure fault information through the display 4. In this embodiment, the user selects "Enter fault repair guidance", and then the interactive system 1 sends a command to the user's smartphone connected to the vehicle to turn on the smartphone's camera device, and prompts the user to aim the camera device at the right rear wheel of the vehicle through the smartphone screen. When the user aims the camera device at the right rear wheel of the vehicle, the smartphone screen displays the message "Already aligned" and emits a corresponding alignment prompt sound to inform the user that the alignment operation has been completed.

[0091] Then Figures 6A-6C As shown, after acquiring the real-time data captured by the camera device of the smart phone, the interactive system 1 superimposes the indicator shown in the figure on the picture captured by the camera device to prompt the user the correct inflation position. Fig. 6A As shown, the focus frames 611 and 612 superimposed in the image 610 displayed on the screen of the smart phone correspond to the inflation port of the wheel and the socket of the inflation device, respectively, and after the user correctly connects the inflation device to the inflation port, the focus frames 611 and 612 are superimposed on the image 610 displayed on the screen of the smart phone, respectively. Figure 6BAs shown in FIG. 1 , focus frames 621 and 622 are superimposed on the image 620 of the inflator to correspond to the tire repair kit opening button and the inflator start button, respectively. After the user correctly starts the inflator, as shown in FIG. Figure 6C As shown, the focus frame prompts the user that the current pressure value of the wheel can be read through the pressure gauge 623 on the inflation device.

[0092] When the user executes the above Figures 6A-6C During the operation shown, the interactive system 1 obtains real-time data of the inflation operation and displays the current inflation progress on the virtual three-dimensional model 400, and when the tire pressure has reached a preset tire pressure threshold, the smartphone and / or the interactive system 1 sends a visual and / or auditory signal to remind the user to stop inflation.

[0093] Then in Figure 7 In the embodiment shown, when the user selects Figure 4 When the fault message 420 of insufficient windshield washer fluid is shown on the virtual three-dimensional model 400, the interactive system 1 can send a command to the user's smartphone connected to the vehicle communication to turn on the smartphone's camera according to the user's choice, and prompt the user through the smartphone screen to open the vehicle's hood and aim the camera at the vehicle's engine compartment after the hood is opened. When the user aims the camera at the filling port of the windshield washer fluid in the vehicle's engine compartment, the smartphone screen displays an "Aligned" message and emits a corresponding alignment prompt tone to inform the user that the alignment operation is complete. The interactive system 1 then acquires the data captured by the camera and displays it as follows. Figure 7 The indicator 711 shown is superimposed on the image 710 captured by the camera device and is aligned with the filling port of the windshield washer fluid. When the user opens the correct filling port and performs the filling operation, the interactive system 1 obtains the liquid level information from the vehicle's liquid level sensor and displays the filling progress in real time on the three-dimensional virtual model 400. When the liquid level reaches the set liquid level threshold, the user is reminded to stop the filling operation through visual and / or auditory signals sent by the smartphone and / or the interactive system 1.

[0094] In addition, if Figures 8A-8C As shown, in another embodiment of the present invention, the interactive system 1 can also demonstrate the vehicle repair steps to the user through a virtual game and guide the user to perform virtual repair operations through the virtual game to familiarize the user with the fault repair process.

[0095] In this embodiment, when the user sees the prompt information 410 of insufficient tire pressure of the vehicle through the virtual three-dimensional model 400, the user clicks on the floating window, bubble or text of the corresponding fault information, and the interactive system 1 enters the "low tire pressure" troubleshooting method selection interface. In this example, the user selects "Enter fault repair guide" to enter the secondary options, and selects the "fault repair demonstration" option in the secondary options. Then the interactive system 1 enters the fault repair demonstration. In this embodiment, the fault repair demonstration is performed in the form of a virtual game.

[0096] In one embodiment of the present invention, the interactive system 1 first demonstrates the fault repair process, and after the demonstration is completed, the user can complete the virtual repair operation process in the virtual game by touching the display 4. In other embodiments, the user can also enter the virtual game by touching the screen of the smart device connected to the vehicle communication.

[0097] During the virtual repair operation, the interactive system 1 receives the user's input for operating the game, and compares the user's virtual repair operation with the standardized repair operation stored in the interactive system 1, and provides feedback to the user on whether the operation is correct or not based on the comparison result. For example, when the user operates in the virtual game, each time a step in the virtual repair operation is completed correctly, the interactive system 1 prompts that the step is correct and proceeds to the next step by emitting a first color (such as green) on the display 4. When a step is operated incorrectly, the interactive system 1 prompts that the step is operated incorrectly with a second color (such as red), re-demonstrates the step, and requires the user to operate again until the operation is correct and proceeds to the next step.

[0098] In another embodiment, the interactive system 1 can also emit a beep and / or vibrate to remind the user that a certain step is not performed correctly. The user can complete the virtual repair operation through the virtual repair process in the above virtual game to help the user understand and become familiar with the repair operation for a specific fault of a specific part of the vehicle, so that the user can correctly eliminate the vehicle fault when performing actual repair operations on the vehicle later.

[0099] In this embodiment, Figures 8A-8C The operation screen in the virtual repair game may be shown. The user can touch the display 4 with a finger to perform the game operation according to the following steps: click the tire valve in the image 810, and an enlarged view 811 of the valve pops up. Then, click the connector of the compressor and slide the finger on the screen to fix the hose on the tire valve through the connector and rotate the finger clockwise on the screen to tighten the connector. After tightening the connector, an enlarged view 812 is displayed on the display 4. Then, in combination with Figure 8B, click the dial A on the compressor view 820 to open the enlarged view 821, then slide the finger to turn the dial A clockwise to the sealant position, click the switch button on the view 820 to pop up the enlarged view 822 of the switch button, then click the switch button B to start the repair and inflation process; then combine Figure 8C , inflate the tire to the pressure listed on the enlarged view 823; when the tire reaches the recommended tire pressure, click the enlarged view 822 of the switch button to turn off the switch button and disconnect the hose from the tire valve.

[0100] During the operation of the above virtual game, a buzzer and / or vibration prompt may be issued through the interactive system 1 or the smart device to remind the user that a certain step is not performed correctly. The user can complete the virtual repair operation through the virtual repair process in the above virtual game to help the user understand and become familiar with the repair operation for a specific fault of a specific vehicle component, so that the user can correctly eliminate the vehicle fault when performing actual repair operations on the vehicle later.

[0101] It is understandable that the manner in which the user interacts with the interactive system 1 described in the above embodiment is merely exemplary, and the resolution of other vehicle status or fault issues may also be carried out in the same manner as described above for interaction and guidance.

[0102] In addition, according to another aspect of the present application, a vehicle 100 is provided, such as Figure 1 As shown. The vehicle 100 includes the vehicle interaction system 1 described above. It should be understood that, in the absence of conflict, all the embodiments, features and advantages described above for the vehicle interaction system 1 according to the present application are also applicable to the vehicle 100 according to the present application. In other words, all the embodiments of the vehicle interaction system 1 described above and their variations can be directly transferred and applied to the vehicle 100 according to the present application and directly combined therewith. For the sake of brevity of the present disclosure, they will not be repeated here.

[0103] The present application proposes a vehicle interaction system 1 and a corresponding vehicle 100 and an interaction method. The system and the corresponding vehicle and method can integrate existing applications, functions and information in the interaction interface, provide a unified and orderly presentation effect for the interaction between the vehicle and the user, and significantly improve the interaction efficiency with the user and user satisfaction.

[0104] In this application, the use of disjunctive conjunctions is intended to include conjunctions. The use of definite or indefinite articles is not intended to indicate cardinality. In particular, reference to "the" object or "a" and "an" objects is intended to represent a possible one of a plurality of such objects. In addition, the conjunction "or" may be used to convey simultaneous features, rather than mutually exclusive solutions. In other words, the conjunction "or" should be understood to include "and / or". The term "include" is inclusive and has the same scope as "comprise".

[0105] The above embodiments are possible examples of implementation methods of the present application, and are only given to enable those skilled in the art to clearly understand the principles of the present application. Those skilled in the art should understand that the above discussion of any embodiment is only exemplary and is not intended to imply that the scope disclosed in the embodiments of the present application (including claims) is limited to these examples; under the overall concept of the present application, the technical features in the above embodiments or different embodiments can also be combined with each other, and produce many other changes in different aspects of the embodiments of the present application as described above, which are not provided in the specific implementation method for simplicity. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection required by the present application.

Claims

1. An interactive system for a vehicle, comprising a processor and a memory, wherein the memory stores processor-executable instructions, and when the instructions are executed by the processor, the following steps are implemented: displaying a virtual model of the vehicle; receiving fault information of the vehicle and displaying a first fault prompt on the virtual model based on the fault information; teaching the occupant to perform a first repair operation based on the first fault indication and at least in part through the virtual model; Acquiring real-time data of the first repair operation performed by the occupant through a sensor device; First feedback information is provided based on the first repair operation.

2. The interactive system according to claim 1, wherein: The sensing device includes at least one of a vehicle sensor and a smart device communicatively connected to the vehicle.

3. The interactive system according to claim 1 or 2, wherein: The step of teaching the occupant to perform the first repair operation also includes simultaneously providing augmented reality guidance.

4. The interactive system according to claim 3, wherein: The steps include: Call the camera device and obtain the real-time data captured by the camera; comparing the real-time data with the virtual model; and When it is determined that the camera device is aimed at the vehicle component corresponding to the first repair operation, the occupant is reminded to perform the first repair operation through a visual signal and / or an auditory signal.

5. The interactive system according to claim 4, wherein: The correct position of the vehicle component corresponding to the first repair operation is marked for the user by an indicator and / or a focus frame on the virtual model and / or on the display of the vehicle and / or the display of the camera device.

6. The interactive system according to claim 4, wherein: When the occupant performs the first repair operation, the first feedback information is displayed in real time on the virtual model according to real-time data of the first repair operation, and the first feedback prompt includes a correct step prompt and / or an incorrect step prompt.

7. The interactive system according to claim 4, wherein: When the occupant performs the first repair operation, the repair progress of the first fault prompt is updated in real time on the virtual model according to the real-time data.

8. The interactive system according to claim 7, wherein: The first fault prompt is a tire pressure shortage prompt, and the tire pressure shortage prompt is displayed at the position of the corresponding virtual wheel of the virtual model; The camera device is called according to the tire pressure underpressure prompt to obtain real-time data captured by the camera device, and the correct position of the wheel and the inflation port corresponding to the virtual wheel is marked for the user according to the real-time data and the virtual model, In which, during the inflation operation performed by the user, real-time data of the tire pressure is obtained and the repair progress is updated on the virtual model, and when the tire pressure reaches a first tire pressure threshold, the user is reminded to stop inflation through visual and / or auditory signals.

9. The interactive system according to claim 1, wherein the steps further include: providing a first virtual game associated with the components of the vehicle, which includes preset operation steps; Receive user input; Feedback is provided based on the degree of correspondence between the user input and the preset operation steps.

10. The interactive system according to claim 1, wherein: The virtual model of the vehicle is rendered using vehicle modeling data matched with the vehicle.

11. An interactive method, the method comprising: displaying a virtual model of the vehicle; receiving fault information of the vehicle and displaying a first fault prompt on the virtual model based on the fault information; teaching the occupant to perform a first repair operation based on the first fault indication and at least in part through the virtual model; Acquiring real-time data of a first repair operation of the occupant; First feedback information is provided based on the first repair operation.

12. The interactive method according to claim 11 further comprises acquiring real-time data of the first repair operation of the occupant through a sensing device, wherein the sensing device comprises at least one of a vehicle sensor and a smart device communicatively connected to the vehicle.

13. The interactive method according to claim 11 or 12, wherein: The step of teaching the occupant to perform the first repair operation also includes simultaneously providing augmented reality guidance.

14. The interactive method according to claim 13, further comprising: the steps comprising: Call the camera device and obtain the real-time data captured by the camera; comparing the real-time data with the virtual model information; as well as When it is determined that the camera device is aimed at the vehicle component corresponding to the first repair operation, the occupant is reminded to perform the first repair operation through a visual signal and / or an auditory signal.

15. The interactive method according to claim 14 further comprises: marking the correct position of the vehicle component corresponding to the first repair operation for the user through an indicator and / or a focus frame on the display of the virtual model and / or the vehicle and / or the display of the camera device.

16. The interactive method according to claim 14 further comprises: when the occupant performs the first repair operation, first feedback information is displayed in real time on the virtual model according to real-time data of the first repair operation, and the first feedback prompt includes a correct step prompt and / or an incorrect step prompt. 17 . The interactive method according to claim 14 , further comprising: when the occupant performs the first repair operation, updating the repair progress of the first fault indication on the virtual model in real time according to the real-time data.

18. The interactive method according to claim 16, wherein: The first fault prompt is a tire pressure shortage prompt, and the tire pressure shortage prompt is displayed on a corresponding virtual wheel of the virtual model; The camera device is called according to the low tire pressure prompt to obtain real-time data captured by the camera device, and the correct positions of the wheels and inflation ports corresponding to the virtual wheels are marked for the user according to the real-time data and the virtual model.

19. The interactive method according to claim 17, wherein: Acquiring real-time data of the tire pressure of the vehicle and updating it on the virtual model during the tire inflation operation by the user, and reminding the user to stop inflation through visual and / or auditory signals when the tire pressure reaches a first tire pressure threshold; and, when the tire pressure exceeds a second tire pressure threshold, displaying a second fault prompt at the virtual wheel position of the virtual model; Wherein, the second tire pressure threshold is greater than the first tire pressure threshold.

20. A vehicle, wherein: The vehicle includes an interactive system, the interactive system includes a processor and a memory, the memory stores processor-executable instructions, and the instructions are executed by the processor to implement the following steps: displaying a virtual model of the vehicle; receiving fault information of the vehicle and displaying a first fault prompt on the virtual model based on the fault information; teaching the occupant to perform a first repair operation based on the first fault indication and at least in part through the virtual model; Acquiring real-time data of the first repair operation of the occupant through a sensor device; First feedback information is provided based on the first repair operation.