Human-vehicle interaction method and apparatus

Abstract

Embodiments of the present disclosure disclose a method and device for human-vehicle interaction. The method comprises: in response to detecting a voice of a user, performing semantic recognition on the voice to identify a user intention; extracting a keyword from the voice; matching the user intention and the keyword with a scenario in a scenario library, wherein the scenario comprises a trigger condition, an executed action, and an implementation object of the action; in response to a failure in the scenario matching, constructing a scenario according to the user intention and the keyword, and displaying a dynamic window associated with the constructed scenario; and storing the constructed scenario in the scenario library. The embodiment can accurately analyze a complex instruction input by a user, identify a real intention and scenario requirement of the user, and automatically create a scenario according to the identified intention and scenario requirement. This method of creating a personalized vehicle use scenario through a simple voice instruction greatly improves the vehicle use experience of the user and makes the vehicle system more close to the actual requirement of the user.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of vehicle technology, specifically to human-vehicle interaction methods and devices. Background Technology

[0002] Currently, to achieve multi-tasking operations in an in-vehicle scenario, a scene engine is often used. The traditional approach of a scene engine involves building an in-vehicle atomic capability library in the backend, then orchestrating scenes based on this library. Administrators create scenes through a drag-and-drop interface in the backend, and different scene combinations form a scene library. Alternatively, an interface is provided to the user, requiring them to set the conditions for triggering a scene and the task chain to be executed. Going further, user habits are collected, and big data analysis is used in the backend to analyze these habits, then a scene is generated and pushed to the user.

[0003] While traditional graphical drag-and-drop methods are intuitive and easy to use, they may lack sufficient flexibility and scalability when faced with complex and ever-changing vehicle usage scenarios. As vehicle functions continue to increase, the challenge lies in how to quickly and efficiently add and modify scenarios.

[0004] While the drag-and-drop graphical interface simplifies scene arrangement, it may still present a learning curve for non-technical users. Users hope that the scene engine can provide a more convenient and intuitive operating method.

[0005] Scene recognition and judgment are rule-based, which may introduce errors, leading to inaccurate scene recommendations, scenes that don't meet user needs, or scenes that aren't recommended when the user needs them. This negatively impacts the user experience and reduces the usability of the scene engine.

[0006] Big data analysis based on user habits has high latency, and as the number of users increases, the backend computing pressure becomes high, resulting in less than ideal performance. Summary of the Invention

[0007] Embodiments of this disclosure present a human-vehicle interaction method and apparatus.

[0008] In a first aspect, embodiments of this disclosure provide a human-vehicle interaction method, comprising: in response to detecting a user's voice, performing semantic recognition on the voice to identify the user's intent; extracting keywords from the voice; matching the user's intent and the keywords with scenarios in a scenario library, wherein the scenario includes triggering conditions, an action to be performed, and an object to which the action is performed; in response to scenario matching failure, constructing a scenario based on the user's intent and the keywords, displaying a dynamic window associated with the constructed scenario; and storing the constructed scenario in the scenario library.

[0009] In some embodiments, the method further includes: analyzing whether there is a correlation between user driving habits and environmental information inside and outside the vehicle based on vehicle logs; in response to determining that there is a correlation, parsing the executed actions and the objects of the actions from the user driving habits, and preprocessing and analyzing the environmental information to obtain triggering conditions; matching the analyzed triggering conditions, executed actions, and objects of the actions with scenes in the scene library; in response to scene matching failure, constructing a scene based on the triggering conditions, executed actions, and objects of the actions, displaying a dynamic window associated with the constructed scene, and storing the constructed scene in the scene library.

[0010] In some embodiments, the method further includes: in response to successful scene matching, acquiring environmental information inside and outside the vehicle; preprocessing and analyzing the environmental information to obtain environmental conditions; matching the environmental conditions with triggering conditions in the successfully matched scene; and executing corresponding actions in response to successful triggering condition matching.

[0011] In some embodiments, the method further includes: in response to receiving negative feedback information from a user, performing data analysis on the vehicle log based on the negative feedback information to analyze the difference between the user's intention and the actual actions performed by the vehicle; and updating the actions in the scenario based on the difference.

[0012] In some embodiments, the method further includes: outputting prompt information for constructing a scene; and constructing a scene based on the feedback information received from the user.

[0013] In some embodiments, the method further includes: outputting prompt information for the action before performing the corresponding action; and updating the constructed scene based on the negative feedback information received from the user.

[0014] In some embodiments, the environmental information includes at least one of the following: in-vehicle temperature, outside vehicle temperature, in-vehicle facial image, music track played by the player, in-vehicle sound, and traffic information.

[0015] In some embodiments, displaying a dynamic window associated with the scene includes: displaying controls and related parameters corresponding to the objects performing actions in the constructed scene in the dynamic window; and editing the controls and related parameters in the dynamic window in response to receiving user feedback.

[0016] Secondly, embodiments of this disclosure provide a human-vehicle interaction device, comprising: a recognition unit configured to perform semantic recognition on the voice of a user in response to detection of the user's voice, and identify the user's intent; an extraction unit configured to extract keywords from the voice; a matching unit configured to match the user's intent and the keywords with scenarios in a scenario library, wherein the scenario includes triggering conditions, an action to be performed, and an object to which the action is performed; and a construction unit configured to construct a scenario based on the user's intent and the keywords in response to scenario matching failure, display a dynamic window associated with the constructed scenario, and store the constructed scenario in the scenario library.

[0017] In some embodiments, the construction unit is further configured to: analyze whether there is a correlation between user driving habits and environmental information inside and outside the vehicle based on vehicle logs; in response to determining that there is a correlation, parse the executed actions and the objects of the actions from the user driving habits, and preprocess and analyze the environmental information to obtain triggering conditions; match the analyzed triggering conditions, executed actions, and objects of the actions with scenes in the scene library; in response to scene matching failure, construct a scene based on the triggering conditions, executed actions, and objects of the actions, display a dynamic window associated with the constructed scene, and store the constructed scene in the scene library.

[0018] In some embodiments, the device further includes a triggering unit configured to: acquire environmental information inside and outside the vehicle in response to successful scene matching; preprocess and analyze the environmental information to obtain environmental conditions; match the environmental conditions with triggering conditions in the successfully matched scene; and execute a corresponding action in response to successful triggering condition matching.

[0019] In some embodiments, the apparatus further includes an update unit configured to: in response to receiving negative feedback information from a user, perform data analysis on the vehicle log based on the negative feedback information, analyze the difference between the user's intention and the actual actions performed by the vehicle; and update the actions in the scenario based on the difference.

[0020] In some embodiments, the building unit is further configured to: output prompt information for building the scenario; and build the scenario based on the feedback information received from the user.

[0021] In some embodiments, the building unit is further configured to: output prompt information for the action before performing the corresponding action; and update the building scenario according to the negative feedback information received from the user.

[0022] In some embodiments, the environmental information includes at least one of the following: in-vehicle temperature, outside vehicle temperature, in-vehicle facial image, music track played by the player, in-vehicle sound, and traffic information.

[0023] In some embodiments, the building unit is further configured to: display controls and related parameters corresponding to the objects implementing actions in the built scene in a dynamic window; and edit the controls and related parameters in the dynamic window in response to receiving user feedback.

[0024] Thirdly, embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more computer programs stored thereon, wherein when the one or more computer programs are executed by the one or more processors, the one or more processors perform the method as described in any one of the first or second aspects.

[0025] Fourthly, embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method as described in any one of the first or second aspects.

[0026] Fifthly, embodiments of this disclosure provide a computer program product including a computer program that, when executed by a processor, implements the method as described in any one of the first or second aspects.

[0027] The human-vehicle interaction method and apparatus provided in the embodiments of this disclosure use Natural Language Processing (NLP) technology to parse user input commands and identify user intent and scenario requirements. The commands are processed through word segmentation, part-of-speech tagging, and semantic analysis to extract key information (such as time, action, and object). If the user is creating a new scenario, a scenario definition is constructed based on the parsed user intent and key information. This includes triggering conditions, execution actions, and target objects. The scenario definition is stored in a scenario library (Agent) for subsequent invocation and execution. If the user is triggering an existing scenario, a matching scenario definition is searched in the scenario library (Agent) based on the user input command or the current environmental state.

[0028] Real-time environmental data inside and outside the vehicle is collected using onboard sensors (such as temperature sensors, cameras, and microphones). The collected data is preprocessed and analyzed to extract useful information (such as temperature values ​​and passenger status). Based on the triggering conditions defined in the scenario and combined with the current environmental perception results, it is determined whether the conditions for scenario execution are met.

[0029] If the conditions are met, the scenario execution agent is invoked. Based on the execution instructions in the scenario definition, the agent calls the APIs of the vehicle's internal systems (such as audio, air conditioning, seat controls, etc.) to execute the specific scenario task. During execution, further information exchange with the environmental perception agent or user interaction agent may be required.

[0030] After the scenario execution is complete, the user is notified of the execution result or feedback through the user interaction agent. User feedback is collected for subsequent scenario optimization and updates.

[0031] Analyze user feedback and scenario execution logs to identify problems and areas for improvement. Based on the data analysis results, optimize and adjust scenario definitions, triggering conditions, and execution actions. Continuously iterate and improve to enhance the intelligence level of scenarios and the user experience.

[0032] The disclosed solution can accurately parse complex user input commands, identify the user's true intentions and scenario requirements, and automatically create scenarios based on the identified intentions and scenario requirements. This ability to create personalized driving scenarios through simple voice commands greatly enhances the user's driving experience and makes the in-vehicle system more closely aligned with the user's actual needs.

[0033] The disclosed solution adopts a modular design, with each agent responsible for different functional modules, such as requirement analysis, scenario construction, environment awareness, condition judgment, and scenario execution, thereby improving the system's scalability and maintainability.

[0034] The disclosed solution can automatically trigger and execute corresponding scenarios based on user-defined conditions or real-time environmental perception results. This intelligent triggering mechanism reduces the user's operational burden and improves the convenience of vehicle use.

[0035] The disclosed solution can automatically generate and optimize driving scenarios based on users' driving habits and real-time environmental data. This adaptability enables the in-vehicle system to continuously evolve with changes in user needs and the external environment, providing services that are more tailored to user requirements.

[0036] The disclosed solution can integrate information from different sensors and data sources for comprehensive analysis and decision-making. This multi-source data fusion capability improves the accuracy and reliability of the system, ensuring the accuracy and effectiveness of scenario execution.

[0037] The disclosed solution can collect user feedback and scenario execution logs for data analysis and learning. Through continuous learning and optimization, the system can automatically adjust parameters such as scenario definitions, triggering conditions, and execution actions, thereby improving its intelligence level and user experience.

[0038] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0039] Other features, objects, and advantages of this disclosure will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0040] Figure 1 This is an exemplary system architecture diagram to which one embodiment of this disclosure can be applied;

[0041] Figure 2 This is a flowchart of an embodiment of the human-vehicle interaction method according to the present disclosure;

[0042] Figure 3 This is a schematic diagram of an application scenario of the human-vehicle interaction method disclosed herein;

[0043] Figure 4 This is a flowchart of yet another embodiment of the human-vehicle interaction method according to the present disclosure;

[0044] Figure 5 This is a schematic diagram of the structure of one embodiment of the human-vehicle interaction device according to the present disclosure;

[0045] Figure 6 This is a schematic diagram of the structure of a computer system suitable for implementing embodiments of the present disclosure. Detailed Implementation

[0046] The present disclosure will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and not intended to limit it. Furthermore, it should be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings.

[0047] It should be noted that, unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0048] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0049] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "over," and "on top" of the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0050] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0051] Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described herein, as well as the features of those embodiments or examples, without contradiction. This disclosure will now be described in detail with reference to the accompanying drawings and embodiments.

[0052] Figure 1 An exemplary system architecture is shown that can be applied to embodiments of the human-vehicle interaction method or human-vehicle interaction device disclosed herein.

[0053] like Figure 1As shown, the system architecture may include a dashcam 110, an in-vehicle camera 120, an in-vehicle display screen 130, a touchpad 140, a cloud server 150, a microphone, a speaker, and a radar unit (not shown in the figures). The system architecture may also include an AR HUD (not shown in the figures) and controllers (not shown in the figures) that establish communication connections with the in-vehicle display screen and the AR HUD, respectively. The radar unit is used to detect objects around the vehicle (vehicles, pedestrians, green belts, etc.) and measure the distance between the vehicle and the objects. Figure 1 The touchpad 140 shown is just an example of one mounted on the steering wheel; it can also be mounted on other vehicle components such as the dashboard and armrest. The windshield can be used as a projection screen for the AR HUD to display augmented reality information.

[0054] The dashcam 110 is used to record video images and sound of the entire driving process of a car.

[0055] The vehicle-mounted camera 120 can be a roof-mounted panoramic camera or cameras mounted on each side of the vehicle body. In some embodiments, the position and angle of the vehicle-mounted camera 120 can be adjusted as needed, and can be adjusted via voice commands, button commands, touch commands, etc. For example, a user can send a voice command such as "adjust the angle of the front camera upwards by 10 degrees" or "adjust the position of the front camera downwards by 1 centimeter." In other embodiments, such as a roof-mounted panoramic camera, it can acquire panoramic images around the vehicle body, and can crop images within a specific angle range from the panoramic images for image display or corner recognition according to instructions.

[0056] The touchpad 140 can be mounted on vehicle components such as the dashboard or steering wheel, allowing users to input touch commands to adjust the position of calibration points. For example, the touchpad consists of multiple piezoelectric vibrators, which can be mounted on the back of the touchpad. When pressure is applied to the surface of the touchpad, elastic waves are generated. These elastic waves are then transmitted to different piezoelectric vibrators, where corresponding elastic waveforms are picked up. These elastic waveforms have essentially the same shape, differing only in their arrival time and amplitude.

[0057] In some implementations, the touch position can be identified based on the TOF (Time of Flight) principle to obtain the touch command received by the touchpad 140: receiving voltage signals collected by multiple piezoelectric vibrators; determining a characteristic time point corresponding to each voltage signal based on at least one voltage point with similarity among the multiple voltage signals; the characteristic time point being determined based on at least one time point corresponding to the at least one voltage point; determining at least three characteristic time point pairs among the multiple characteristic time points; and determining the touch position information based on the relative time difference corresponding to the at least three characteristic time point pairs and the position information of a pair of piezoelectric vibrators corresponding to each relative time difference.

[0058] In other embodiments, the touch position can be identified and the touch command received by the touchpad 140 can be obtained by geometric calculation based on the relationship between the reciprocal of the detected voltage value and the distance: by receiving electrical signals collected by multiple piezoelectric vibrators respectively, and determining a first electrical signal point corresponding to each of the multiple electrical signals based on at least one electrical signal point with similarity among the multiple electrical signals, the first electrical signal point obtained can characterize the signal value of the touch point at the first time point collected by the piezoelectric vibrator; at the same time, by determining the proportional relationship between multiple first touch distances based on the first electrical signal point corresponding to each of the multiple electrical signals, the first touch distance is the distance between the piezoelectric vibrator and the touch point at the first time point. Based on the position information of multiple piezoelectric vibrators, the position information of the touch point at the first time point is determined. This takes into account the principle that the farther away from the touch point, the greater the attenuation of the mechanical elastic wave, the smaller the pressure sensing of the mechanical elastic wave on the piezoelectric vibrator, and the smaller the corresponding electrical signal output. By combining the position information of each piezoelectric vibrator, the touch point can be located through geometric relationships. At the same time, since the piezoelectric vibrator can be perfectly integrated with the surface material of the vehicle and has the characteristics of sun exposure resistance, it has stable performance when facing the complex usage scenarios of the vehicle. By determining the proportional relationship between the first touch distance through the first electrical signal point, and then combining the position information of the piezoelectric vibrator, the position information of the touch point can be accurately determined.

[0059] The vehicle-mounted display screen 130 can be various types of displays, such as the display screen of a DVR (Digital Video Recorder), or a central control screen, instrument panel screen, or passenger-side screen. It can also be an electronic device display screen that establishes a communication connection with the vehicle. The vehicle-mounted camera 120 and the vehicle-mounted display screen 130 can be connected via wired or wireless communication. For example, images captured by the vehicle-mounted camera 120 can be transmitted to the vehicle-mounted display screen 130 for display via WiFi, Bluetooth, or satellite imagery technology.

[0060] In some embodiments, the vehicle display 130 may be a touch screen for receiving instructions to adjust the displayed image, such as zooming the displayed image by swiping with a finger.

[0061] AR HUD is configured to project content from in-vehicle displays.

[0062] The controller is configured to receive signals from the user via an in-vehicle display, touchpad, microphone, or buttons. It can perform speech recognition on the received voice to obtain text information. It can also perform semantic understanding on the text information to understand the user's intent.

[0063] Cloud servers can provide map and navigation data.

[0064] The vehicle camera 120 can be a 360-degree panoramic camera. It is connected to a processor at the vehicle's infotainment port, which can read and process vehicle data. Radar sensors connected to the processor are installed at both ends of the front and rear bumpers on the vehicle, and the turn signal switch wires are connected to the processor.

[0065] The vehicle-mounted camera 120 collects image data from around the vehicle, creating a 360-degree panoramic overhead view of the vehicle's surroundings. The processor reads this image data. The processor reads data from the vehicle's infotainment system via a chip, then processes the vehicle's electronic power steering data along with the vehicle's track width and wheelbase data to obtain vehicle trajectory prediction data. This trajectory prediction data is then merged with the 360-degree panoramic overhead view to produce a panoramic overhead image with predicted driving trajectory.

[0066] The vehicle camera 120 may also include a camera located inside the vehicle for capturing facial images of passengers inside the vehicle and identifying the number of passengers and their emotional state through the images.

[0067] It should be noted that the human-vehicle interaction method provided in the embodiments of this disclosure is generally executed by a controller.

[0068] Continue to refer to Figure 2 The diagram illustrates a flow 200 of an embodiment of a human-vehicle interaction method according to the present disclosure. This human-vehicle interaction method includes the following steps:

[0069] Step 201: In response to detecting the user's voice, perform semantic recognition on the voice to identify the user's intent.

[0070] In this embodiment, the entity executing the human-vehicle interaction method (e.g., a controller) can acquire the user's voice from an in-vehicle microphone via a wired or wireless connection. Speech recognition is performed using a large language model to output text information. Then, semantic recognition is performed on the text information using the large language model to identify the user's intent.

[0071] For example, a user might say to the car's infotainment system, "When I get in the car before 8:30 AM and after 5:30 PM each day, play the daily recommended music, set the air conditioning to a comfortable temperature, and turn on the air freshener." The large language model installed on the infotainment system can recognize the user's intent and create scenarios according to the user's instructions. The system can then control the in-car devices to perform corresponding actions at scheduled times as requested by the user.

[0072] Mr. Zhang often travels with his family. To improve his family's comfort, he told the car's infotainment system, "When there is someone in the passenger seat, automatically turn on the seat massage and raise the seat temperature." The large language model installed on the infotainment system recognizes the user's intent, creates scenarios according to the user's instructions, and when the scenario trigger conditions are met, it controls the in-car equipment to perform the corresponding actions at regular intervals.

[0073] The user tells the car's infotainment system, "When it's not raining, get in the car and open all the windows a crack, then close them after 3 minutes." The large language model installed on the infotainment system recognizes the user's intent, creates a scenario according to the user's instructions, and, by acquiring information about the external environment, detects when the scenario trigger conditions are met, controls the in-car equipment to perform the corresponding actions at regular intervals.

[0074] The examples above all demonstrate how the in-vehicle infotainment system doesn't explicitly tell the user they want to create a scenario. Instead, the system uses a large language model to understand the user's intent. Users can also input explicit commands to create scenarios via voice or touchscreen. For example, a user could say to the system, "Create a 'ventilation' scenario: When it's not raining, get in the car and open the sunroof."

[0075] Step 202: Extract keywords from the speech.

[0076] In this embodiment, speech is recognized and converted into text information. Then, a keyword extraction model is used for word segmentation, part-of-speech tagging, and semantic analysis to extract key information (such as time, action, and object). For example, "When I get in the car before 8:30 am and after 5:30 pm every day, I play the daily recommended music, turn the air conditioning to a comfortable temperature, and turn on the aromatherapy diffuser." The key information extracted is: time: getting in the car before 8:30 am and after 5:30 pm every day; actions: playing music, turning on the air conditioning, and turning on the aromatherapy diffuser; objects of the actions: the daily recommended music and the comfortable temperature.

[0077] Step 203: Match user intent and keywords with scenarios in the scenario library.

[0078] In this embodiment, a scene library pre-stores some pre-defined scenes. Each scene can also be named according to the user's intent. The scene includes the trigger condition, the action to be performed, and the object to which the action is performed. For example, scene name (i.e., user intent): ventilation, trigger condition: when getting into the car every morning, action to be performed: open the car window, object to which the action is performed: all car windows.

[0079] The user intent identified in step 201 and the keywords extracted in step 202 can be matched with scenarios in the scenario library. The identified user intent can be matched with the scenario name in the scenario library, and the keywords can be matched with the triggering condition, the action performed, and the object of the action. This matching can be not only string matching but also semantic matching, such as synonyms or near-synonyms. For example, if the user intent is "ventilation" and the scenario name in the scenario library is "air exchange," the semantics are the same, and a successful match can be made. If the identified keyword is "darkness" and the triggering condition in the scenario library is "night," a successful match can be made. If the identified keyword is "listen to music," and the action performed in the scenario library is "play music," a successful match can be made. If the identified keyword is "front window," and the object of the action in the scenario library is "the driver's and passenger's windows," a successful match can be made.

[0080] If the user's intent and keywords match the scene name, triggering conditions, actions to be performed, and objects to be performed in the scene library, then the match is considered successful and there is no need to create a new scene.

[0081] Step 204: In response to scene matching failure, construct a scene based on user intent and keywords, display a dynamic window associated with the constructed scene, and store the constructed scene in the scene library.

[0082] In this embodiment, if scene matching fails, it indicates that the user wants to create a new scene. A scene name can be created based on the user's intent, and triggering conditions, actions, and objects implementing those actions can be created based on keywords. A dynamic window that can receive user feedback can also be displayed. This dynamic window is related to the created scene; for example, it may contain controls such as an air conditioner control lever or a music player. The dynamic window differs for different scenes and is customized. It can also receive user feedback.

[0083] Once created, the scenarios can be stored in the scenario library (Agent) for later invocation and execution.

[0084] The method provided in the above embodiments of this disclosure can accurately parse complex commands input by users, identify the user's true intentions and scenario requirements, and automatically create scenarios based on the identified intentions and scenario requirements. This method of creating personalized car usage scenarios through simple voice commands greatly enhances the user's car usage experience and makes the in-vehicle system more closely aligned with the user's actual needs.

[0085] The system adopts a modular design, with each agent responsible for different functional modules, such as requirement analysis, scenario construction, environment awareness, condition judgment, and scenario execution, which improves the system's scalability and maintainability.

[0086] The system can automatically trigger and execute corresponding scenarios based on user-defined conditions or real-time environmental perception results. This intelligent triggering mechanism reduces the user's operational burden and improves the convenience of vehicle use.

[0087] The system can automatically generate and optimize driving scenarios based on users' driving habits and real-time environmental data. This adaptability allows the in-vehicle system to continuously evolve as user needs and the external environment change, providing services that are more tailored to user requirements.

[0088] The system can integrate information from different sensors and data sources for comprehensive analysis and decision-making. This multi-source data fusion capability improves the system's accuracy and reliability, ensuring the accuracy and effectiveness of scenario execution.

[0089] The system can collect user feedback and scenario execution logs for data analysis and learning. Through continuous learning and optimization, the system can automatically adjust parameters such as scenario definitions, triggering conditions, and execution actions, thereby improving its intelligence level and user experience.

[0090] In some optional implementations of this embodiment, the method further includes: analyzing whether there is a correlation between user driving habits and environmental information inside and outside the vehicle based on vehicle logs; in response to determining that a correlation exists, parsing the executed actions and the objects of the actions from the user driving habits, and preprocessing and analyzing the environmental information to obtain triggering conditions; matching the analyzed triggering conditions, executed actions, and objects of the actions with scenes in the scene library; in response to scene matching failure, constructing a scene based on the triggering conditions, executed actions, and objects of the actions, displaying a dynamic window associated with the constructed scene, and storing the constructed scene in the scene library.

[0091] The system can analyze vehicle logs to identify correlations between user driving habits and environmental information inside and outside the vehicle. For example, based on the logs, user habits might show that when a user first opens the car door in the morning and the outside temperature is below 15 degrees Celsius, they will likely turn on the air conditioning, seat heaters, steering wheel heater, play the morning news, and navigate to the office. Therefore, a "winter commute" scenario can be automatically generated based on these habits, automatically activating the air conditioning, seat heaters, steering wheel heater, playing the morning news, and navigating to the office. After the scenario is generated, the user is asked if they want to save it. If the user agrees, subsequent actions will be automatically triggered according to this scenario definition.

[0092] For example, by analyzing vehicle logs to understand user habits, it can detect when the user lowers the volume, turns on the seat ventilation, and sets the air conditioning to a comfortable temperature, indicating that the front passenger or rear passenger may be asleep. Therefore, a "sleep mode" scenario can be automatically generated based on these habits. When the front passenger or rear passenger is detected to be asleep, the system will automatically lower the volume, turn on the seat ventilation, and set the air conditioning to a comfortable temperature. After the scenario is generated, the user is asked if they want to save it. If the user agrees, the scenario will be automatically triggered and activated subsequently.

[0093] For example, based on vehicle log analysis of user habits, if a crying child in the back seat is detected, the user might choose to play soothing music and activate the breathing-like vibrations on the smart seat surface to calm the child. Therefore, a "soothing" scenario can be automatically generated based on user habits. When a crying child in the back seat is detected, soothing music and the breathing-like vibrations on the smart seat surface will automatically play to calm the child. After the scenario is generated, the user is asked if they want to save it. If the user agrees, the scenario will be automatically triggered and activated according to this definition in subsequent use.

[0094] For example, based on vehicle log analysis of user driving habits, a "parking" scenario can be created: when a car enters a parking lot, the system can automatically detect the parking area, find available parking spaces, and manage the entire payment process. After the scenario is generated, the user is asked if they want to save it. If the user agrees, the scenario will be automatically triggered and activated in subsequent use.

[0095] In some optional implementations of this embodiment, the method further includes: in response to successful scene matching, acquiring environmental information inside and outside the vehicle; preprocessing and analyzing the environmental information to obtain environmental conditions; matching the environmental conditions with triggering conditions in the successfully matched scene; and executing corresponding actions in response to successful triggering condition matching.

[0096] Environmental information can include both in-vehicle and external environmental information. In-vehicle environmental information may include the number of passengers, passenger mood, in-vehicle music, temperature, and humidity. External environmental information may include weather, traffic information, and nearby points of interest (POIs). For example, it can acquire real-time information such as the number of people in the vehicle, their status, the music currently playing, and POIs passed by during the vehicle's journey. By accurately determining the number of people in the vehicle through in-vehicle sensors and cameras, and by connecting to the music playback device to obtain information such as music type, the system can identify surrounding POIs using the in-vehicle navigation system and map data, providing basic information for scene analysis.

[0097] Based on the triggering conditions in the scene definition and the current environmental perception results, determine whether the conditions for scene execution are met.

[0098] If the conditions are met, the scenario execution agent is invoked, and according to the execution instructions in the scenario definition, the APIs of the vehicle's internal systems (such as audio, air conditioning, seat control, etc.) are invoked to execute the specific scenario task.

[0099] During execution, further information exchange may be required with the environment-aware agent or the user interaction agent.

[0100] For example, when a child is detected crying, which meets the triggering conditions for a "soothing" scenario, the corresponding action is executed: automatically playing soothing music and causing the smart seat surface to breathe rhythmically to calm the child's emotions.

[0101] In some optional implementations of this embodiment, the method further includes: responding to receiving negative feedback from the user, performing data analysis on the vehicle logs based on the negative feedback, analyzing the difference between the user's intention and the actual actions performed by the vehicle; and updating the actions in the scenario based on the difference. The user can provide feedback via voice or through a scenario created by a dynamic window associated with the constructed scenario. If the user does not provide feedback within a predetermined time after the AI ​​assistant finishes speaking, it is considered that the user has implicitly agreed, i.e., positive feedback. The user can also explicitly express positive feedback; for example, if the AI ​​assistant says, "Want to listen to xx's song?", and the user says "Yes," it indicates positive feedback; if the user says "Change it," it indicates negative feedback.

[0102] Based on negative user feedback, previously created scenarios can be modified. For example, when the "soothing" scenario is triggered, soothing music and breathing rhythm on the smart seat surface can be played automatically. If the user says "do not activate the smart seat," the "soothing rhythm on the smart seat surface" action can be removed from the "soothing" scenario.

[0103] In some optional implementations of this embodiment, the method further includes: outputting prompts for constructing a scene; and constructing a scene based on the feedback received from the user. The prompts for constructing the scene can be output via voice or a dynamic window. The user can provide feedback via voice or a scene created by a dynamic window associated with the constructed scene. The user can supplement the feedback with information such as triggering conditions, the actions to be performed, and the objects to which the actions are performed, and the scene is created based on the information supplemented by the user.

[0104] In some optional implementations of this embodiment, the method further includes: outputting prompt information for the action before executing the corresponding action; and updating the constructed scene based on the negative feedback information received from the user. After detecting that the trigger condition is met, prompt information for the action can be output before executing the corresponding action in the scene. For example, the AI ​​assistant can say, "The child is crying, entering the soothing scene; soothing music and breathing rhythm on the smart seat surface will be played soon." If the user says, "Do not activate the smart seat," then the "smart seat surface breathing rhythm" action can be removed from the actions executed in the "soothing" scene.

[0105] In some optional implementations of this embodiment, the environmental information includes at least one of the following: in-vehicle temperature, outside vehicle temperature, in-vehicle facial image, music track played by the player, in-vehicle sound, and traffic information.

[0106] A pre-trained facial recognition model can be used to identify a user's emotional state and the number of users in a vehicle by analyzing facial images. This model, a neural network, first segments faces in an image. The number of bounding boxes determines the number of faces in the image, i.e., the number of people in the vehicle. Then, for each segmented face, facial expressions are identified. The model detects key facial features such as eyes, eyebrows, and mouth. The position of these key features, such as the distance between eyebrows and the direction of the corners of the mouth, can be used to determine facial expressions. Facial expressions can then be analyzed to discern emotions, such as happiness or annoyance. Based on the user's emotional state and the number of users, the current scenario can be determined, allowing an AI assistant to generate appropriate guidance statements and provide human-computer interaction based on user preferences, thereby increasing user interest and satisfaction.

[0107] The emotions detected by facial recognition can also be combined with other environmental information to generate more accurate emotions. For example, the user's voice can be recognized and combined with the user's facial expressions to further determine the user's emotions.

[0108] For example, if the system detects repeated phrases like "How much longer until we can leave?" or "Why can't we set off yet?"; facial expressions such as furrowed brows and downcast eyes; and actions like repeatedly looking out the window or getting on and off vehicles, the AI ​​assistant will say: "This traffic jam is really annoying. It looks like it'll be a while before it clears. How about we play a game together?"

[0109] Vehicles can obtain real-time location information, which, combined with maps, allows users to find nearby Points of Interest (POIs) that might interest them, such as stadiums hosting concerts or newly opened shopping malls. Users can search the official websites, public accounts, or review apps of the POIs to obtain information on events held at the corresponding venues. Content that users might be interested in can be filtered and read aloud by an AI assistant. User interests can be determined through their historical behavior, such as frequently visited or searched locations. A large language model can also answer various user questions, such as the concert schedule of a certain celebrity. User interests can be determined through their query records. A pre-trained interest recognition model can be used, inputting user behavior information to identify the user's interest categories. For example, liking a certain celebrity or liking hot pot.

[0110] The system collects in-vehicle sounds using a microphone; it then identifies these sounds to determine the user's emotional state.

[0111] The sounds inside the car can include the user's voice and music. The sound collected by the microphone can be input into a pre-trained speech recognition model to identify the text information. This text information is then input into a pre-trained semantic understanding model to understand the user's intent. For example, if a user says "Why aren't we there yet?", after text recognition and language understanding, the model can interpret the user's intent as complaining about the car's slow speed, being late, and feeling anxious.

[0112] It can also perform voiceprint recognition to identify the user's identity, thereby determining the number of people making the voice and performing semantic understanding of the dialogue, rather than mistakenly believing it to be the voice of the same person.

[0113] In addition, it can identify timbre information such as sound intensity and frequency, thereby more accurately judging the user's emotions. For example, if a user shouts hysterically, it indicates an emotional breakdown.

[0114] The user's emotional state affects the prompting statements generated by the large language model, which then uses these prompting statements to soothe the user's emotions.

[0115] The system can capture in-car audio via microphone to determine whether it's user voice or music. Music can originate from two sources: the car's built-in player or other mobile devices, such as the user's phone or tablet. Since the car's infotainment system (the controller mentioned earlier) communicates with the player, it can read the attribute information of the music playing. Based on this information, it can then query artist information and related music culture. For example, if artist A's song is playing, it can query artist A's recent concert information and upcoming movie releases. If folk music from country X is playing, it can introduce the music culture of country X. The system can also engage in multi-round dialogues with the user, encouraging them to learn more about related music culture.

[0116] The controller can also collect music played by other players through the microphone, perform voice recognition, obtain lyrics, and then search for song titles, singers, and other attribute information based on the lyrics. It can then query singer updates and related music culture knowledge based on the attribute information.

[0117] In some optional implementations of this embodiment, displaying a dynamic window associated with the scene includes: displaying controls and related parameters corresponding to the objects performing actions in the constructed scene in the dynamic window; and editing the controls and related parameters in the dynamic window in response to receiving user feedback.

[0118] For example, when a user opens the car door for the first time in the morning and the outside temperature is below 15 degrees Celsius, a "winter commute" scenario can be automatically generated based on the user's driving habits. This scenario automatically turns on the air conditioning, seat heating, steering wheel heating, and displays the morning news, and navigates to the company. The dynamic window can then display control icons for the air conditioning, seats, steering wheel, audio player, and navigation. It can also display the relevant parameters for each control, such as air conditioning set to 25 degrees Celsius and seat heating. Users can edit the dynamic window via voice or touch; for example, they can manually add or delete controls.

[0119] See also Figure 3 , Figure 3 This is a schematic diagram illustrating an application scenario of the human-vehicle interaction method according to this embodiment. Figure 3In application scenarios, users can create scenarios with a single sentence, such as "Play concert songs every time I pass a stadium where a concert is being held." The created scenario can be named "Concert," with the trigger condition being: passing a stadium where a concert is being held; the action being: playing music; and the object of the action being: concert songs. This is then saved to the scenario library. Subsequently, while the vehicle is in motion, the location determines that the vehicle is near a stadium, and concert information can be obtained. If a concert is currently being held at the stadium, the concert songs can be played on the in-car player. Music can be played from an officially released song list, or the sounds of fans gathered outside the vehicle can be collected to identify the currently playing song and play it synchronously inside the vehicle, giving users the feeling of being at a live concert.

[0120] Further reference Figure 4 This illustrates a flow 400 of another embodiment of the human-vehicle interaction method. Flow 400 of this human-vehicle interaction method includes the following steps:

[0121] Step 401: Analyze the vehicle logs to determine if there is a correlation between user driving habits and environmental information inside and outside the vehicle.

[0122] In this embodiment, the vehicle log records environmental information such as time and weather, as well as information on device operation. The frequency of identical environmental information can be statistically analyzed to identify user operating habits and determine the correlation between user habits and environmental information inside and outside the vehicle; for example, opening all windows each time the user gets in the car.

[0123] Step 402: In response to the determination of the existence of a correlation, the actions to be performed and the objects to be implemented are parsed from the user's driving habits, and the environmental information is preprocessed and analyzed to obtain the triggering conditions.

[0124] In this embodiment, after determining the environmental conditions that will trigger the vehicle's equipment actions, the environmental information can be preprocessed and analyzed to obtain the triggering conditions. For example, when it is detected that the front passenger or rear passenger is asleep, the volume is automatically lowered, the seat ventilation is turned on, and the air conditioning is set to a suitable temperature. Therefore, the triggering condition is: the front passenger or rear passenger is asleep, and the actions performed are: lowering the player volume, adjusting the air conditioning temperature, and turning on the seat ventilation. The target of these actions is the equipment located at the position of the sleeping user.

[0125] Step 403: Match the analyzed triggering conditions, the actions to be performed, and the objects to be performed with the scenes in the scene library.

[0126] In this embodiment, a scene library pre-stores some pre-defined scenes. Each scene can also be named according to the user's intent. The scene includes the trigger condition, the action to be performed, and the object to which the action is performed. For example, scene name (i.e., user intent): ventilation, trigger condition: when getting into the car every morning, action to be performed: open the car window, object to which the action is performed: all car windows.

[0127] The triggering conditions, actions, and objects of the actions analyzed in step 402 can be matched with scenes in the scene library. This matching can be not only string matching but also semantic matching, such as synonyms or near-synonyms. For example, if the analyzed triggering condition is "it gets dark" and the triggering condition in the scene library is "night," then the match will be successful. If the analyzed action is "listen to music," and the action executed in the scene library is "play music," then the match will be successful. If the analyzed action's object is "front window," and the action's object in the scene library is "driver's and passenger's windows," then the match will be successful.

[0128] If the analyzed triggering conditions, the actions to be performed, and the objects to which the actions are performed match every item in the scene library, then the match is considered successful and there is no need to create a new scene.

[0129] Step 404: In response to scene matching failure, construct a scene based on the triggering conditions, the executed action, and the object implementing the action, display a dynamic window associated with the constructed scene, and store the constructed scene in the scene library.

[0130] In this embodiment, if scene matching fails, it indicates that the user wants to create a new scene. A scene name can be created based on the user's intent, and triggering conditions, actions, and objects implementing those actions can be created based on keywords. A dynamic window that can receive user feedback can also be displayed. This dynamic window is related to the created scene; for example, it may contain controls such as an air conditioner control lever or a music player. The dynamic window differs for different scenes and is customized. It can also receive user feedback.

[0131] Once created, the scenarios can be stored in the scenario library (Agent) for later invocation and execution.

[0132] The method provided in the above embodiments of this disclosure can automatically trigger and execute corresponding scenarios based on user-defined conditions or real-time environmental perception results. This intelligent triggering mechanism reduces the user's operational burden and improves the convenience of vehicle use.

[0133] The system can automatically generate and optimize driving scenarios based on users' driving habits and real-time environmental data. This adaptability allows the in-vehicle system to continuously evolve as user needs and the external environment change, providing services that are more tailored to user requirements.

[0134] The system can integrate information from different sensors and data sources for comprehensive analysis and decision-making. This multi-source data fusion capability improves the system's accuracy and reliability, ensuring the accuracy and effectiveness of scenario execution.

[0135] The system can collect user feedback and scenario execution logs for data analysis and learning. Through continuous learning and optimization, the system can automatically adjust parameters such as scenario definitions, triggering conditions, and execution actions, thereby improving its intelligence level and user experience.

[0136] Further reference Figure 5 As an implementation of the methods shown in the above figures, this disclosure provides an embodiment of a human-vehicle interaction device, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0137] like Figure 5 As shown, the human-vehicle interaction device 500 of this embodiment includes: a recognition unit 501, an extraction unit 502, a matching unit 503, and a construction unit 504. The recognition unit 501 is configured to perform semantic recognition on the voice detected from a user, identifying the user's intent; the extraction unit 502 is configured to extract keywords from the voice; the matching unit 503 is configured to match the user's intent and the keywords with scenes in a scene library, wherein the scene includes triggering conditions, an action to be performed, and an object to which the action is performed; the construction unit 504 is configured to construct a scene based on the user's intent and the keywords in response to scene matching failure, display a dynamic window associated with the constructed scene, and store the constructed scene in the scene library.

[0138] In this embodiment, the specific processing of the recognition unit 501, extraction unit 502, matching unit 503, and construction unit 504 of the human-vehicle interaction device 500 can be referred to Figure 2 The corresponding steps are 201, 202, 203, and 204 in the embodiment.

[0139] In some optional implementations of this embodiment, the construction unit 504 is further configured to: analyze whether there is a correlation between user driving habits and environmental information inside and outside the vehicle based on vehicle logs; in response to determining that there is a correlation, parse the executed action and the object of the action from the user driving habits, and preprocess and analyze the environmental information to obtain the triggering condition; match the analyzed triggering condition, executed action, and object of the action with the scenes in the scene library; in response to scene matching failure, construct a scene based on the triggering condition, executed action, and object of the action, display a dynamic window associated with the constructed scene, and store the constructed scene in the scene library.

[0140] In some optional implementations of this embodiment, the device 500 further includes a triggering unit (not shown in the figures), configured to: acquire environmental information inside and outside the vehicle in response to successful scene matching; preprocess and analyze the environmental information to obtain environmental conditions; match the environmental conditions with triggering conditions in the successfully matched scene; and execute corresponding actions in response to successful triggering condition matching.

[0141] In some optional implementations of this embodiment, the device 500 further includes an update unit (not shown in the figures), configured to: in response to receiving negative feedback information from a user, perform data analysis on the vehicle log based on the negative feedback information, analyze the difference between the user's intention and the actual actions performed by the vehicle; and update the actions in the scenario based on the difference.

[0142] In some optional implementations of this embodiment, the construction unit 504 is further configured to: output prompt information for constructing the scene; and construct the scene based on the feedback information received from the user.

[0143] In some optional implementations of this embodiment, the construction unit 504 is further configured to: output prompt information for the action before performing the corresponding action; and update the construction scene according to the negative feedback information received from the user.

[0144] In some optional implementations of this embodiment, the environmental information includes at least one of the following: in-vehicle temperature, outside vehicle temperature, in-vehicle facial image, music track played by the player, in-vehicle sound, and traffic information.

[0145] In some optional implementations of this embodiment, the construction unit 504 is further configured to: display the controls and related parameters corresponding to the objects implementing the actions in the constructed scene in a dynamic window; and edit the controls and related parameters in the dynamic window in response to receiving user feedback.

[0146] It should be noted that the collection, gathering, updating, analysis, processing, use, transmission, and storage of user personal information involved in this disclosed technical solution all comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.

[0147] According to embodiments of this disclosure, this disclosure also provides an electronic device and a readable storage medium.

[0148] An electronic device includes: one or more processors; and a storage device having one or more computer programs stored thereon, which, when executed by the one or more processors, cause the one or more processors to implement the method described in process 200 or 400.

[0149] A computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method described in process 200 or 400.

[0150] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0151] like Figure 6 As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 602 or a computer program loaded from storage unit 608 into random access memory (RAM) 603. RAM 603 may also store various programs and data required for the operation of device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0152] Multiple components in device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of monitors, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0153] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as road planning methods. For example, in some embodiments, the road planning method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the road planning method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the road planning method by any other suitable means (e.g., by means of firmware).

[0154] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0155] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0156] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0157] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0158] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0159] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be servers in distributed systems or servers incorporating blockchain technology. Servers can also be cloud servers, or intelligent cloud computing servers or intelligent cloud hosts with artificial intelligence technology.

[0160] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0161] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A human-vehicle interaction method, comprising: In response to detecting a user's voice, semantic recognition is performed on the voice to identify the user's intent; Extract keywords from the speech; The user intent and the keywords are matched with scenarios in the scenario library, wherein the scenario includes triggering conditions, the action to be performed, and the object to which the action is performed; In response to a scene matching failure, a scene is constructed based on the user intent and the keywords, a dynamic window associated with the constructed scene is displayed, and the constructed scene is stored in the scene library.

2. The method according to claim 1, wherein, The method further includes: Analyze vehicle logs to determine if there is a correlation between user driving habits and environmental information inside and outside the vehicle; In response to the determination of a correlation, the actions to be performed and the objects to be implemented are parsed from the user's driving habits, and the environmental information is preprocessed and analyzed to obtain the triggering conditions; The analyzed triggering conditions, the actions to be performed, and the objects to which the actions are performed are matched with the scenes in the scene library; In response to a scene matching failure, a scene is constructed based on the triggering conditions, the executed action, and the object of the action. A dynamic window associated with the constructed scene is displayed, and the constructed scene is stored in the scene library.

3. The method according to claim 1, wherein, The method further includes: Upon successful scene matching, obtain environmental information inside and outside the vehicle; The environmental information is preprocessed and analyzed to obtain the environmental conditions; Match the environmental conditions with the triggering conditions in the successfully matched scenario; In response to a successful match of the triggering conditions, the corresponding action is executed.

4. The method according to claim 3, wherein, The method further includes: In response to receiving negative feedback from the user, the vehicle log is analyzed based on the negative feedback to analyze the difference between the user's intention and the actual actions performed by the vehicle. Update the actions in the scene based on the differences.

5. The method according to claim 1, wherein, The method further includes: Output prompts for the constructed scene; In response to receiving user feedback, a scenario is constructed based on the feedback.

6. The method according to claim 3, wherein, The method further includes: Before performing the corresponding action, output a prompt message for the action; In response to receiving negative feedback from the user, the constructed scene is updated based on the negative feedback.

7. The method according to claim 2 or 3, wherein, The environmental information includes at least one of the following: vehicle interior temperature, vehicle exterior temperature, in-vehicle facial image, music track played by the player, in-vehicle sound, and traffic information.

8. The method according to claim 1 or 2, wherein, The dynamic window displayed is associated with the scene, including: The dynamic window displays the controls and related parameters corresponding to the objects that perform the actions in the constructed scene; In response to user feedback, the controls and related parameters in the dynamic window are edited.

9. A human-vehicle interaction device, comprising: The recognition unit is configured to perform semantic recognition on the voice in response to detecting the user's voice, and to identify the user's intent; The extraction unit is configured to extract keywords from the speech; The matching unit is configured to match the user intent and the keywords with scenarios in a scenario library, wherein the scenario includes triggering conditions, an action to be performed, and the object to which the action is performed; The construction unit is configured to, in response to a scene matching failure, construct a scene based on the user intent and the keywords, display a dynamic window associated with the constructed scene, and store the constructed scene in the scene library.

10. An electronic device, comprising: One or more processors; Storage device, on which one or more computer programs are stored, When the one or more computer programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-8.

11. A computer-readable medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-8.

Images