In-vehicle routine generation using sound commands

Abstract

Various embodiments disclose a computer-implemented method comprising: receiving an auditory speech signal generated by a user; mapping a first speech portion in the auditory speech signal to a first condition, wherein the first condition corresponds to a value associated with operation of a first component of the vehicle; mapping a second speech portion in the auditory speech signal to a first action, wherein the first action corresponds to a first action performed by a second component of the vehicle; generating a routine including the first condition and the first action; comparing the routine with a set of stored rules; upon determining that the routine does not overlap with at least one stored rule, storing the routine as a first stored rule; after the first stored rule is stored, determining that the first condition has been satisfied; and causing the second component of the vehicle to perform the first action.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to co-pending Indian patent application titled “IN-VEHICLE GENERATION OF ROUTINES USING VOICE COMMANDS” filed on May 11, 2022 and having application number 202241027139. The subject matter of this related application is hereby incorporated herein by reference. Technical Field

[0003] Embodiments disclosed herein relate to digital assistants, and in particular, to in-vehicle routine generation using voice commands. Background Art

[0004] Various digital systems include digital assistants that assist users in performing tasks. For example, various vehicles include digital assistants that communicate with subsystems such as advanced driver assistance systems (ADAS), infotainment systems, navigation systems, etc. to assist users of the vehicle. In some vehicles, various subsystems may execute routines in which the vehicle performs specific actions based on specific conditions associated with the vehicle. For example, the ADAS may turn on the vehicle's headlights based on specific conditions such as determining the time of day or upon receiving sensor data indicating low visibility. Typically, a user of the vehicle may interact with the digital assistant to add other routines for the vehicle to perform. For example, a user may program an infotainment unit to play a specific audio source immediately after the vehicle is started.

[0005] One disadvantage of conventional digital assistants in vehicles is that such systems are not configured to allow or encourage non-expert users to generate routines for execution by the vehicle. In particular, such systems have complex interfaces that can only be navigated by experienced users who have a deep understanding of the conditions that the vehicle can detect and the actions that the vehicle can perform. As a result of this architecture, current digital systems cannot be extended for programming by users who do not have a deep understanding of the interface to program new vehicle routines. Due to this limitation, many digital assistants are not effectively utilized by vehicle users and result in the user manually performing certain tasks. In addition, because the user has not programmed the digital assistant to perform certain routines in response to detected conditions, the user may experience negative interactions with the vehicle. For example, in the absence of a programmed routine, the user may have to respond to detected precipitation by manually turning on one or more sets of windshield wipers and closing all vehicle windows.

[0006] As previously stated, there is a need in the art for more efficient techniques for programming personalized routines associated with the operation of a vehicle. Summary of the invention

[0007] Various embodiments disclose a computer-implemented method comprising: receiving a set of auditory speech signals generated by a user; mapping a first speech portion in the set of auditory speech signals to a first condition, wherein the first condition corresponds to at least a first value associated with the operation of a first component of a vehicle; mapping a second speech portion in the set of auditory speech signals to a first action, wherein the first action corresponds to at least a first action performed by a second component of the vehicle; generating a routine including the first condition and the first action; comparing the routine to a set of stored rules, wherein each stored rule in the set of stored rules includes at least one condition and at least one action; upon determining that the routine does not overlap with at least one stored rule in the set of stored rules, storing the routine as a first stored rule; upon storing the first stored rule, determining that the first condition has been satisfied; and causing the second component of the vehicle to perform the first action.

[0008] At least one technical advantage of the disclosed technology over the prior art is that, utilizing the disclosed technology, a user who is not proficient in programming customized vehicle routines can also effectively generate and approve customized vehicle routines that respond to various detected conditions, thereby greatly improving the in-vehicle experience of the user of the vehicle. In particular, by processing the user's voice to generate effective routines for storage and monitoring, the routine management application relieves the user of the vehicle from physical and mental strain in performing various manual tasks associated with the operation of the vehicle. In addition, by interpreting the user's emotions when dictating the personalized routines, the personalized routine management system provides a wider range of available conditions to monitor and operations to perform than menu-based user interfaces. These technical advantages provide one or more technical advances over prior art methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order that the manner in which the above-mentioned features of various embodiments can be understood in detail, the inventive concept briefly summarized above may be described in more detail by reference to various embodiments, some of which are shown in the accompanying drawings. It should be noted, however, that the drawings illustrate only typical embodiments of the inventive concept and are therefore not to be considered as limiting its scope in any way, and that other equally effective embodiments exist.

[0010] Figure 1 A personalized routine management system is shown according to one or more embodiments.

[0011] Figure 2 Shown according to various embodiments include Figure 1 A routine manages an application and an exemplary vehicle system of a hybrid interface engine.

[0012] Figure 3Shown according to various embodiments Figure 1 An exemplary personalized routine table included in the personalized routine management system of FIG.

[0013] Figure 4 Shown according to various embodiments Figure 1 The invention relates to a technology for creating stored rules and responding to triggers of a personalized routine management system.

[0014] Figure 5 Shown according to various embodiments Figure 1 Diagram of the interactions between the various components of a personalized routine management system.

[0015] Figure 6 A flow chart setting forth method steps for generating rules corresponding to a personalized routine specified by a user, according to various embodiments.

[0016] Figure 7 A flow chart illustrating method steps for responding to detection of one or more satisfied conditions of a rule by performing an action specified by the rule, according to various embodiments. DETAILED DESCRIPTION

[0017] In the following description, numerous specific details are set forth to provide a more thorough understanding of various embodiments. However, it will be apparent to one skilled in the art that the inventive concept may be practiced without one or more of these specific details.

[0018] Overview

[0019] Embodiments disclosed herein include a personalized routine management system, the personalized routine management system including a routine management application that generates and stores personalized routines for execution by the vehicle from speech portions provided by a user in the vehicle. The personalized routine management system also includes a hybrid reasoning engine that responds to the stored personalized routines by causing components of the vehicle to perform specified actions when specific conditions are met. A processing unit included in the personalized routine management system operates to receive one or more input auditory speech signals corresponding to a phrase portion spoken by a user requesting the generation of a personalized routine. The processing unit generates an intent from the phrase portion, wherein the intent includes a context portion detailing one or more conditions and an action portion detailing one or more actions to be performed when the context portion is met. The processing unit verifies the intent using a set of pre-existing conditions that the vehicle can monitor and a set of actions that the vehicle can perform in response to a trigger. The processing unit generates the personalized routine after verification and converts the personalized routine into a format interpretable by the reasoning engine in various embodiments ( For example ,rule).

[0020] The processing unit also operates to monitor a set of vehicle conditions and determine whether the set of vehicle conditions satisfies any personalized routines. In some embodiments, the processing unit operates an inference engine, such as a hybrid inference engine, that compares the vehicle conditions to a set of stored rules corresponding to the personalized routines. When the processing unit detects a change in vehicle conditions, the processing unit compares the current set of vehicle conditions to a set of stored personalized routines to determine whether the conditions specified in one of the personalized routines are satisfied. When the processing unit determines that the conditions of the personalized routine are satisfied, the processing unit identifies a set of actions specified in the personalized routine and causes the available components of the vehicle to perform the specified actions.

[0021] System Overview

[0022] Figure 1 A personalized routine management system 100 is shown according to one or more embodiments. As shown, the personalized routine management system 100 includes, but is not limited to, a computing device 110, a network 150, an external data store 152, a sensor 172, an input device 174, and an output device 176. The computing device 110 includes, but is not limited to, a memory 120, a processing unit 140, a network interface 142, and an input / output (I / O) device interface 144. The memory 120 includes, but is not limited to, a user interface 122, a hybrid inference engine (HIE) 124, a data store 126, and a routine management application 130. The routine management application 130 includes, but is not limited to, a sound recognition module 132, a routine creator 134, and a rule management module 136. The data store includes rules 128 ( For example , 128(1), 128(2), etc.).

[0023] For purposes of illustration, multiple instances of similar objects are represented by a reference number identifying the object and an additional number in parentheses identifying the instance when necessary. In addition, the personalized routine management system 100 includes multiple instances of an element, even if not shown. For example, the personalized routine management system 100 may include multiple sensors 172 ( For example , 172(1), 172(2), 172(3), etc.), input device 174 ( For example , 174(1), 174(2), 174(3), etc.) and / or output device 176 ( For example , 176(1), 176(2), 176(3), etc.), and still be within the scope of the disclosed embodiments.

[0024] In operation, computing device 110 executes routine management application 130 to generate and store one or more rules 128 corresponding to a personalized routine specified by a user using one or more speech parts 162. In some embodiments, routine management application 130 receives one or more speech parts 162 specifying conditional statements corresponding to a desired routine via user interface 122; alternatively, in some embodiments, routine management application 130 generates and stores one or more rules 128 corresponding to a personalized routine specified by a user using one or more speech parts 162. For example , request for condition, request for vehicle action, etc.) to receive one or more speech parts 162 respectively. After receiving the speech parts, the sound recognition module 132 processes the one or more speech parts 162 to generate the user's intention. The routine creator 134 verifies the content of the intent against a set of predefined conditions and / or a set of predefined actions, and generates the routine after determining that the conditions and actions included in the intent are valid. In some embodiments, the routine management application 130 compares the generated routine with the stored routines to determine whether this is an overlap; after determining that the generated routine overlaps with the existing routine, the routine management application 130 can avoid storing the generated routine or replace the conflicting routine. In various embodiments, the rule management module 136 can convert the routine into a different format, such as a specific type of rule 128 ( For example , C Language Integrated Production System (CLIPS) rules).

[0025] The hybrid inference engine (HIE) 124 detects the presence of a vehicle component ( For example , one or more input devices 174) monitor various vehicle conditions associated with the operation of the vehicle or applications operating within the vehicle ( For example , vehicle status, environmental conditions, application events, etc.). In various embodiments, the HIE 124 may determine a given rule 128 by comparing the current set of conditions to the rule 128 ( For example , whether the condition of rule 128(1) is satisfied, thereby responding to the detected change in the condition. After determining that the condition specified by rule 128(1) is satisfied, HIE 124 determines one or more actions specified by rule 128(1) and causes the corresponding vehicle component ( For example , one or more output devices 176) perform one or more actions specified by rule 128(1).

[0026] The computing device 110 may include a processing unit 140 and a memory 120. In various embodiments, the computing device 110 may be a device, such as a system on a chip (SoC), that includes one or more processing units 140. In various embodiments, the computing device 110 may be a mobile computing device that is wirelessly connected to other devices in the vehicle, such as a tablet computer, a mobile phone, a media player, etc. In some embodiments, the computing device 110 may be a head unit or part of a head unit included in a vehicle system. In some embodiments, the computing device 110 may be split among multiple physical devices in one or more locations. For example, one or more remote devices ( For example , cloud servers, remote services, etc.) may perform one or more aspects of the disclosed technology, such as speech analysis, intent determination, rule generation, deduplication, etc. Additionally or alternatively, the computing device 110 may be a detachable device installed in a portion of the vehicle as part of a separate console. In general, the computing device 110 may be configured to coordinate the overall operation of the personalized routine management system 100. The embodiments disclosed herein contemplate any technically feasible system configured to implement the functionality of the personalized routine management system 100 via the computing device 110. The functionality and techniques of the personalized routine management system 100 are also applicable to other types of vehicles, including consumer vehicles, commercial trucks, airplanes, helicopters, spacecraft, ships, submarines, etc.

[0027] Processing unit 140 may include one or more central processing units (CPUs), digital signal processing units (DSPs), microprocessors, application specific integrated circuits (ASICs), neural processing units (NPUs), graphics processing units (GPUs), field programmable gate arrays (FPGAs), and the like. Processing unit 140 typically includes a programmable processor that executes program instructions to manipulate input data and generate output. In some embodiments, processing unit 140 may include any number of processing cores, memory, and other modules for facilitating program execution. For example, processor unit 140 may receive input from a user via input device 174 and generate pixels for display on output device 176 ( For example In some embodiments, the processing unit 140 may be configured to execute the routine management application 130 so as to generate a routine based on the input device 174 ( For example , microphone) to create and store personalized routines. Additionally or alternatively, the processing unit 140 may be configured to execute the HIE 124 to monitor a set of vehicle conditions and cause one or more vehicle components to perform a set of actions ( For example , vehicle behavior, application events, etc.).

[0028] Memory 120 may include a memory module or a collection of memory modules. Memory 120 typically includes a memory chip, such as a random access memory (RAM) chip, that stores applications and data for processing by processing unit 140. In various embodiments, memory 120 may include non-volatile memory, such as an optical drive, a magnetic drive, a flash drive, or other storage device. In some embodiments, a separate data store (such as an external data store 152 connected via a network 150 ("cloud storage device")) may be connected to routine management application 130 and / or HIE 124. User interface 122, HIE 124, and / or routine management application 130 within memory 120 may be executed by processing unit 140 to implement the overall functionality of computing device 110, and thus coordinate the operation of personalized routine management system 100 as a whole.

[0029] In various embodiments, the routine management application 130 may implement one or more modules 132-136 to process one or more speech portions 162 via the input device 174 to determine whether the user 160 has provided content for a valid personalized routine for execution by the vehicle and store a version of the personalized routine ( For example , rule 128(1). When the routine management application 130 determines that the user 160 has provided valid conditions and actions for a valid personalized routine, the routine management application 130 stores the personalized routine as rule 128(1) for monitoring by the HIE 124.

[0030] The voice recognition module 132 performs various natural language processing (NLP) and natural language understanding (NLU) techniques, sentiment analysis, and / or speech analysis to recognize phrases spoken by the user 160. In some embodiments, the voice recognition module 132 receives the speech portion 162 from the user 160 via the user interface 122. Alternatively, in some embodiments, another component of the vehicle ( For example , infotainment system and entertainment subsystem, etc.) or another device ( For example , a connected digital assistant) can process the speech portion 162. Additionally or alternatively, the sound recognition module 132 can perform various speech-to-text techniques on the speech portion 162 to classify the phrase as part of the intent.

[0031] In various embodiments, the sound recognition module 132 may determine the semantic meaning of the speech portion 162 spoken by the user 160 in order to determine whether the provided speech portion 162 is part of a valid routine. Alternatively, in some embodiments, another component of the vehicle or another device may generate the intent and / or other parameters of the speech portion 162. In such cases, the sound recognition module 132 may determine the semantic meaning of the intent and may modify the intent to include the determined semantic meaning. In some embodiments, the sound recognition module 132 performs sentiment analysis to determine the intent of the phrase and / or the meaning of the phrase. In some embodiments, the sound recognition module 132 may employ various statistical methods, machine learning (ML) methods, state machines, and / or various other data structures in order to identify deviations of the phrases spoken by the user 160 from predefined conditions or predefined actions associated with the vehicle.

[0032] Routine creator 134 generates a routine based on the intent provided by voice recognition module 132. In various embodiments, routine creator 134 may compare the portion of the intent provided by user 160 and / or other parameters associated with speech portion 162 to one or more predefined conditions and / or one or more predefined actions to determine whether the user is attempting to create a valid personalized routine. For example, user 160 may provide a speech portion, i.e., "Create a routine: When the song 'Definition' by Black Star starts, turn up the bass." In such a case, routine creator 134 may receive an intent that specifies a context portion and an action portion phrase.

[0033] {IF current track = “Definition” AND current artist = ‘Black Star’}

[0034] {THEN cause equalizer to increase BASS by 2}

[0035] Here, the "IF" parsed version of the speech portion 162 corresponds to the context portion, and the "THEN" parsed version of the speech portion 162 corresponds to the action portion. The routine creator 134 can determine whether the HIE 124 is capable of monitoring the specified conditions by determining whether there are any predefined conditions monitoring the current track and artist being played by the vehicle entertainment system. The routine creator 134 can also determine whether the HIE 124 is capable of causing the entertainment system to change the equalizer settings in a manner specified by the user 160. In some embodiments, the sound recognition module 132 and / or the routine creator 134 can replace a specific value in the intent based on the determined semantic meaning. For example, the sound recognition module 132 can use the NLU process to determine that "turn up the bass" corresponds to a valid action, such as increasing the bass level from the current level or setting the bass level to a specific level.

[0036] In various embodiments, when the routine creator 134 verifies the intent, the routine creator 134 may generate a draft personalized routine and provide the draft routine to the user ( For example , dictating the content of the draft routine, displaying a summary of the draft routine, etc.) for approval. Additionally or alternatively, when routine creator 134 does not verify the intent, routine creator 134 may provide a notification indicating that user 160 did not provide the content of a valid personalized routine, and may suggest that user 160 try again or may enter guided mode.

[0037] When the routine management application 130 enters the guided mode, the routine management application 130 exchanges a conversation with the user 160 by providing a prompt for the user to provide a speech portion 162 that corresponds to a valid context portion or a valid action portion. In such a case, the routine management application 130 may list vehicle components associated with pre-existing conditions or actions. For example, the routine management application 130 may initiate the guided mode by providing a prompt that reads, "Please provide a condition to be checked. The vehicle currently monitors the heating and cooling system, power windows, and media players." The routine management application 130 may then receive the first speech portion 162 (1) and may compare the first speech portion to the list of pre-existing conditions to verify that the first speech portion 162 specifies a valid context portion before providing additional prompts. In such a case, the prompt may request a valid vehicle action, i.e., "Please provide an action for the vehicle to perform. The vehicle currently controls the door locks, equalizer, and power windows." The routine management application 130 may then receive the second speech portion 162(2) and may compare the second speech portion to the pre-existing action list to verify the second speech portion 162 as a specified valid action portion. After verifying the content of the individual speech portions 162(1), 162(2), the routine creator 134 may generate a draft personalized routine including a context portion and an action portion.

[0038] The rule management module 136 converts the draft personalized routine into rules that the HIE 124 can monitor. In various embodiments, the rule management module 136 can convert the context portion and the action portion of the personalized routine into rules 128 having a specific format. For example, a CLIPS rule can include a fact portion and an action portion. In various embodiments, the HIE 124 can monitor information about one or more services provided by the vehicle ( For example , vehicle status and conditions, monitored environmental conditions, application status and events, etc.) to determine whether a given fact of the rule is true ( For example , based on the information provided by the sensor 172, determines whether the fact that "there is precipitation on the vehicle" is currently true). When the HIE 124 determines that the fact of a given rule 128 is valid, the HIE 124 performs the action included in the rule.

[0039] In various embodiments, the rule management module 136 can communicate with the HIE 124 and / or other modules when generating the rules 128 from the personalization routine. For example, the rule management module 136 can refer to the use case interface and / or the service interface to determine the applicable identifier of the system and / or device associated with the rule 128. In some embodiments, the rule management module 136 can forward the personalization routine to a separate code generator that generates the rules 128 from the personalization routine. Additionally or alternatively, the rule management module 136 can store the rules 128 in the local data store 126.

[0040] The user interface 122 enables the user 160 to provide input regarding specific data, such as a voice portion 162, approval of a draft personalization routine, etc. In some embodiments, the user interface 122 may receive instructions for downloading remotely stored rules ( For example , stored in the external data store 152). Additionally or alternatively, the user interface 122 displays corresponding to pre-existing conditions, pre-existing actions, examples, use cases, and / or error notifications ( For example , invalid intent, duplicate or overlapping rules, etc.). In some embodiments, the user interface 122 can take any feasible form to provide the functionality described herein, such as one or more buttons, two-state toggle switches, sliders, dials, knobs, etc., or as a graphical user interface (GUI).

[0041] In various embodiments, the user interface 122 may be provided by any component of the personalized routine management system 100. In one embodiment, the user interface 122 may be provided by a separate computing device communicatively coupled to the computing device 110, such as by an application running on a user's mobile or wearable computing device. Additionally or alternatively, the infotainment system ( For example, included in the entertainment subsystem of the vehicle head unit) can provide the user interface 122 via a display and / or a voice digital assistant. In another example, the user interface 122 can receive a verbal command for the user selection. In this case, the computing device 110 can perform speech recognition on the received verbal command and / or compare the verbal command with the commands stored in the memory 120. After confirming the received verbal command, the computing device 110 can then execute the commanded function for the personalized routine management system 100.

[0042] The hybrid inference engine (HIE) 124 is an in-vehicle inference service that receives information about one or more services being operated by the vehicle and causes actions to be performed by the vehicle. In various embodiments, the HIE 124 can be an inference engine that applies logical rules from a set of determined facts. In some embodiments, the HIE 124 can include an HIE service application that communicates with other components of the vehicle and a HIE framework that includes modules that process information associated with locally stored rules 128 and existing conditions and / or existing actions that may be included in the effective rules 128.

[0043] In various implementations, the HIE 124 may receive various signals indicating various vehicle conditions or states as facts from which the HIE 124 applies logic rules ( For example , rules 128). In various embodiments, the HIE 124 may receive a set of current vehicle conditions and may compare the set of vehicle conditions to the stored rules 128 to determine whether the conditions of a given rule 128 have been satisfied ( For example , the fact given the rule is true). In some implementations, HIE 124 can process several types of logic rules, such as CLIPS rules.

[0044] In various embodiments, HIE 124 responds to a determination that rule 128(1) applies by identifying one or more actions specified in rule 128(1). After identifying the one or more actions, HIE 124 may generate one or more commands to cause the applicable system to perform the specified actions. For example, after determining that a triggering event has occurred to satisfy rule 128(2), HIE 124 may identify the specified action as closing a window. HIE 124 may then transmit commands to the individual vehicle subsystems that cause the vehicle subsystems to close any windows that are open.

[0045] The data store 126 stores values ​​and other data that are retrieved by the processing unit 140 to coordinate the operation of the personalized routine management system 100. In various embodiments, in operation, the processing unit 140 may be configured to store values ​​in the data store 126 and / or retrieve values ​​stored in the data store 126. For example, the data store 126 may store sensor data, audio content ( For example , audio clips, speech portions, etc.), a personalized routine table and / or one or more created rules 128.

[0046] In some embodiments, computing device 110 may use input / output (I / O) device interface 144 to communicate with other devices such as sensor 172, input device 174, and / or output device 176. In such cases, I / O device interface 144 may include any number of different I / O adapters or interfaces for providing the functionality described herein. For example, I / O device interface 144 may include wired and / or wireless connections, and may use various formats or protocols. In another example, computing device 110 may receive auditory signals from input device 174 via I / O device interface 144, may use sensor 172 to detect physiological data, visual data, etc., and may provide output signals to output device 176 to generate various types of output ( For example , visual indications, sound waves, tactile sensations, etc.).

[0047] In some embodiments, computing device 110 can use network interface 142 and network 150 to communicate with other devices, such as external data store 152. In some embodiments, other types of networked computing devices (not shown) can be connected to computing device 110 via network interface 142. Examples of networked computing devices include servers, desktop computers, mobile computing devices (such as smart phones or tablet computers), and / or wearable devices (such as watches or headphones or head-mounted display devices). In some embodiments, networked computing devices can be used as sensors 172, input devices 174, and / or output devices 176.

[0048] The network 150 includes a plurality of network communication systems such as routers and switches configured to facilitate data communication between the computing device 110 and the external data storage 152. Those skilled in the art will recognize that there are many technically feasible techniques for constructing the network 150, including techniques practiced when deploying Internet communication networks. For example, the network 150 may include a wide area network (WAN), a local area network (LAN), and / or a wireless (Wi-Fi) network, etc.

[0049] The external data store 152 includes various libraries that provide several types of information. For example, the external data store 152 may include a backend for a search engine, map data, etc. Additionally or alternatively, the external data store 152 may include additional services in the form of additional conditions that the HIE 124 may monitor and / or actions that the HIE 124 may cause to be performed. In such cases, the personalized routine management system 100 may expand capabilities by downloading the additional services from the external data store 152.

[0050] Sensors 172 include one or more devices that collect data associated with objects in the environment. In various embodiments, sensors 172 may include a group of sensors that acquire different sensor data. For example, sensors 172 may include reference sensors that can acquire auditory data, visual data, physiological data, etc., such as microphones and / or visual sensors ( For example , cameras, thermal imagers, linear position sensors, etc.).

[0051] In various embodiments, the sensor 172 and / or the input device 174 may include an audio sensor, such as a microphone and / or microphone array that acquires acoustic data. In various embodiments, the microphone may be directional ( For example , user-facing microphones, beamforming microphone arrays, etc.) and acquires auditory data from a particular person (e.g., user 160). Such acoustic data may be processed by the personalized routine management system 100 and / or another audio processing device using various audio processing techniques. The audio sensor may include multiple microphones or other transducers or sensors capable of converting sound waves into electrical signals. The audio sensor may include a sensor array that includes a single type of sensor or a plurality of different sensors.

[0052] Sensor 172 may include one or more devices that perform measurements and / or acquire data related to certain objects in the environment. In various embodiments, sensor 172 may generate sensor data related to user 160. For example, sensor 172 may collect biometric data related to user 160 ( For example , visible sweat, muscle movement, breathing rate, pupil size, eye glances, temporary changes in skin color, etc.) and / or biometric data associated with the user 160 while speaking ( example like , heart rate, brain activity, skin conductance, blood oxygenation, galvanic skin response, blood pressure level, average blood glucose concentration, etc.). In addition, sensor 172 may include a user-facing camera that records the face of user 160 as image data. Similarly, sensor 172 may include sensors that measure specific muscle contractions and associated activities of user 160 ( For example, raised eyebrows, clenched jaw, etc.)

[0053] In another example, the sensor 172 may include acquiring a bio-signal and / or a physiological signal of the user 160 while speaking ( For example , sweat, heart rate, heart rate variability (HRV), blood flow, blood oxygen level, breathing rate, galvanic skin response (GSR), sounds generated by the user, user's behavior, etc.). In addition, the sensor 172 may include a pupil sensor ( For example , focusing the camera on the eyes of the user 160). The personalized routine management system 100 can then perform various pupil measurement techniques to detect eye parameters ( For example , pupil diameter fluctuations, eye gaze direction, eyelid position, eye saccades, etc.) as physiological data.

[0054] The input device 174 is a device capable of receiving one or more inputs. In various embodiments, the input device 174 may include one or more audio input devices, such as a microphone, a group of microphones, and / or a microphone array. In some embodiments, the input device 174 may include a device capable of receiving input, such as a keyboard, a mouse, a touch-sensitive screen, and / or other input devices for providing input data to the computing device 110. Such inputs may include gestures, such as various movements or orientations of hands, arms, eyes, or other parts of the body received via a camera. In addition or alternatively, the input device 174 may include multiple types of sensors, including vehicle sensors (e.g., outward-facing cameras, accelerometers, etc.), occupant-facing sensors (e.g., cameras, microphones, motion sensors, etc.), and / or cabin non-occupant-facing sensors (e.g., pressure sensors, temperature sensors, etc.). In various embodiments, the input device 174 may provide a combination of sensor data describing the environment of the vehicle and the occupants present in the vehicle. For example, a sensor may provide a set of values ​​associated with a user of the vehicle ( For example , user's location, noise level, etc.).

[0055] The one or more output devices 176 include devices capable of providing output, such as a display screen, a loudspeaker, a tactile output device, etc. For example, the output device 176 may be a headset, an earbud, a speaker system ( For example , one or more loudspeakers, amplifiers, etc.) or any other device that generates a sound field. In another example, output device 176 may include a tactile output device, such as an ultrasonic transducer, an air vortex generator, an air bag, and / or any type of device configured to generate a tactile output. In various embodiments, various input devices 174 and / or output devices 176 may be incorporated into computing device 110, or may be located outside computing device 110.

[0056] In various embodiments, the output device 176 may be implemented using any number of different conventional form factors, such as discrete loudspeaker devices, circumaural, supra-aural, or in-ear headphones, hearing aids, wired or wireless headsets and / or personal speakers, body-worn (head, shoulder, arm, etc.) speaker devices, body-worn close-range directional speakers or speaker arrays, body-worn ultrasonic speaker arrays, and the like. In some embodiments, the output device 176 includes other forms of output, such as a display device that provides visual output. In some embodiments, the output device 176 may be worn by the user 160, or may be separately placed in a fixed location, or may be movable. Additionally or alternatively, the output device 176 may include various components that control the operation of the vehicle. For example, the output device 176 may include a controller that operates one or more windows, windshield wipers, locks, and the like. In some embodiments, the output device 176 may include various vehicle subsystems, such as an air conditioning subsystem, a navigation subsystem, an entertainment subsystem, and the like. In such cases, the output device 176 may receive commands and generate signals to cause specific actions ( For example , opening the driver's side window) is executed.

[0057] Figure 2 Shown according to various embodiments include Figure 1 1 and 124. As shown, the vehicle system includes, but is not limited to, an input module 210, a host unit 220, and an output module 240. The input module includes a human-machine interface (HMI) instance 212 (1). The host unit 220 includes, but is not limited to, the HIE 124, the routine management application 130, a navigation subsystem 222, an advanced driver assistance system (ADAS) 224, a network module 226, and an entertainment subsystem 228. The output module 240 includes an HMI instance 212 (2), a set of vehicle behaviors 242, a set of application parameters 244, and a set of application events 246.

[0058] In various embodiments, the personalized routine management system 100 may be included in the vehicle system 200 to locally generate and store personalized routines for execution by various components of the vehicle. In various embodiments, the HIE 124 may monitor information from one or more subsystems and may respond to a triggering event of a given rule 128 by causing one or more subsystems to perform an action. In various embodiments, the HIE 124 may cause the subsystems to perform actions in response to conditions associated with different subsystems. For example, the HIE 124 may receive information from the navigation subsystem and may respond to the applicability of the rule 128 by controlling one or more components of the entertainment subsystem 228.

[0059] Input module 210 may include an HMI instance 212(1) that includes user interface 122. In such instances, user 160 may provide input by providing voice portion 162. In some embodiments, user 160 may provide other input via HMI 212(1), such as pressing a button and / or numeric input to initiate the routine creation process, confirm approval of the personalized routine, etc.

[0060] The host unit 220 is a component of the vehicle system 200 that is mounted anywhere within the vehicle passenger compartment in any technically feasible manner. In some embodiments, the host unit 220 may include any number and type of instruments and applications, and may provide any number of input and output mechanisms. For example, the host unit 220 may allow the user ( example like , the driver and / or passengers) can control the navigation subsystem 222 and / or the entertainment subsystem 228. The head unit 220 supports any number of input and output data types and formats, as known in the art. For example, the head unit 220 may include built-in Bluetooth for hands-free calling and / or audio streaming, a universal serial bus (USB) connection, voice recognition, a rear-view camera input, a sensing module, video output for any number and type of displays via output module 240, and any number of audio outputs. In general, any number of sensors ( For example , sensors 172), displays, receivers, transmitters, etc. may be integrated into the host unit 220, or may be implemented externally to the host unit 220. In various embodiments, the external device may communicate with the host unit 220 in any technically feasible manner.

[0061] The entertainment subsystem 228 provides various information to the controlling user and / or one or more other occupants of the vehicle via the output module 240. For example, the head unit 220 may provide route information associated with the vehicle to the driver via the HMI 212. In various embodiments, the HIE 124 may control various components associated with the entertainment subsystem 228, such as media sources ( For example , via an internal source or external media provider of the network module 226) and / or an output device ( example like , output device 176, speaker, display and / or HMI 212) and / or other vehicle components. In some embodiments, network module 226 can transmit data acquired by host unit 220. Additionally or alternatively, one or more modules connected to host unit 220 can receive data from a remote source via network module 226.

[0062] The output module 240 performs one or more actions in response to commands from the HIE 124 and / or the routine management application 130. For example, the output module 240 can generate one or more output signals that cause applications and / or other vehicle components to perform actions. For example, the output module 240 can generate one or more output signals to modify the HMI 212(2) to display a notification message and / or alert or to cause the vehicle to perform a particular vehicle behavior 242. For example, the vehicle behavior 242 can include vehicle climate control settings ( For example , window control, passenger compartment temperature, increase fan speed, etc.) and / or olfactory parameters, such as emitting a particular scent that is calming or stimulating. In various embodiments, vehicle behavior 242 may include emergency call parameters, such as triggering a call to one or more emergency phone numbers or suggesting that the user connect to a particular contact situation that may require immediate assistance and / or response. In another example, output module 240 may generate one or more output signals to modify an application. In such cases, the output signal may include application parameters 244 and / or application events 246.

[0063] Figure 3 Shown according to various embodiments Figure 1 1. An exemplary personalized routine table 300 included in the personalized routine management system 100 of FIG. 1. As shown but not limited to, the personalized routine table 300 includes a column 310 for the stored context portion of the rule 128 and a column 350 for the stored action portion of the rule 128. The personalized routine table 300 includes a set of stored condition sets 312-318 and a set of action sets 352-358.

[0064] The routine management application 130 can generate personalized routines ( For example , stored condition set 312 and stored action set 352) and can store personalized routines in a specific format (such as CLIPS rules). In other embodiments, the routine management application 130 can store personalized routines outside the table.

[0065] For the first entry, the routine management application 130 may generate a personalized routine in response to the one-time speech portion 162, which includes two parts of the conditional statement and a phrase indicating a request to create a new rule. For example, the user 160 may say "Create a routine: When I turn on the AC, make sure all windows are closed." The routine management application 130 may perform various NLU techniques to determine semantic meaning and generate an intent that identifies the air conditioning system as a system that provides status information to the HIE 124 and a specific condition of the system ("ON"). The routine management application 130 may also identify specific actions that one or more vehicle components can take to cause the windows to be in a specified state. When the vehicle is in operation, the HIE 124 may receive vehicle behavior indicating the state of the air conditioning system. When the HIE 124 detects a change ( For example , from OFF to ON), the HIE 124 determines that the condition 312 has been met and identifies the specified action 352. After identifying the specified action 352, the HIE 124 can generate a command to cause the vehicle to close any open windows.

[0066] For the second entry, the routine management application 130 can generate a personalized routine based on the preferences of the user 160 to minimize the bass level of the audio when a call is in progress. For example, the user 160 can voice a request to change the bass level during a phone call. The routine management application 130 can respond by generating a condition 314 based on a state associated with the phone application and a corresponding action 354 for the entertainment subsystem 228 to perform. During operation, the HIE 124 can receive information including information related to the application event ( For example , phone application opened, call occurred, call ended, etc.), and after determining that the condition 314 has been met, the corresponding action 354 is identified and an applicable command is generated to cause the entertainment subsystem 228 to modify the bass level.

[0067] For the third entry, the routine management application 130 may store a routine that includes a condition set 316 of two or more conditions. In various embodiments, the routine may include various Boolean operators ( For example , AND, OR, XOR, etc.) to combine two or more conditions. In operation, HIE 124 can obtain For example , navigation subsystem 222, internal clock, etc.) receives a set of vehicle conditions and can determine whether two conditions in condition set 316 are met. When HIE 124 determines that condition set 316 is met, HIE 124 generates a command corresponding to action 356, in which the navigation subsystem sets a specific location ( For example , the stored “home” location) is set as the destination for the navigation route.

[0068] For the fourth entry, the routine management application 130 may store a routine that includes a condition set 318 (including two or more conditions) and an action set 358 (including two or more actions). In operation, the HIE 124 may compare a current set of vehicle conditions to the condition set 318 and determine whether any of the conditions in the condition set 318 are satisfied. When the HIE 124 determines that the condition set 318 is satisfied, the HIE 124 identifies each action in the action set 358 and generates a command that causes the applicable vehicle subsystem to perform the specified action.

[0069] In-car personalized routine generation

[0070] Figure 4 Shown according to various embodiments Figure 1 The invention relates to a technology for creating stored rules 128 and responding to triggers of stored rules 128 of the personalized routine management system 100. As shown, the personalized routine management system 400 includes, but is not limited to, a user interface 122, a routine management application 130, an HIE 124, a data store 126, and an output module 240. The routine management application 130 includes, but is not limited to, a voice recognition module 132, a routine creator 134, a rule management module 136, an intent 410, and a routine 420. The data store includes, but is not limited to, a set of conditions 412, a set of actions 414, and a set of rules 128. The HIE 124 includes, but is not limited to, a use case interface 430, a rule engine 440, and a service interface 450.

[0071] In operation, the routine management application 130 may create a routine 420 based on the intent 410 derived from one or more speech portions 162 provided by the user 160. The routine management application 130 may generate rules 128 corresponding to the routine 420 and may store the rules 128 in a data store. During subsequent operation, the HIE 124 may receive various vehicle conditions 460 and may determine whether any of the vehicle conditions satisfy the rules 128. When the HIE 124 determines that the vehicle conditions satisfy the rules 128, the HIE 124 identifies one or more actions specified by the rules 128 and transmits the command to the output module 240. The output module 240 performs the specified action that causes one or more vehicle behaviors and / or one or more application events 246.

[0072] In various embodiments, the sound recognition module 132 generates an intent 410 from one or more speech portions 162. In some embodiments, the sound recognition module 132 may receive one or more speech portions 162 provided by a user via the user interface 122. In such cases, the sound recognition module 132 may parse the speech portions 162 and generate an intent 410. The intent 410 may include various fields that specify the intent of the user to generate a personalized routine. For example, the intent 410 may include: a context field that includes information associated with the conditional statement of the routine; and an action field that includes information associated with the action statement of the routine. In some embodiments, the sound recognition module 132 may perform various speech-to-text techniques to generate text included in the intent.

[0073] The routine creator 134 responds to the receipt of the intent 410 by validating the contents of the intent 410 using various sets of pre-existing information related to the capabilities of the vehicle system 200. For example, the routine creator 134 may reference a set of conditions 412 to determine whether the contents of the context field match one or more of the set of conditions 412. Similarly, the routine creator 134 may reference a set of existing actions to determine whether the contents of the action field match one or more of the set of actions 414. In some embodiments, the set of conditions 412 and / or the set of actions 414 may be stored in a resource file maintained by the HIE 124 via a use case interface 430. For example, the use case interface 430 may include a set of general routines, a set of predefined use cases, and / or other personalized use cases from which a set of conditions 412 and / or a set of actions 414 may be derived. In another example, the service interface 450 may maintain a list of available services and corresponding conditions and actions associated with the available services.

[0074] When the routine creator 134 successfully verifies the intent 410, the routine creator 134 generates a routine 420 by mapping the contents of the intent 410 to the identified conditions and / or the identified actions. After creating the routine 420, the routine creator 134 presents the routine 420 to the user 160 for approval. Otherwise, when the routine creator 134 does not verify the intent 410, the routine creator 134 provides a notification message to the user indicating that the user 160 requested to create an invalid routine.

[0075] In some embodiments, the routine creator 134 may store the routine 420 in the data store 126. Additionally or alternatively, in various embodiments, the rule management module 136 may convert the routine 420 into the rules 128. For example, the rule management module 136 may transmit the routine 420 to a code generator (not shown) that generates the rules 128. In such cases, the rules 128 may be in a specific format ( For example, CLIPS rules, .JavaScript Object Notation (JSON) files, etc.).

[0076] In some embodiments, the rule management module 136 may provide the currently generated rule to the HIE 124, wherein the rule engine 440 determines whether the currently generated rule 128 overlaps with an existing rule ( For example , whether the currently generated rule 128 is repeated with an existing rule, whether the currently generated rule 128 conflicts with an existing rule, etc.). When the rule engine 440 determines that the currently generated rule 128 does not overlap with an existing rule, the rule engine 440 causes the rule management module 136 to store the rule 128 in the data store 126. Otherwise, when the rule engine 440 determines that the currently generated rule 128 overlaps with an existing rule, the rule engine 440 stops the rule management module 136 from storing the rule 128 in the data store 126 and generates an error message indicating that the rule 128 will not be stored and the reason ( For example , duplicate). For example, when the rule engine 440 determines that the rule 128 is a duplicate, the rule engine 440 can notify the user 160 that the rule 128 will not be stored and will start over. In another example, when the rule engine 440 determines that the rule 128 conflicts with an existing rule, the rule engine 440 can provide a prompt with various options for the user 160 to respond, such as by replacing the characteristics of the rule 128 ( For example , changing the specified action, changing the condition, etc.), replacing the existing rule or starting over to generate a different rule 128.

[0077] In various embodiments, the HIE 124 may receive vehicle conditions 460 associated with the operation of the vehicle. In such cases, the HIE 124 may determine whether the received vehicle conditions 460 satisfy the rules 128. When the HIE 124 determines that the rules 128 have been satisfied, the HIE 124 generates a command for the output module 240 to perform certain actions in the form of vehicle behaviors 242 and / or application events 246.

[0078] In various embodiments, the rule engine 440 may receive vehicle conditions 460 from one or more vehicle components. In some embodiments, the rule engine 440 may receive a subset of vehicle conditions 460 when a value associated with the vehicle condition changes. For example, when the system changes state ( For example , the air conditioning system changes from OFF to ON), the rules engine may receive a subset of vehicle conditions 460 including an updated state of the air conditioning system. Additionally or alternatively, the rules engine 440 may periodically receive vehicle conditions 460. The vehicle conditions 460 may include various values ​​monitored by vehicle components ( For example, environmental, physiological and / or other data acquired by sensor 172), information associated with the application being run by the vehicle component ( For example , information associated with content items being played by the entertainment subsystem 228 via the media player), vehicle component status, etc.

[0079] The rules engine 440 compares the vehicle condition 460 to the rules 128 stored in the data store 126. In various embodiments, the rules engine 440 can compare the information in the vehicle condition 460 to the various condition sets 312 to 318 to determine whether any of the condition sets 312 to 318 have been met. For example , condition set 318) has been met, the rule engine 440 identifies the satisfied action set of rule 128 ( For example , action set 358).

[0080] In various embodiments, the service interface 450 may generate one or more commands for a particular service to perform an action specified by the rule 128. For example, when causing a vehicle component to perform an action set 358, the service interface 450 may identify the entertainment subsystem 228 as an applicable service and may generate a command to cause the entertainment subsystem 228 to change a playlist. Similarly, the service interface 450 may identify a vehicle subsystem that controls windshield wipers and generate a command to cause the windshield wipers to start.

[0081] The output module 240 receives one or more commands from the service interface 450 and causes applicable vehicle components to execute the commands. Following the above example, the output module 240 can transmit a command to the entertainment subsystem 228 to execute a specific application event 246 to play a specific playlist. The output module 240 can also transmit a command to the wiper control to cause the vehicle behavior 242 to change by activating the wipers.

[0082] Figure 5 Shown according to various embodiments Figure 1 500 of the interactions between the various components of the personalized routine management system 100. Although the interactions between the devices are shown in sequence, those skilled in the art will appreciate that the interactions may be performed in a different order, the interactions may be repeated or skipped, and / or may be performed by Figure 5 Components other than those described are executed.

[0083] like Figure 5162 to the user interface 122. In various embodiments, when requesting the routine management application 130 to create a personalized routine, the user 160 may provide one or more voice portions 162 to the user interface. In some embodiments, the user 160 creates a personalized routine using a single utterance including a voice portion 162 having a conditional statement specifying the context and action of the routine. Alternatively, in some embodiments, the user interface 122 may provide one or more prompts (not shown) that cause the user to provide a voice portion 162 including a context portion and an action portion. The user interface 122 provides 504 the voice portion 162 to the routine management application 130.

[0084] The routine management application 130 performs an action 506 to validate the intent 410 corresponding to the speech portion 162. In various embodiments, the routine management application 130 can generate the intent 410 including a context field whose contents include (i) one or more conditions and (ii) one or more actions corresponding to the speech portion 162. In various embodiments, the routine management application 130 can validate the intent 410 by comparing the contents of the intent 410 to the sets of conditions 412 and / or the sets of actions 414 that the personalized routine management system 100 can determine and execute, respectively.

[0085] When the routine management application 130 does not successfully validate the intent 410, the routine management application 130 provides 508 an error message to the user indicating that the requested routine is invalid. Otherwise, when the routine management application 130 successfully validates the intent 410, the routine management application 130 creates 510 a routine 420 corresponding to the intent 410. In various embodiments, the routine management application 130 can generate a routine by mapping the content of the intent 410 to one or more conditions 412 and / or one or more actions 414.

[0086] In various embodiments, after creating the routine 420, the routine management application 130 may provide 512 the routine 420 to the HIE 124. The HIE 124 may then provide 514 the routine 420 to the user interface 122, which requests 516 approval from the user 160 to store the routine 420. In various embodiments, the user interface 122 may request approval 516 in various formats, such as by speaking the routine 420 via a loudspeaker, displaying the routine 420 via a display device, etc. When the user 160 reviews and approves the routine 420, the user 160 may provide approval 518 to the user interface 122. In some embodiments, the approval provided by the user 160 may be in a different format than the approval request 516. For example, the user interface 122 may display the routine 420 and a prompt requesting approval. The user 160 may then speak a speech portion 162 indicating approval to the user interface 122. When the user interface 122 receives the approval, the user interface provides 520 the approval to the HIE 124.

[0087] In various embodiments, the HIE 124 may perform various actions 522 to check whether the routine 420 overlaps with an existing routine stored locally in the data store 126. In some embodiments, the routine management application 130 may convert the routine 420 into a rule 128. In such cases, the HIE 124 may compare the rule 128 corresponding to the approved routine 420 with the existing rule 128 stored in the data store 126. In some embodiments, the HIE 124 may determine whether the routine 420 is a duplicate of an existing routine or whether the routine 420 conflicts with the existing routine 420. For example, a user may approve a routine 420 that responds to the air conditioner turning on by setting the volume level of the speaker to level 10. When the HIE 124 identifies an existing routine that responds to the air conditioner turning on by setting the volume to level 5, the HIE 124 may determine that the two routines overlap.

[0088] When the HIE 124 identifies at least one overlapping routine, the HIE 124 provides 524 an error message to the user interface 122. In various embodiments, the user interface 122 may then provide 526 an error message indicating that the routine 420 approved by the user 160 will not be stored and the reason for the error ( For example , overlapping with an existing routine). In some embodiments, HIE 124 may provide a prompt with various options for user 160 to respond to the overlap, such as by replacing the features of the routine ( For example , changing the specified action, changing the conditions, etc.), replacing the existing routine, starting again to generate a different routine, etc.

[0089] When the HIE 124 determines that the routine 420 does not overlap with an existing routine, the HIE 124 performs various actions 528 to store the routine in the data store 126 .

[0090] Figure 6 A flow chart illustrating method steps for generating rules corresponding to a personalized routine specified by a user according to various embodiments. Figures 1 to 5 Although the method steps are described in the embodiments of the present disclosure, those skilled in the art will understand that any system configured to implement the method steps in any order falls within the scope of the present disclosure.

[0091] like Figure 6 As shown, method 600 begins at step 602, where the routine management application 130 generates an intent 410 from one or more received speech portions 162. In various embodiments, the voice recognition module 132 included in the routine management application 130 generates the intent 410 from the one or more speech portions 162. In some embodiments, the voice recognition module 132 may receive one or more speech portions 162 provided by the user via the user interface 122. In such cases, the voice recognition module 132 may parse the speech portions 162 and generate the intent 410. The intent 410 may include various fields specifying the intent of the user to generate a personalized routine. For example, the intent 410 may include: a context field, which includes information associated with the conditional statement of the routine; and an action field, which includes information associated with the action statement of the routine.

[0092] At step 604, the routine management application 130 determines whether the intent 410 is valid. In various embodiments, the routine creator 134 can respond to the receipt of the intent 410 by validating the content of the intent 410. In such cases, the routine creator 134 can refer to a set of conditions 412 to determine whether the content of the context field included in the intent 410 matches one or more of the set of conditions 412. Similarly, the routine creator 134 can refer to a set of actions 414 to determine whether the content of the action field matches one or more of the set of actions 414. When the routine creator 134 successfully validates the intent 410, the routine management application 130 proceeds to step 608. Otherwise, when the routine creator 134 does not validate the intent 410, the routine management application 130 provides an error message and ends the method 600.

[0093] At step 606, the routine management application 130 generates a routine 420 from the intent 410. In various embodiments, the routine creator 134 can generate the routine 420 by mapping the content of the intent 410 to the identified conditions and / or identified actions for verifying the intent 410. At step 608, the routine management application 130 provides the routine 420 for approval. In various embodiments, after creating the routine 420, the routine creator 134 presents the routine 420 to the user 160 for approval. At step 610, the routine management application 130 determines whether the user 160 approves the routine 420. When the routine management application 130 receives an indication that the user 160 approves the routine, the routine management application 130 proceeds to step 614. Otherwise, when the routine management application 130 determines that the user 160 does not approve the routine 420, the routine management application 130 ends the method 600.

[0094] At step 612, the routine management application 130 determines whether the routine 420 overlaps with an existing routine. In various embodiments, the routine management application 130 may examine one or more stored routines stored in the data store 126 to determine whether the routine 420 overlaps with an existing routine. In some embodiments, the data store 126 may store routines in different formats ( For example , one or more rules 128). In such cases, the routine management application 130 may convert the routine to a different format before checking for overlap. Additionally or alternatively, the routine management application 130 may cause the HIE 124 to check for overlap. When the routine management application 130 does not identify at least one overlapping routine, the routine management application 130 proceeds to step 614 and stores the routine 420 in the data store 126 before ending the method 600. Otherwise, when the routine management application 130 identifies at least one overlapping routine, the routine management application 130 proceeds to step 616 and rejects the routine 420. In such cases, the routine management application 130 may provide a notification to the user 160 that the routine 420 will not be stored. Additionally or alternatively, the routine management application 130 may prompt the user 160 with available actions to take in response to the overlap, such as replacing the characteristics of the routine 420 ( For example , changing the specified action, changing the condition, etc.), replacing the existing routine, starting over to generate a different routine, etc. After rejecting the routine 420, the routine management application 130 ends the method 600.

[0095] Figure 7 A flowchart illustrating method steps for responding to detection of one or more satisfied conditions of a rule by performing an action specified by the rule according to various embodiments. Figures 1 to 5Although the method steps are described in the embodiments of the present disclosure, those skilled in the art will understand that any system configured to implement the method steps in any order falls within the scope of the present disclosure.

[0096] like Figure 7 As shown, the method 700 begins at step 702, where the HIE 124 receives a set of vehicle conditions 460. In various embodiments, the rule engine 440 included in the HIE 124 may receive a set of vehicle conditions 460 provided by one or more vehicle components. The vehicle conditions 460 may include various values ​​monitored by the vehicle components ( example such as environmental, physiological, and / or other data acquired by sensors 172), information associated with applications being run by vehicle components ( example For example, information associated with a content item being played via a media player by the entertainment subsystem 228), vehicle component status, etc. In some embodiments, the rule engine 440 may receive a subset of vehicle conditions 460 when a particular value changes. Additionally or alternatively, the rule engine 440 may receive vehicle conditions 460 periodically.

[0097] At step 704, the HIE 124 determines whether the vehicle conditions 460 are applicable to the routine 420. In various embodiments, the rule engine may compare the vehicle conditions 460 to the contents of the routines stored in the data storage 126. In such cases, the rule engine 440 compares a set of vehicle conditions 460 to a set of conditions for a given routine 420 to determine whether the vehicle conditions 460 are applicable to the routine 420. For example, when the rule engine 440 compares the vehicle conditions 460 that include a change in the audio track being played by the media player, the rule engine 440 may determine that the given routine 420 does not include any conditions associated with the media player. When the rule engine 440 determines that the vehicle conditions 460 are applicable to the routine 420, the HIE 124 proceeds to step 706. Otherwise, when the rule engine 440 determines that the vehicle conditions 460 are not applicable to any routine 420, the HIE 124 returns to step 702 to receive additional vehicle conditions 460.

[0098] At step 706, the HIE 124 determines whether the conditions of the routine 420 are met. In various embodiments, the rule engine 440 can compare the information in the vehicle condition 460 with the condition set ( For example , condition set 316) to determine whether the condition set 316 has been met. When the rule engine 440 determines that the condition set has been met, the HIE 124 proceeds to step 708. Otherwise, when the rule engine 440 determines that the condition set 316 has not been met, the HIE 124 returns to step 702 to wait for the additional vehicle conditions 460.

[0099] At step 708, the HIE 124 determines actions from the routine 420. In various embodiments, after determining that the condition set 316 has been satisfied, the rule engine 440 identifies the action set 356 included in the routine 420 that has been satisfied.

[0100] At step 710, the HIE 124 causes the action to be performed. In various embodiments, the service interface 450 included in the HIE 124 can generate one or more commands for a particular service (to perform the action specified by the routine 420). For example, when causing a vehicle component to perform an action set 356, the service interface 450 can identify the navigation subsystem 222 as an applicable service and can generate a command that causes the navigation subsystem 222 to set the stored address ("home") as the target destination of the route. In such an example, the output module 240 can receive the one or more commands from the service interface 450 and cause the applicable vehicle component to perform the command.

[0101] In summary, the routine management application receives one or more auditory voice signals from the user to generate a customized routine. In some embodiments, the routine management application may receive a single auditory voice signal specifying the components of a conditional statement, the components including a context portion and an action portion. Alternatively, the routine management application may provide one or more prompts that cause the user to separate the auditory voice signal and a separate auditory voice signal including an action portion. After receiving one or more auditory voice signals, the routine management application parses the auditory voice signal to generate an intention based on the components of the conditional statement. The routine management application verifies the intention based on one or more pre-existing conditions that the vehicle can monitor and one or more pre-existing actions that the vehicle can perform in response to a trigger condition. After verifying the intention, the routine management application generates a routine based on the intention, the routine including one or more conditions and one or more corresponding actions. When the user approves the generated routine, the routine management application compares the routine with one or more local storage rules. When the routine management application determines that the generated routine does not overlap with at least one local storage rule, the routine management application publishes the routine by generating and storing rules corresponding to the routine.

[0102] Additionally or alternatively, the hybrid inference engine monitors a set of values ​​associated with conditions specified in a set of stored rules. When a change in one of the set of values ​​is detected, the hybrid inference engine determines whether the conditions for one or more of the stored rules are satisfied. When the hybrid inference engine determines that the conditions for a given stored rule have been satisfied, the hybrid inference engine identifies one or more actions specified in the given stored rule and causes one or more components of the vehicle to perform the one or more specified actions.

[0103] At least one technical advantage of the disclosed technology over the prior art is that, utilizing the disclosed technology, a user who is not proficient in programming customized vehicle routines can also effectively generate and approve customized vehicle routines that respond to various detected conditions, thereby greatly improving the in-vehicle experience of the user of the vehicle. In particular, by processing the user's voice to generate effective routines for storage and monitoring, the routine management application reduces the physical and mental strain of the user of the vehicle in performing various manual tasks associated with the operation of the vehicle. In addition, by interpreting the user's emotions when dictating the personalized routines, the personalized routine management system provides a wider range of available conditions to monitor and operations to perform than menu-based user interfaces. These technical advantages provide one or more technical advances over prior art methods.

[0104] 1. In various embodiments, a computer-implemented method includes: receiving a set of auditory speech signals generated by a user; mapping a first speech portion in the set of auditory speech signals to a first condition, wherein the first condition corresponds to at least a first value associated with operation of a first component of a vehicle; mapping a second speech portion in the set of auditory speech signals to a first action, wherein the first action corresponds to at least a first action performed by a second component of the vehicle; generating a routine including the first condition and the first action; comparing the routine to a set of stored rules, wherein each stored rule in the set of stored rules includes at least one condition and at least one action; upon determining that the routine does not overlap with at least one stored rule in the set of stored rules, storing the routine as a first stored rule; upon storing the first stored rule, determining that the first condition has been satisfied; and causing the second component of the vehicle to perform the first action.

[0105] 2. A computer-implemented method according to clause 1, wherein mapping the first speech portion to the first condition includes comparing the first speech portion with a set of predefined conditions, wherein each predefined condition in the set of predefined conditions identifies at least one of a state value of a component of the vehicle or a measured value generated by a component of the vehicle or a value determined by a component of the vehicle, determining that the first speech portion corresponds to at least a first predefined condition in the set of predefined conditions, and setting the first condition to match the first predefined condition.

[0106] 3. A computer-implemented method according to clause 1 or 2, further comprising: determining that a set of conditions included in a second stored rule has been satisfied, wherein the set of conditions includes at least a second condition and a third condition; and causing a third component of the vehicle to perform a second action of the second stored rule.

[0107] 4. A computer-implemented method according to any of clauses 1 to 3, wherein mapping the second speech portion to the first action includes comparing the second speech portion to a set of predefined actions, wherein each predefined condition in the set of predefined actions identifies at least one of a vehicle behavior performed by a component of the vehicle or an application event performed by a component of the vehicle, or determining that the second speech portion corresponds to at least a first predefined action in the set of predefined actions, and setting the first action to match the first predefined action.

[0108] 5. A computer-implemented method according to any of clauses 1 to 4, further comprising causing the second component of the vehicle to perform a set of actions included in the routine, wherein the set of actions includes at least the first action and a second action.

[0109] 6. A computer-implemented method according to any of clauses 1 to 5, further comprising receiving an indication of a change in one or more values ​​associated with the operation of the first component of the vehicle, wherein determining that the first condition has been satisfied comprises determining that the change in the one or more values ​​associated with the operation of the first component of the vehicle satisfies the first condition.

[0110] 7. A computer-implemented method as described in any of clauses 1 to 6, wherein the first component of the vehicle or the second component of the vehicle is an infotainment subsystem, a navigation subsystem, an advanced driver assistance system (ADAS), or a temperature control subsystem.

[0111] 8. A computer-implemented method according to any one of clauses 1 to 7, further comprising: determining the first speech portion from a first auditory speech signal included in the set of auditory speech signals; and determining the second speech portion from the first auditory speech signal.

[0112] 9. A computer-implemented method according to any one of clauses 1 to 8, further comprising: providing a first auditory prompt associated with the first condition before receiving a first auditory speech signal from the set of auditory speech signals, wherein the first speech portion is determined from the first auditory speech signal; providing a second auditory prompt associated with the first action after receiving the first auditory speech signal; and receiving a second auditory speech signal from the set of auditory speech signals.

[0113] 10. A computer-implemented method according to any one of clauses 1 to 9, further comprising: parsing a first auditory speech signal included in the set of auditory speech signals to generate a set of speech parts; generating a first intent based on the set of speech parts, the first intent specifying (i) a value or state and (ii) a set of target components; validating the first intent using at least one of a set of predefined conditions or a set of predefined actions, wherein the routine is generated in response to validating the first intent.

[0114] 11. A computer-implemented method according to any one of clauses 1 to 10, further comprising determining that the routine overlaps with the at least one stored rule when the routine duplicates the at least one stored rule, or when the routine conflicts with at least one action included in the at least one stored rule.

[0115] 12. A computer-implemented method as described in any of clauses 1 to 11, further comprising, upon determining that the routine overlaps with at least one stored rule in the set of stored rules, providing a prompt to the user to select at least one of the following: (i) replacing the at least one stored rule with the routine, (ii) modifying the routine, or (iii) discarding the routine.

[0116] 13. In various embodiments, one or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the following steps: receiving a set of auditory speech signals generated by a user; mapping a first speech portion in the set of auditory speech signals to a first condition, wherein the first condition corresponds to at least a first value associated with operation of a first component of a vehicle; mapping a second speech portion in the set of auditory speech signals to a first action, wherein the first action corresponds to at least a first action performed by a second component of the vehicle; generating a routine including the first condition and the first action; comparing the routine to a set of stored rules, wherein each stored rule in the set of stored rules includes at least one condition and at least one action; upon determining that the routine overlaps with at least one stored rule in the set of stored rules, providing a notification to the user that the routine should not be stored as the first stored rule; or upon determining that the routine does not overlap with the at least one stored rule in the set of stored rules, storing the routine as the first stored rule; upon storing the first stored rule, determining that the first condition has been satisfied; and causing the second component of the vehicle to perform the first action.

[0117] 14. The one or more non-transitory computer-readable media of clause 13, wherein the first stored rules comprise C Language Integrated Production System (CLIPS) rules.

[0118] 15. One or more non-transitory computer-readable media as described in claim 13 or 14, further storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the following steps: receiving an indication of a change in one or more values ​​associated with the operation of the first component of the vehicle, wherein determining that the first condition has been satisfied includes determining that the change in the one or more values ​​associated with the operation of the first component of the vehicle satisfies the first condition.

[0119] 16. One or more non-transitory computer-readable media as described in any of clauses 13 to 15, wherein mapping the second speech portion to the first action includes: comparing the second speech portion with a set of predefined actions, wherein each predefined condition in the set of predefined actions identifies at least one of a vehicle behavior performed by a component of the vehicle or an application event performed by a component of the vehicle; or determining that the second speech portion corresponds to at least a first predefined action in the set of predefined actions, and setting the first action to match the first predefined action.

[0120] 17. One or more non-transitory computer-readable media as described in clauses 13 to 16, wherein determining that the routine overlaps with the at least one stored rule includes determining that (i) the routine duplicates the at least one stored rule, or (ii) the routine conflicts with at least one action included in the at least one stored rule, and the notification includes prompting the user to select at least one of the following: (i) replace the at least one stored rule with the routine, (ii) modify the routine, or (iii) discard the routine.

[0121] 18. In various embodiments, a system includes: a memory storing a routine management application; and a processor coupled to the memory, the processor executing the routine management application by performing the following steps: receiving a set of auditory speech signals generated by a user; mapping a first speech portion in the set of auditory speech signals to a first condition, wherein the first condition corresponds to at least a first value associated with the operation of a first component of a vehicle; mapping a second speech portion in the set of auditory speech signals to a first action, wherein the first action corresponds to at least a first action performed by a second component of the vehicle; generating a routine including the first condition and the first action; comparing the routine to a set of stored rules, wherein each stored rule in the set of stored rules includes at least one condition and at least one action; upon determining that the routine does not overlap with at least one stored rule in the set of stored rules, storing the routine as a first stored rule; upon storing the first stored rule, determining that the first condition has been satisfied; and causing the second component of the vehicle to perform the first action.

[0122] 19. A system according to clause 18, wherein the memory also stores a hybrid inference engine (HIE) service, and the processor also executes the HIE service by performing the step of receiving an indication of a change to one or more values ​​associated with the operation of the first component of the vehicle, wherein determining that the first condition has been satisfied includes determining that the change to the one or more values ​​associated with the operation of the first component of the vehicle satisfies the first condition.

[0123] 20. The system of clause 18 or 19, wherein the first component of the vehicle or the second component of the vehicle comprises at least one of an infotainment subsystem, a navigation subsystem, an advanced driver assistance system (ADAS), or a temperature control subsystem.

[0124] Any and all combinations of any of the claim elements recited in any of the claims and / or any of the elements described in this application, in any form, are within the contemplated scope of the invention and protection.

[0125] The description of the various embodiments has been presented for purposes of illustration, but the description is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments.

[0126] Aspects of the present embodiment may be embodied as a system, method or computer program product. Therefore, aspects of the present disclosure may take the following forms: a complete hardware implementation, a complete software implementation (including firmware, resident software, microcode, etc.) or an implementation combining software aspects and hardware aspects, which are generally all referred to as "modules", "systems" or "computers" in this article. In addition, any hardware and / or software technology, process, function, component, engine, module or system described in the present disclosure may be implemented as a circuit or a group of circuits. In addition, aspects of the present disclosure may take the form of a computer program product, which is embodied in one or more computer-readable media on which computer-readable program codes are embodied.

[0127] Any combination of one or more computer-readable media may be utilized. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or apparatus, or any suitable combination of the foregoing. More specific examples (non-exhaustive list) of computer-readable storage media would include the following: an electrical connection with one or more wires, a portable computer floppy disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer-readable storage medium may be any tangible medium that may contain or store a program for use by or in connection with an instruction execution system, device, or apparatus.

[0128] Aspects of the present disclosure are described above with reference to the flowchart illustration and / or block diagram of the method, device (system) and computer program product according to the embodiments of the present disclosure. It should be understood that each frame in the flowchart illustration and / or block diagram and the frame combination in the flowchart illustration and / or block diagram can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine. The instructions enable the function / action specified in one or more frames of the flowchart and / or block diagram to be realized when executed via the processor of a computer or other programmable data processing device. Such processors can be, but are not limited to, general-purpose processors, special-purpose processors, special application processors or field programmable gate arrays.

[0129] The flow charts and block diagrams in the accompanying drawings illustrate the architecture, functions and operations of possible implementations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this respect, each frame in the flow chart or block diagram may represent a module, segment or part of a code, and the code includes one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions proposed in the frame may not occur in the order proposed in the accompanying drawings. For example, in fact, depending on the functions involved, the two frames shown in succession can be executed substantially simultaneously, or sometimes the frames can be executed in reverse order. It should also be noted that each frame illustrated in the block diagram and / or flow chart and the combination of frames in the block diagram and / or flow chart can be implemented by a system based on special-purpose hardware that performs a specified function or action or by a combination of special-purpose hardware and computer instructions.

[0130] While the foregoing is directed to embodiments of the present disclosure, other and further embodiments of the present disclosure may be devised without departing from the basic scope thereof, the scope of which is determined by the following claims.

Claims

1. A computer-implemented method, wherein include: receiving a set of auditory speech signals generated by a user; mapping a first speech portion in the set of auditory speech signals to a first condition, wherein the first condition corresponds to at least a first value associated with operation of a first component of a vehicle; mapping a second speech portion in the set of auditory speech signals to a first action, wherein the first action corresponds to at least a first action performed by a second component of the vehicle; generating a routine including the first condition and the first action; comparing the routine to a set of stored rules, wherein each stored rule in the set of stored rules includes at least one condition and at least one action; upon determining that the routine does not overlap with at least one stored rule in the set of stored rules, storing the routine as a first stored rule; After storing the first stored rule, determining that the first condition has been satisfied; and The second component of the vehicle is caused to perform the first action.

2. The computer-implemented method of claim 1 , wherein mapping the first portion of speech to the first condition include: The first speech portion is compared to a set of predefined conditions, wherein each predefined condition in the set of predefined conditions identifies at least one of: a status value of a component of the vehicle, or a measured value generated by a component of the vehicle, or a value determined by a component of said vehicle; determining that the first speech portion corresponds to at least a first predefined condition of the set of predefined conditions; as well as The first condition is set to match the first predefined condition.

3. The computer-implemented method of claim 1 , further comprising: include: determining that a set of conditions included in a second stored rule has been satisfied, wherein the set of conditions includes at least a second condition and a third condition; as well as A third component of the vehicle is caused to perform a second action of the second stored rule.

4. The computer-implemented method of claim 1 , wherein mapping the second speech portion to the first action include: The second speech portion is compared to a set of predefined actions, wherein each predefined action in the set of predefined actions identifies at least one of: the vehicle actions performed by components of the vehicle, or an application event performed by a component of the vehicle; or determining that the second speech portion corresponds to at least a first predefined action in the set of predefined actions; as well as The first action is set to match the first predefined action.

5. The computer-implemented method of claim 1 , further comprising: include: causing the second component of the vehicle to perform a set of actions included in the routine, The set of actions includes at least the first action and the second action.

6. The computer-implemented method of claim 1, further comprising: include: receiving an indication of a change in one or more values ​​associated with the operation of the first component of the vehicle, Wherein determining that the first condition has been satisfied comprises determining that the change in the one or more values ​​associated with the operation of the first component of the vehicle satisfies the first condition.

7. The computer-implemented method of claim 1, wherein the first component of the vehicle or the second component of the vehicle is an infotainment subsystem, a navigation subsystem, an advanced driver assistance system (ADAS), or a climate control subsystem.

8. The computer-implemented method of claim 1, further comprising: include: determining the first speech portion from a first auditory speech signal included in the set of auditory speech signals; as well as The second speech portion is determined from the first auditory speech signal.

9. The computer-implemented method of claim 1, further comprising: include: Prior to receiving a first auditory speech signal in the set of auditory speech signals, providing a first auditory cue associated with the first condition, wherein determining the first speech portion from the first auditory speech signal; After receiving the first auditory voice signal, providing a second auditory prompt associated with the first action; as well as A second auditory speech signal from the set of auditory speech signals is received.

10. The computer-implemented method of claim 1, further comprising: include: parsing a first auditory speech signal included in the set of auditory speech signals to generate a set of speech parts; generating a first intent based on the set of speech parts, the first intent specifying (i) a value or state and (ii) a set of target components; A first intent is validated using at least one of a set of predefined conditions or a set of predefined actions, wherein the routine is generated in response to validating the first intent.

11. The computer-implemented method of claim 1 , further comprising determining that the routine overlaps with the at least one stored rule if: The routine repeats with the at least one stored rule; or The routine conflicts with at least one action included in the at least one stored rule.

12. The computer-implemented method of claim 11 further comprising, upon determining that the routine overlaps with at least one stored rule in the set of stored rules, providing a prompt to the user to select at least one of: (i) replacing the at least one stored rule with the routine, (ii) modifying the routine, or (iii) discarding the routine.

13. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform the following steps: receiving a set of auditory speech signals generated by a user; mapping a first speech portion in the set of auditory speech signals to a first condition, wherein the first condition corresponds to at least a first value associated with operation of a first component of a vehicle; mapping a second speech portion in the set of auditory speech signals to a first action, wherein the first action corresponds to at least a first action performed by a second component of the vehicle; generating a routine including the first condition and the first action; comparing the routine to a set of stored rules, wherein each stored rule in the set of stored rules includes at least one condition and at least one action; upon determining that the routine overlaps with at least one stored rule in the set of stored rules, providing notification to the user that the routine is not to be stored as a first stored rule; or Upon determining that the routine does not overlap with the at least one stored rule in the set of stored rules, then: storing the routine as the first stored rule; After storing the first stored rule, determining that the first condition has been satisfied; as well as The second component of the vehicle is caused to perform the first action.

14. The one or more non-transitory computer-readable media of claim 13, wherein the first stored rules comprise C Language Integrated Production System (CLIPS) rules.

15. The one or more non-transitory computer-readable media of claim 13, further storing instructions that, when executed by the one or more processors, cause the one or more processors to perform the following steps: receiving an indication of a change in one or more values ​​associated with the operation of the first component of the vehicle, Wherein determining that the first condition has been satisfied comprises determining that the change in the one or more values ​​associated with the operation of the first component of the vehicle satisfies the first condition.

16. The one or more non-transitory computer-readable media of claim 13, wherein the second speech portion is mapped to the first action include: The second speech portion is compared to a set of predefined actions, wherein each predefined action in the set of predefined actions identifies at least one of: the vehicle actions performed by components of the vehicle, or an application event performed by a component of the vehicle; or determining that the second speech portion corresponds to at least a first predefined action in the set of predefined actions; as well as The first action is set to match the first predefined action.

17. One or more non-transitory computer readable media according to claim 13, in: Determining that the routine overlaps with the at least one stored rule includes determining that (i) the routine duplicates the at least one stored rule, or (ii) the routine conflicts with at least one action included in the at least one stored rule; and The notification includes prompting the user to select at least one of: (i) replace the at least one stored rule with the routine, (ii) modify the routine, or (iii) discard the routine.

18. A system, wherein include: a memory storing routine management applications; as well as A processor coupled to the memory, the processor executing the routine management application by performing the following steps: receiving a set of auditory speech signals generated by a user; mapping a first speech portion in the set of auditory speech signals to a first condition, wherein the first condition corresponds to at least a first value associated with operation of a first component of a vehicle; mapping a second speech portion in the set of auditory speech signals to a first action, wherein the first action corresponds to at least a first action performed by a second component of the vehicle; generating a routine including the first condition and the first action; comparing the routine to a set of stored rules, wherein each stored rule in the set of stored rules includes at least one condition and at least one action; upon determining that the routine does not overlap with at least one stored rule in the set of stored rules, storing the routine as a first stored rule; After storing the first stored rule, determining that the first condition has been satisfied; and The second component of the vehicle is caused to perform the first action.

19. A system according to claim 18, in: The memory further stores a hybrid inference engine (HIE) service; and The processor also executes the HIE service by performing the following steps: receiving an indication of a change in one or more values ​​associated with the operation of the first component of the vehicle, Wherein determining that the first condition has been satisfied comprises determining that the change in the one or more values ​​associated with the operation of the first component of the vehicle satisfies the first condition.

20. The system of claim 18, wherein the first component of the vehicle or the second component of the vehicle comprises at least one of an infotainment subsystem, a navigation subsystem, an advanced driver assistance system (ADAS), or a temperature control subsystem.

Images