Systems and methods for using vehicle in-cabin sensors for interacting with personal devices

Abstract

Systems and methods are described for leveraging an occupant monitoring system (OMS) of a vehicle to provide additional user interface options for interacting with a personal computing device while travelling in the vehicle. When an occupant of the vehicle is using the personal computing device in the vehicle, the occupant may select to interact with the personal computing device via a gestural interface, where the occupant may perform physical gestures (e.g., hand gestures) in the air that are captured by the OMS system and then mapped to functionalities of the personal computing device. A trained machine learning (ML) model may be used to detect one of various supported physical gestures in images captured by the OMS system. In various examples, the user may define desired gestural controls from a library of a set of predefined physical gestures on which the gestural detection model is trained.

Description

Attorney Docket No. P240063WOSYSTEMS AND METHODS FOR USING VEHICLE IN-CABIN SENSORS FOR INTERACTING WITH PERSONAL DEVICESTECHNICAL FIELD

[0001] Embodiments of the subject matter disclosed herein relate to integrating personal computing devices into a smart ecosystem within a cabin of a vehicle.BACKGROUND

[0002] Occupants of a vehicle may use devices, such as smart phones or tablets, while sitting in the vehicle. Modem vehicle systems support wireless communication between such devices and control systems of the vehicle, where a control system may be controlled using an interface on the device. For example, an in-vehicle infotainment (IVI) system of the vehicle may be configured to display video content to the occupants of the vehicle on personal display screens positioned in front of passenger seats of the vehicle. The video content may include information about one or more systems of the vehicle; information about environmental settings of a cabin of the vehicle, information about music playing in the cabin, cabin temperature, etc. In some vehicles, an occupant may control video display or audio output of the IVI system, or settings of the one or more systems of the vehicle, via a user interface on a smart phone. This may include adjusting controls of a vehicle audio source or settings, and / or playing audio or video content accessed via the smart phone on in-cabin display or output devices. In this way, a smart phone experience may be partially or fully mirrored in the IVI system. However, modem vehicle systems may not support mirroring of an in-cabin experience on a personal device.SUMMARY

[0003] The current disclosure at least partially addresses one or more of the above identified issues by a method for a controller of a vehicle, the method comprising detecting a personal computing device of an occupant of the vehicle via a radio frequency (RF) signal; detecting a physical gesture of the occupant via an in-cabin sensing device of the vehicle while the occupant is using the personal computing device, the physical gesture performed in the air without touching the personal computing device; mapping the physical gesture to an action, the action performed via a user interface (UI) of the personal computing device; and notifying a gestural interface service running on the personal computing device of the action; wherein the action is performed by the gestural interface service on the personal computing device.Attorney Docket No. P240063WO

[0004] The above advantages and other advantages, and features of the present description will be readily apparent from the following Detailed Description when taken alone or in connection with the accompanying drawings. It should be understood that the summary above is provided to introduce in simplified form a selection of concepts that are further described in the detailed description. It is not meant to identify key or essential features of the claimed subject matter, the scope of which is defined uniquely by the claims that follow the detailed description. Furthermore, the claimed subject matter is not limited to implementations that solve any disadvantages noted above or in any part of this disclosure.BRIEF DESCRIPTION OF THE DRAWINGS

[0005] Various aspects of this disclosure may be better understood upon reading the following detailed description and upon reference to the drawings in which:

[0006] FIG. 1 is a schematic diagram of a vehicle, according to one or more embodiments of the disclosure;

[0007] FIG. 2 shows a block diagram of an exemplary in-cabin monitoring system of a vehicle, according to one or more embodiments of the disclosure;

[0008] FIG. 3 shows a high-level block diagram illustrating a process for leveraging a vehicle in-cabin monitoring system to control a personal computing device of an occupant of a vehicle, according to one or more embodiments of the disclosure;

[0009] FIG. 4 is a flowchart illustrating an exemplars’ method for detecting gestures of an occupant of a vehicle used to control a personal computing device of the occupant using an incabin monitoring system, according to one or more embodiments of the disclosure;

[0010] FIG. 5 is a flowchart illustrating an exemplary method for controlling an application of a smart phone using gestural data provided by a vehicle in-cabin monitoring system, according to one or more embodiments of the disclosure;

[0011] FIG. 6 shows an exemplary dashboard of a vehicle including a plurality of controls, in accordance with one or more embodiments of the present disclosure; and

[0012] FIG. 7 is a schematic block diagram that shows an in-vehicle computing system and a control system of a vehicle, in accordance with one or more embodiments of the present disclosure.

[0013] The drawings illustrate specific aspects of the described systems and methods. Together with the following description, the drawings demonstrate and explain the structures, methods, and principles described herein. In the drawings, the size of components may be exaggerated or otherwise modified for clarity. Well-known structures, materials, or operationsAttorney Docket No. P240063WO are not shown or described in detail to avoid obscuring aspects of the described components, systems, and methods.DETAILED DESCRIPTION

[0014] Methods are disclosed herein for leveraging vehicle systems, in particular, an occupant monitoring system (OMS), to provide an expanded set of user interface (UI) options for occupants interacting with personal computing devices while travelling in a vehicle. Specifically, in accordance with the systems and methods described herein, a user of a personal computing device, such as a phone, tablet, eBook reading device, etc., can optionally perform actions on the personal computing device by making physical gestures in the air, which may be interpreted from video or images captured by the OMS system. For example, a child in a rear seat of the vehicle may wish to play a first person shooter video game on a personal tablet positioned on a back side of the rear seat, and may interact with the video game by using their hand as a gun. As a second example, a person reading a book in an eBook reader positioned in front of them may turn pages of the eBook by waving their hand.

[0015] To facilitate such interactions, a device interface manager component of a controller of the vehicle may communicate with a gestural interface service installed on the personal computing device. When the personal computing device pairs with the vehicle, for example, using Bluetooth®, a user of the personal computing device may be prompted to enable the gestural interface service. Using a UI of the gestural interface service on the personal computing device, the user may define mappings of physical gestures performed in the air to actions performed in one or more applications running on the personal computing device. The mappings may be sent from the gestural interface service to the device interface manager, and the device interface manager may monitor images generated of the user captured by an OMS camera positioned at a seat of the user for gestures defined in the mappings. When a control gesture is detected, the device interface manager may send a notification to the personal computing device to perform the action mapped to the gesture in the defined mappings.

[0016] By allowing users of personal computing devices to interact with applications running on the personal computing devices via physical gestures, a user experience while interacting with the applications may be more satisfying. Actions may be performed in a simpler, more intuitive manner than via a screen interface of the computing device. A personal computing device may be placed in a stable viewing position in front of a passenger, such as on the back of a seat in front the passenger, where interacting with a display screen interface of the personal computing device may demand that the passenger uncomfortably extend an armAttorney Docket No. P240063WO each time a control is selected. Additionally, when the vehicle is in motion, particularly in city traffic or on bumpy roads, selecting controls on a display screen of the personal computing device may be difficult and may result in selection errors.

[0017] Turning now to the figures, FIG. 1 schematically shows an exemplary vehicle 100. Vehicle 100 may be a car, a bus, a truck, or a different ty pe of machinery7or vehicle operated by an operator. Vehicle 100 may also be a train, aircraft, or a different type of vehicle that carries passengers. Vehicle 100 may be powered by an internal combustion engine, or vehicle 100 may be an electric vehicle powered by an electrical power source, or vehicle 100 may be a hybrid vehicle powered by both an internal combustion engine and an electrical power source. Vehicle 100 may also be a specialized vehicle used in a specific environment, such as, for example, a golf cart or transportation vehicle used in certain areas of a private facility such as an indoor facility. Vehicle 100 may be operated on public and / or private roads and highways, or on a set of tracks or rails (e.g., a train). In general, vehicle 100 may be any type of vehicle operated by an operator, or an autonomous or semi-autonomous vehicle, which may operate without a driver or without a direct involvement of the driver.

[0018] Vehicle 100 includes a dashboard 102, a driver seat 104, a first passenger seat 106. a second passenger seat 108, and a third passenger seat 110. In other examples, the vehicle 100 may include more or fewer passenger seats. The driver seat 104 and the first passenger seat 106 are located in a front of the vehicle, proximate to the dashboard 102, and therefore may be referred to as front seats. The second passenger seat 108 and the third passenger seat 110 are located at a rear of the vehicle and may be referred to as back (or rear) seats.

[0019] Vehicle 100 includes a plurality of integrated speakers 114, w hich may be arranged around a periphery of the vehicle 100 and / or at seats (e.g., headrests) of vehicle 100. In some embodiments, the integrated speakers 114 are electronically coupled to an electronic control system of the vehicle, such as to a computing system 120, via a wired connection. In other embodiments, the integrated speakers 114 may wirelessly communicate with the computing system 120. As an example, an audio file may be generated by computing system 120 or selected by an occupant of the vehicle 100, and the selected audio file may be played at one or more of the integrated speakers 114. In some examples, audio alerts may be generated by the computing system 120 and also may be played at the integrated speakers 114. In some embodiments, audio files, signals, and / or alerts may be selected by or generated for an occupant of vehicle 100, and may be played an integrated speaker 114 associated with and / or proximal to a seat of the occupant. Further, in some embodiments, audio files, signals, and / or alertsAttorney Docket No. P240063WO selected by or generated for the occupant may be played at an integrated speaker 114 associated with and / or proximal to a seat of one or more other occupants of vehicle 100.

[0020] The vehicle 100 may further include a driver seat sensor 124 coupled to or within the driver seat 104 and a passenger seat sensor 126 coupled to or within the first passenger seat 106. The back seats may also include seat sensors, such as a passenger seat sensor 128 coupled to the second passenger seat 108 and a passenger seat sensor 130 coupled to the third passenger seat 110. The driver seat sensor 124 and the passenger seat sensor 126 may each include one or a plurality of sensors, such as a weight sensor, a pressure sensor, and one or more seat position sensors that output a measurement signal to the computing system 120. For example, the output of the weight sensor or pressure sensor may be used by the computing system 120 to determine whether or not the respective seat is occupied, and if occupied, a weight of a person occupying the seat (e.g., to estimate an age of an occupant). As another example, the output of the one or more seat position sensors may be used by the computing system 120 to determine one or more of an occupant of vehicle 100, a seat height, a longitudinal position with respect to the dashboard 102 and the back seats, and an angle (e.g., tilt) of a seat back of a corresponding seat.

[0021] Computing system 120 may include a user interface (UI) 116. Computing system 120 may receive inputs via UI 116 as well as output information to UI 116. The user interface 116 may be included in a digital cockpit, for example, and may include a display and one or more input devices. The one or more input devices may include one or more touchscreens, knobs, dials, hard buttons, and soft buttons for receiving user input from a vehicle occupant. UI 116 may include a display screen on which information, images, videos, and the like may be displayed. The display screen may be a touchscreen, and an occupant of vehicle 100 may interact with UI 116 via control elements displayed on the display screen.

[0022] Computing system 120 may also include an in-vehicle infotainment (IVI) system 115, which may be configured to display video content and / or receive commands for selecting and interacting with the video content on UI 116. The video content may be displayed in coordination with audio content outputted by an audio system, which may transmit audio streams to one or more audio devices and / or speakers located within the vehicle, such as speakers 114.

[0023] Computing system 120 may include one or more additional UIs 117, which may be positioned at locations accessible to back seat passengers of vehicle 100. In various embodiments, the one or more additional UIs 117 may comprise or be comprised by a rear seat entertainment (RSE) system. In other words, a first UI 117 may be positioned on a rear side ofAttorney Docket No. P240063WO front seat 106, such that a first rear passenger sitting in rear seat 108 may interact with the first UI 117. A second UI 117 may be positioned on a rear side of driver’s seat 104, such that a second rear passenger sitting in rear seat 110 may interact with the second UI 117. In vehicles including additional rear seats, additional UIs 117 may be arranged similarly such that each occupant of vehicle 100 may interact with a UI 117.

[0024] Each additional UI 117 may include the same or similar features included in UI 116, such as physical and / or virtual control elements, one or more display screens, etc. Additionally, each additional UI 117 may be configured to receive the same or different video and / or audio content from computing system 120 and display the content independently of other UI 117s. For example, computing system 120 may generate a first set of visual content on a first display screen associated with first UI 117; a second, different set of visual content on a second display screen associated with second UI 117; a third, different set of visual content on a third display screen associated with third UI 117; and so on. Similarly, a first occupant of vehicle 100 may interact with a first UI 117 to generate a first set of visual content on a first display screen associated with first UI 117; a second occupant of vehicle 100 may interact with a second UI 117 to generate a second, different set of visual content on a second display screen associated with second UI 117; a third occupant of vehicle 100 may interact with a third UI 117 to generate a third, different set of visual content on a third display screen associated with third UI 117; and so on. The sets of visual content may include associated audio content, which may be played at a respective speaker 114.

[0025] Additionally or alternatively, audio and / or visual content may be played at personal computing devices of occupants of vehicle 100, such as a personal computing device 150 of an occupant of front passenger seat 106; a personal computing device 151 of an occupant of rear seat 108; and a personal computing device 152 of an occupant of rear seat 110. Personal computing devices 150-152 may include, for example, smart phones, tablets, laptop computers, and / or other types of computing devices.

[0026] Vehicle 100 may include a plurality of microphones 176 positioned at each seat of vehicle 100. For example, each microphone 176 may be positioned within a seat back of a respective seat, or on a ceiling or wall portion of the cabin proximate to the respective seat. Occupants sitting in seats 104, 106, 108, and 110 may communicate with computer 120 using verbal commands via a respective microphone 176.

[0027] Computing system 120 includes a processor 142 configured to execute machine readable instructions stored in a memory’ 144. The processor 142 may be single core or multicore, and the programs executed by processor 142 may be configured for parallel or distributedAttorney Docket No. P240063WO processing. In some embodiments, the processor 142 is a microcontroller. The processor 142 may optionally include individual components that are distributed throughout two or more devices, which may be remotely located and / or configured for coordinated processing. In some embodiments, one or more aspects of the processor 142 may be virtualized and executed by remotely-accessible networked computing devices configured in a cloud computing configuration. For example, the computing system 120 may be communicatively coupled with a wireless network.

[0028] Computing system 120 may communicate with personal computing devices 150, 151, and 152 via short-range communication protocols, such as Bluetooth®. In some embodiments, the computing system 120 may include other electronic components capable of carrying out processing functions, such as a digital signal processor, a field-programmable gate array (FPGA), or a graphic board. In some embodiments, the processor 142 may include multiple electronic components capable of carrying out processing functions. For example, the processor 142 may include two or more electronic components selected from a plurality of possible electronic components, including a central processor, a digital signal processor, a field-programmable gate array, and a graphics board. In still further embodiments, the processor 142 may be configured as a graphical processing unit (GPU), including parallel computing architecture and parallel processing capabilities.

[0029] Further, the memory 144 may include any non- transitory' tangible computer readable medium in which programming instructions are stored. As used herein, the term “tangible computer readable medium” is expressly defined to include any type of computer readable storage. The example methods described herein may be implemented using coded instruction (e.g., computer readable instructions) stored on a non-transitory computer readable medium such as a flash memory, a read-only memory (ROM), a random-access memory (RAM), a cache, or any other storage media in which information is stored for any duration (e.g. for extended period time periods, permanently, brief instances, for temporarily buffering, and / or for caching of the information).

[0030] Computer memory of computer readable storage mediums as referenced herein may include volatile and non-volatile or removable and non-removable media for a storage of electronically formatted information, such as computer readable program instructions or modules of computer readable program instructions, data, etc. that may be stand-alone or as part of a computing device. Examples of computer memory may include any other medium which can be used to store the desired electronic format of information and which can be accessed by the processor or processors or at least a portion of a computing device. In variousAttorney Docket No. P240063WO embodiments, the memory 144 may include an SD memory card, an internal and / or external hard disk, USB memory device, or a similar modular memory’.

[0031] Computing system 120 further includes a driver monitoring system 148 and an occupant monitoring system 149, which may monitor the driver and occupants of the vehicle, respectively. DMS 148 may monitor the driver via a DMS camera 181, and OMS 149 may monitor the occupants of the vehicle via a plurality’ of OMS cameras 180 positioned in front of a respective passenger seat of the vehicle. For example, the OMS cameras may be positioned in a ceiling of the cabin, or in a seat positioned in front of the respective passenger. In particular, the OMS cameras 180 may detect whether an occupant is using a personal computing device 150, 151, or 152, and may detect gestures made by the occupant intended to control content played at the personal computing device, as described in greater detail below. In some examples, each OMS camera 180 may be paired with a time-of-flight (TOF) camera 182. The time-of-flight camera has the advantage that it provides depth information natively and can therefore increase the availability’ and precision of gesture recognition. Additionally or alternatively, in some examples, the OMS camera may be supported by an in-cabin radar 183, which may be capable of capturing movements with greater precision than OMS camera 180 and TOF camera 182.

[0032] In some examples, the computing system 120 may include a plurality’ of subsystems or modules tasks with performing specific functions related to performing image acquisition and analysis. As used herein, the terms "‘system,” “unit.” or “module” may include a hardware and / or software system that operates to perform one or more functions. For example, a module, unit, or system may include a computer processor, controller, or other logic-based device that performs operations based on instructions stored on a tangible and non- transitory computer readable storage medium, such as a computer memory. Alternatively, a module, unit, or system may include a hard-wired device that performs operations based on hard-wired logic of the device. Various modules or units shown in the attached figures may represent the hardware that operates based on software or hardwired instructions, the software that directs hardware to perform the operations, or a combination thereof.

[0033] Systems and interfaces of vehicle 100 are described in further detail below in reference to FIGS. 6 and 7.

[0034] Referring now to FIG. 2, a schematic diagram of an exemplary’ gestural interface system 200 of a vehicle is shown, which may rely on components of vehicle 100 of FIG. 1. Gestural interface system 200 comprises a controller 202, which may be included in computing system 120; a microphone 212, which may be anon-limiting example of microphones 176; anAttorney Docket No. P240063WOOMS camera 214, which may be a non-limiting example of cameras 180; a TOF camera 216, which may be a non-limiting example of TOF cameras 182; an in-cabin radar 218, which may be a non-limiting example of in-cabin radar 183; and one or more personal computing devices 220 (e.g., personal computing devices 150, 151, 152) of a respective one or more users 240, where users 240 are occupants of the vehicle. Controller 202 includes a processor 204, a memory 206, a device interface manager 208, a communication module 210, and a gesture detection model 211. Personal computing devices 220 include a processor 221, a UI 222, and a memory 223 which may store one or more apps 224, and a gestural interface service 226, described below.

[0035] As described herein, a memory (such as memory 206 and memory 223) may include one or more data storage structures, such as optical memory devices, magnetic memory devices, or solid-state memory devices, for storing programs and routines executed by a processor (e.g., processor 204) to carry out various functionalities disclosed herein. Memory may include any desired type of volatile and / or non-volatile memory such as, for example, static random access memory’ (SRAM), dynamic random access memory (DRAM), flash memory, read-only memory (ROM), etc. Processor 204, as well as other processors described herein, may be any' suitable processor, processing unit, or microprocessor, or a multi-processor system including one or more additional processors that are identical or similar to each other and that are communicatively coupled via an interconnection bus.

[0036] Communication module 210 may facilitate communication between controller 202, other wired and wireless components of the vehicle, and personal computing devices 220. Communication via communication module 210 may be implemented using one or more protocols. Communication module 210 may include a wired interface (e.g., a data bus, a Universal Serial Bus (USB) connection, etc.) and / or a wireless interface (e.g.. radio frequency, infrared, near field communication (NFC), etc.). For example, the communication module may communicate via wired local area network (LAN), wireless LAN, wide area network (WAN), via radio frequencies (RF), etc. using any past, present, or future communication protocol (e.g., BLUETOOTH™, USB 2.0, USB 3.0, etc ).

[0037] When users 240 use personal computing devices 220 in the vehicle, physical gestures performed by users 240 in the air (e.g., meaning, physical gestures that do not involve touching the personal computing devices 220) may be captured by one or more of OMS camera 214, TOF camera 216, and in-cabin radar 218. Therefore, OMS camera 214, TOF camera 216, and in-cabin radar 218 may be advantageously used to provide additional user interface options to a user 240 with respect to an application (app) 224 being used by the user 240 on a respectiveAttorney Docket No. P240063WO personal computing device 220. That is, rather than interacting with the app 224 via UI 222 of the personal computing device 220, the user 240 may optionally interact with the app 224 using physical gestures performed in the air, which are captured by OMS camera 214, TOF camera 216, and / or in-cabin radar 218. In some examples, the user 240 may also interact with the app 224 via voice commands, which may be captured by microphone 212.

[0038] Images of the captured physical gestures may be transmitted from OMS camera 214, TOF camera 216. and / or in-cabin radar 218 to device interface manager 208, and audio input from the user 240 captured at microphone 212 may be transmitted to device interface manager 208 of controller 202. Device interface manager 208 may process the captured physical gestures and audio input to determine whether the physical gestures and audio input relate to controlling UI 222 or app 224. For such purpose, device interface manager 208 may communicate with a gestural interface service 226 installed on the personal computing device 220.

[0039] Device interface manager 208 may request and / or retrieve a set of gestural controls 228 associated with UI 222 or the app 224 via a connection with personal computing device 220 made using communication module 210. Different gestural controls 228 may be predefined by the user 240 for different controls of UI 222 and / or each app 224, and may comprise a mapping of various gestures to controls of UI 222 which may be specific to an app 224. For example, each mapping may include a control of UI 222 specific to the app 224 (e.g., an action performed within the app 224), and a sequence of images or video of a corresponding gesture that when performed by the user 240, causes the action to be performed within the app 224. The sequence of images or video of the corresponding gesture may be captured from the perspective of OMS camera 214 and / or TOF camera 216.

[0040] Device interface manager 208 may determine whether the captured physical gestures and / or audio input match the predefined gestural controls 228 for UI 222 and / or app 224. For such purpose, device interface manager 208 may rely on gesture detection model 211, where images, video, or data acquired via OMS camera 214, TOF camera 216, and / or in-cabin radar 218 may be inputted into gesture detection model 211, and gesture detection model 211 may output a determination of whether a gesture matching a sequence of images, video, or data of a predefined gestural control 228 was detected. For example, gesture detection model 211 may output an encoding of a gestural control 228 detected in the images, video, or point cloud data, where the encoding indicates an action to take in UI 222 or the app 224. In various embodiments, gesture detection model 211 may be a trained machine learning (ML) model trained to detect the gestures using a gradient descent algorithm, in accordance with variousAttorney Docket No. P240063WO techniques known in the art. For gesture detection and classification, various network architectures may be used. Long Short-Term Memory (LSTM), Gated Recurrent Units (GRUs) or Recurrent Neural Networks (RNNs) are likely candidates, as a non-limiting list. Vision transformers (ViT) and newer techniques are equally feasible. For example, an input layer into gesture detection model 211 may be a representation of the image(s) or point cloud stream, and an output may be a previously learned gesture.

[0041] If a physical gesture captured by OMS camera 214, TOF camera 216, and / or in- cabin radar 218 matches a gestural control 228, device interface manager 208 may notify personal computing device 220 via communication module 210. Personal computing device 220 may control UI 222 based on the notification, where an action may be performed in UI 222 and / or the app 224 in accordance with the physical gesture made by the user 240.

[0042] FIG. 3 shows a high-level schematic diagram 300 summarizing a process that may be carried out by device interface manager 208 to determine whether a captured physical gesture matches a predefined gestural control 228. A device association task 302 may first be performed to determine whether personal computing devices of one or more occupants of the vehicle have been paired to vehicle systems. Personal computing devices in use may then be mapped to seats of the vehicle in a device / occupant mapping task 304. In various examples, users of the personal computing devices may be prompted to indicate a seat in which a user is sitting.

[0043] Once the devices are associated with seats of the vehicle, in a video / audio acquisition task 306, images of a user 240 are captured by an OMS camera 214, TOF camera 216, and / or in-cabin radar 218 associated with a seat of the user 240, and audio input may be captured by a microphone 212 associated with the seat of the user 240. A gesture detection task 308 may then be performed to detect specific gestures relevant to the users interaction with the app 224, as defined in the predefined gestural controls 228. If a gesture defined in the predefined gestural controls 228 is detected, a gesture classification task 310 may be performed to classify the gesture to a specific user control. The classification may be performed using a set of mappings provided by the user. The specific user control may then sent to the personal computing device 220 in a personal computing device notification task 312.

[0044] For example, an occupant of a vehicle (e.g., vehicle 100) in a rear passenger seat of a vehicle may wish to read an article on a personal tablet. The occupant may position the tablet on the back of seat in front of the occupant for comfortable reading. However, manually selecting controls of the tablet to turn pages of the article may entail reaching forward and touching or swiping a specific location of a screen of the tablet, which may be cumbersome,Attorney Docket No. P240063WO tiring, or difficult as a result of a driving style of the driver or road environment. The occupant may prefer to turn pages by making a hand-waving gesture in the air in front of the tablet, which may be performed more quickly and easily than interacting with the tablet. The occupant may enable on the tablet a gestural interface service that may communicate with the device interface manager of the vehicle. Via a UI of the gestural interface service, the occupant may define the hand-waving gesture to indicate a page turn. The gestural interface service may communicate the association of the hand-waving gesture with the page turn to the device interface manager of the vehicle. When the hand-waving gesture is detected at an OMS camera, TOF camera, or in-cabin radar positioned at a seat of the occupant, the device interface manager may map the hand-waving gesture to the page turn, and may notify the gestural interface service that a page turn action should be performed on the tablet. In this way, a wider variety of user interface options may be provided to users of personal devices in vehicles.

[0045] FIG. 4 shows a high-level method for detecting physical gestures or verbal commands of an occupant of a vehicle, such as vehicle 100 of FIG. 1, that are mapped to controls of an interface of a personal computing device of the occupant, such that the occupant may interact with the personal computing device via physical gestures and verbal controls rather than using controls included on a display screen of the personal computing device. As described herein, physical gestures refer to physical gestures performed in the air and not involving touching the personal computing device, such as, waving a hand, pointing or moving a finger, swiping a hand or finger in a direction, etc. Method 400 may be carried out by a processor of a computing system or controller of the vehicle, such as processor 142 or 204, based on instructions stored in a memory of the vehicle such as memory 144 or 206, respectively.

[0046] Method 400 begins at 402, where method 400 includes measuring / estimating operating conditions of the vehicle. Measuring / estimating the vehicle operating conditions may include determining whether the vehicle is in operation, is moving, or in a stopped position. Measuring / estimating the vehicle operating conditions may also include determining a number and location of one or more occupants of the vehicle.

[0047] At 404, method 400 includes detecting one or more personal computing devices (e.g., personal computing devices 220) of the occupants of the vehicle. The personal computing devices comprise smart phones, tablets, and / or any other kind of personal computing devices that are capable of communicating wirelessly with vehicles systems, via RF signals (e.g., Bluetooth®) or a different type of wireless framework or network. Each occupant may have one or more personal computing devices, which the occupant may use at a seat of the vehicle.Attorney Docket No. P240063WOIn various embodiments, the personal computing devices may be detected when they are paired with the vehicle, via a conventional pairing process. When the personal computing devices are detected, wireless connections may be created between the controller and the personal computing devices, via the RF signals or via a different wireless network.

[0048] At 406, method 400 includes determining whether a gestural interface service is enabled on the personal computing devices detected within the vehicle. The gestural interface service may be an app that is installed in a personal computing device. To determine whether the gestural interface service is enabled on the personal computing devices, a device interface manager (e.g., device interface manager 208 of FIG. 2) may attempt to communicate with the gestural interface service. If no response is received from one or more personal computing devices, it may be inferred that the gestural interface service is not enabled on the one or more personal computing devices, whereby method 400 proceeds to 408. At 408, method 400 includes sending a notification to prompt users of the one or more personal computing devices to enable the gestural interface service. For example, a link or user interface control may be provided to enable the gestural interface service.

[0049] If at 406 responses are received from the personal computing devices detected within the vehicle, it may be inferred that the gestural interface service is enabled on the personal computing devices. If the gestural interface service is enabled on the personal computing devices, method 400 proceeds to 410. At 410, method 400 includes sending a notification to users of the personal computing devices, via the gestural interface service, prompting the users to identify seat locations of the users. For example, the notification sent to each personal computing device user may include a list of seats of the vehicle, or an image of seats of the vehicle, where the user may select the seat of the user in the list or image. In this way, the user may communicate to the vehicle a location of the user within the vehicle.

[0050] In some embodiments, the device association / localization may be performed automatically, or a suggestion may be provided for the users to select. For example, in one embodiment, w hen a user enters vehicle, a device of the user may be detected using Bluetooth, the OMS camera, ToF camera, and / or in-cabin radar, and localization may be performed or estimated using Bluetooth. WiFi-localization / ranging, and. depending on a number of antennas, triangulation techniques.In another embodiment, a new user in the vehicle may be identified using a FacelD, and a new device detected within the vehicle at the time of identification may be assigned to the user.Attorney Docket No. P240063WOUsers may be tracked while in the vehicle, such that the location of the user may always be known.

[0051] At 412, method 400 includes receiving OMS / TOF camera, in-cabin radar, and / or microphone output at the received seat locations. That is, for each occupant having a personal computing device, images and / or video of the occupant may be acquired via an OMS camera (e.g., cameras 180), a TOF camera (e.g., TOF cameras 182), and / or in-cabin radar (e.g., incabin radar 183) positioned at the seat of the occupant, and audio input may be acquired via a microphone positioned at the seat of the occupant (e.g., microphones 176).

[0052] At 414, method 400 includes determining, from the acquired images, whether a user at one or more of the indicated seats is using a personal computing device. For example, some users may not use their personal computing devices while in the vehicle, while other users may be using their personal computing devices while in the vehicle. In various embodiments, the device interface manager may rely on an ML model, such as an image classification model trained on a plurality of images of users, in which some of the images show the users using devices. The ML model may take images of a user as input, and output an indication of whether the user is using a personal computing device.

[0053] If at 414 at least one device in use is detected from the acquired images, method 400 proceeds to 418. If at a respective seat no device is detected as being used by the user, method 400 proceeds to 416. where method 400 includes monitoring the images or point cloud data acquired at the respective seat for device usage.

[0054] At 418, method 400 includes receiving a set of gestural controls (e.g., gestural controls 228) for interacting with a UI of the personal computing device (e.g., UI 222) and / or an application (e.g., an app) of the personal computing device being used by the user, from the gestural interface service running on the personal computing device of the user. In some embodiments, the device interface manager may request the set of gestural controls from the gestural interface service. In other embodiments, the gestural interface sendee may transmit the gestural controls to the device interface manager in response to the personal computing device being paired with the vehicle when the personal computing device and / or application is launched or active. The gestural controls may include mappings of a set of physical gestures to specific actions performed in the UI and / or the application. The set of physical gestures may include physical gestures on which a gesture detection model (e.g., gesture detection model 211) has been trained, where the gestural detection model can detect the physical gestures in images of the user and / or disambiguate the physical gestures from other physical gestures that are not gestural controls. The gestural controls may also include a mapping of verbal commandsAttorney Docket No. P240063WO to specific actions performed in the UI or application. The gestural controls may be stored in a file that is created in advance by the user and transmitted from the personal computing device to the device interface manager when the user is using the personal computing device or application. The creation of the gestural controls is described in greater detail below in reference to FIG. 5.

[0055] At 420, method 400 includes determining whether a gestural control (e.g., a physical gesture of the user on which the gesture detection model has been trained that is mapped to an action in the UI or application) is detected in the images received from the OMS / TOF camera / in-cabin radar at the seat of the user where the device is being used. The device interface manager may determine whether the gestural control is detected using the gestural detection model. For example, the gestural detection model may output a first encoding if a first physical gesture is detected; a second encoding if a second physical gesture is detected; and so on.

[0056] If at 420 a gestural control is not detected in the images, method 400 proceeds to 422, where method 400 includes monitoring the OMS / TOF camera / in-cabin radar output for gestural controls. Alternatively, if a gestural control is detected in the images, method 400 proceeds to 424.

[0057] At 424, method 400 includes sending a notification of the detected gestural control to the gestural interface service of the device of the user. In various embodiments, a detected physical gesture captured by the OMS / TOF camera / in-cabin radar may be mapped to an action associated with the physical gesture in the predefined gestural control file received by the device interface manager, and the action may be notified to the device. In other embodiments, the mapping of the gestural control to the action may be performed at the personal computing device. Method 400 ends.

[0058] Referring now to FIG. 5, a method 500 is shown for a personal computing device or personal computing device (e.g., personal computing devices 150, 151, and 152 of FIG. 1 and / or personal computing devices 220 of FIG. 2) used in a vehicle, for managing an interaction between the personal computing device and a device interface manager (e.g., device interface manager 208 of FIG. 2) of a controller of the vehicle that detects and maps physical gestures or verbal commands made by a user of the personal computing device to controls of an interface of the personal computing device for controlling an application running on the personal computing device. Method 400 may be carried out by a processor of the personal computing device. One or more steps of method 400 may be performed by a gestural interface service (e.g., gestural interface service 226) running at the personal computing device.Attorney Docket No. P240063WO

[0059] Method 500 begins at 502, where method 500 includes determining whether the personal computing device is paired with a vehicle (e.g., vehicle 100 of FIG. 1). where a wireless connection is created between the personal computing device and the controller. If at 502 it is determined that the personal computing device is not paired with a vehicle, method 500 proceeds to 504. At 504, method 500 includes waiting until the personal computing device is paired with a vehicle, and method 500 proceeds back to 502.

[0060] Alternatively if at 502 it is determined that the personal computing device is paired with a vehicle, method 500 proceeds to 506. 506, method 500 includes determining whether a user input has been received to enable the gestural interface service. That is, when the personal computing device is paired with the vehicle, the device interface manager at the vehicle may query the personal computing device via the wireless connection to determine whether the gestural interface service is enabled at the personal computing device, as described above in reference to method 400. If the gestural interface service is not enabled at the personal computing device, the device interface manager may send a notification to the personal computing device prompting the user to enable the gestural interface service.

[0061] If at 506 the user input is received to enable the gestural interface service, method 500 proceeds to 508. At 508, method 500 includes enabling the gestural interface service. Alternatively, if at 506 no user input is received to enable the gestural interface service, it may be inferred that the gestural interface service is enabled at the personal computing device, whereby method 500 proceeds to 510. At 510, method 500 includes starting the gestural interface service at the personal computing device.

[0062] At 512, at 500 includes receiving a user input of a seat selection from the gestural interface service. In other words, when the gestural interface service is started, the device interface manager at the vehicle may communicate with the gestural interface service via the wireless connection to determine a seat in which the user of the personal computing device is sitting. The gestural interface service may prompt the user to select the seat, for example, from a list of seats of the vehicle or in an image of the seats of the vehicle. As described above, the device interface manager may use the seat selection to identify an OMS / TOF camera / in-cabin radar associated with the seat. In some embodiments, UWB may also be used to identify the OMS / TOF camera / in-cabin radar. At 514, method 500 includes transmitting the seat selection from the gestural interface service to the device interface manager of the vehicle, via the wireless connection.

[0063] At 516, method 500 includes determining whether a gestural control mapping has been defined for the personal computing device or an application being used by the user on theAttorney Docket No. P240063WO personal computing device. As described above, the gestural control mapping may include mappings of one or more physical gestures of the user to actions performed in a UI of the personal computing device and / or the application. For example, a first physical gesture of the user waving a hand to the right may be mapped to a first action performed in the application; a second physical gesture of the user waving a hand to the left may be mapped to a second action performed in the application; and so on.

[0064] If at 516 it is determined that a gestural control mapping has been defined for the UI or application being used, method 500 proceeds to 520. At 520, method 520 includes transmitting the gestural control mapping to the device interface manager from the gestural interface sendee. Alternatively, if at 516 it is determined that a gestural control mapping has not been defined for the UI or application being used, method 500 proceeds to 518.

[0065] At 518, method 500 includes displaying, in the UI, a set of control elements for defining one or more gestural controls for the UI or application. In various embodiments, a plurality of physical gestures, such as hand gestures, may be specified or listed in the UI (e.g., a library of supported physical gestures), and the user may associate one or more physical gestures of the plurality of physical gestures with one or more actions to be performed in the UI or application using the control elements (e.g., common selection tools such as buttons, checkboxes, etc.) The plurality7of physical gestures may include physical gestures used to train a gestural detection model used to detect the phy sical gestures in images captured by an incabin sensing device (e.g.. an in-cabin camera such as OMS camera 181, a TOF camera such as TOF camera 182, and / or an in-cabin radar such as in-cabin radar 183). It should be appreciated that as used herein, in-cabin sensing device and in-cabin camera may be used interchangeably, and references to an in-cabin camera may include in-cabin radar.

[0066] At 522, method 500 includes receiving the gestural controls from the user, and storing the gestural controls in a memory of the personal computing device. Method 500 then proceeds to 520 where the gestural control mapping is transmitted to the device interface manager.

[0067] At 524, method 500 includes determining whether a notification of a detection of a gestural control has been received from the device interface manager. If at 524. no notification of a detection of a gestural control is received from the device interface manager, method 500 proceeds to 526, where method 500 includes waiting until such notification is received. Alternatively, if at 524 the detection of the gestural control is received from the device interface manager, method 500 proceeds to 528. At 528, method 500 includes performing the action that is mapped to the gestural control in the application, and method 500 ends.Attorney Docket No. P240063WO

[0068] FIG. 6 shows an interior of a cabin 600 of a vehicle 602, which may be a nonlimiting example of vehicle 100 of FIG. 1, in which a driver and / or one or more passengers may be seated. Vehicle 602 includes an in-vehicle computing system 609 (e.g., computing system 120 of FIG. 1). The in-vehicle computing system 609 may include an autonomous vehicle control system for at least partially controlling vehicle systems during autonomous driving. As an example, while operating in an autonomous mode, the autonomous vehicle control system may monitor vehicle surroundings via a plurality of sensors (e.g., such as cameras, radars, ultrasonic sensors, a GPS signal, and the like). The in-vehicle computing system 609 is described in greater detail below in reference to FIG. 7.

[0069] As shown, an instrument panel 606 may include various displays and controls accessible to a human user (e.g., a driver or a passenger) of vehicle 602. For example, instrument panel 606 may include a touch screen 608 of an in-vehicle computing system or infotainment system 609 (e.g., an infotainment system), an audio system control panel, and an instrument cluster 610. Touch screen 608 may receive user input to the in-vehicle computing system or infotainment system 609 for controlling audio output, visual display output, user preferences, control parameter selection, and so on. In some examples, instrument panel 606 may include one or more controls for driver assistance programs, such as a cruise control system, a collision avoidance system, and the like. Further, additional user interfaces, not shown, may be present in other portions of the vehicle, such as proximate to at least one passenger seat. For example, the vehicle may include a row of back seats with at least one touch screen controlling the in-vehicle computing system 609.

[0070] While the example system shown in FIG. 6 includes audio system controls that may be performed via a user interface of in-vehicle computing system or infotainment system 609, such as touch screen 608 without a separate audio system control panel, in other embodiments, the vehicle may include an audio system control panel, which may include controls for a conventional vehicle audio system such as a radio, compact disc player, MP3 player, and so on. The audio system controls may include features for controlling one or more aspects of audio output via one or more speakers 612 of a vehicle speaker system. For example, the in-vehicle computing system or the audio system controls may control a volume of audio output, a distribution of sound among the individual speakers of the vehicle speaker system, an equalization of audio signals, and / or any other aspect of the audio output. In further examples, in-vehicle computing system or infotainment system 609 may adjust a radio station selection, a playlist selection, a source of audio input (e.g., from radio or CD or MP3), and so on, based on user input received directly via touch screen 608, or based on data regarding the user (suchAttorney Docket No. P240063WO as a physical state and / or environment of the user) received via one or more external devices 650 and / or a mobile device 628. The audio system of the vehicle may include an amplifier (not shown) coupled to plurality of loudspeakers (not shown). In some embodiments, one or more hardware elements of in-vehicle computing system or infotainment system 609, such as touch screen 608, display screen 611, various control dials, knobs and buttons, memory, processor(s), and any interface elements (e.g.. connectors or ports) may form an integrated head unit that is installed in instrument panel 606 of the vehicle. The head unit may be fixedly or removably attached in instrument panel 606. In additional or alternative embodiments, one or more hardware elements of the in-vehicle computing system or infotainment system 609 may be modular and may be installed in multiple locations of the vehicle.

[0071] Cabin 600 may also include one or more user objects, such as mobile device 628, that are stored in the vehicle before, during, and / or after travelling. The mobile device 628 may include a smart phone, a tablet, a laptop computer, a portable media player, and / or any suitable mobile computing device. The mobile device 628 may be connected to the in-vehicle computing system via a communication link 630. The communication link 630 may be wired (e.g.. via Universal Serial Bus (USB). Mobile High-Definition Link (MHL). High-Definition Multimedia Interface (HDMI), Ethernet, and so on) or wireless (e.g., via Bluetooth®, Wi-Fi®, Wi-Fi Direct®, Near-Field Communication (NFC), cellular connectivity7, and so on) and configured to provide two-way communication between the mobile device and the in-vehicle computing system. (Bluetooth® is a registered trademark of Bluetooth SIG, Inc., Kirkland, WA. Wi-Fi® and Wi-Fi Direct® are registered trademarks of Wi-Fi Alliance, Austin, Texas.) The mobile device 628 may include one or more wireless communication interfaces for connecting to one or more communication links (e.g., one or more of the example communication links described above). The wireless communication interface may include one or more physical devices, such as antenna(s) or port(s) coupled to data lines for carrying transmitted or received data, as well as one or more modules / drivers for operating the physical devices in accordance with other devices in the mobile device. For example, the communication link 630 may provide sensor and / or control signals from various vehicle systems (such as vehicle audio system, chmate control system, and so on) and the touch screen 608 to the mobile device 628 and may provide control and / or display signals from the mobile device 628 to the in-vehicle systems and the touch screen 608. The communication link 630 may also provide power to the mobile device 628 from an in-vehicle power source in order to charge an internal battery of the mobile device.Attorney Docket No. P240063WO

[0072] In-vehicle computing system or infotainment system 609 may also be communicatively coupled to additional devices operated and / or accessed by the user but located external to vehicle 602, such as one or more external devices 650. In the depicted embodiment, external devices are located outside of vehicle 602 though it will be appreciated that in alternate embodiments, external devices may be located inside cabin 600. The external devices may include a server computing system, personal computing system, portable electronic device, electronic wrist band, electronic head band, portable music player, electronic activity tracking device, pedometer, smart-watch, GPS system, and so on. External devices 650 may be connected to the in-vehicle computing system via a communication link 636 which may be wired or wireless, as discussed with reference to communication link 630, and configured to provide two-way communication between the external devices and the in-vehicle computing system. For example, external devices 650 may include one or more sensors and communication link 636 may transmit sensor output from external devices 650 to in-vehicle computing sy stem or infotainment system 609 and touch screen 608. External devices 650 may also store and / or receive information regarding contextual data, user behavior / preferences, operating rules, and so on. and may transmit such information from the external devices 650 to in-vehicle computing system or infotainment system 609 and touch screen 608.

[0073] In-vehicle computing system or infotainment system 609 may analyze the input received from external devices 650, mobile device 628, and / or other input sources and select settings for various in-vehicle systems (such as climate control system or audio system), provide output via touch screen 608 and / or speakers 612, communicate with mobile device 628 and / or external devices 650, and / or perform other actions based on the assessment. In some embodiments, all or a portion of the assessment may be performed by the mobile device 628 and / or the external devices 650.

[0074] In some embodiments, one or more of the external devices 650 may be communicatively coupled to in-vehicle computing system or infotainment system 609 indirectly, via mobile device 628 and / or another of the external devices 650. For example, communication link 636 may communicatively couple external devices 650 to mobile device 628 such that output from external devices 650 is relayed to mobile device 628. Data received from external devices 650 may then be aggregated at mobile device 628 with data collected by mobile device 628, the aggregated data then transmitted to in-vehicle computing system or infotainment system 609 and touch screen 608 via communication link 630. Similar data aggregation may occur at a server system and then transmitted to in-vehicle computing systemAttorney Docket No. P240063WO or infotainment system 609 and touch screen 608 via communication link 636 and / or communication link 630.

[0075] FIG. 7 shows a block diagram of an in-vehicle computing system or infotainment system 609 configured and / or integrated inside vehicle 602. In-vehicle computing system or infotainment system 609 may perform one or more of the methods described herein in some embodiments. In some examples, the in-vehicle computing system or infotainment system 609 may be a vehicle infotainment system configured to provide information-based media content (audio and / or visual media content, including entertainment content, navigational services, and so on) to a vehicle user to enhance the operator’s in-vehicle experience. The in-vehicle computing system or infotainment system 609 may include, or be coupled to, various vehicle systems, sub-systems, hardware components, as well as software applications and systems that are integrated in, or integratable into, vehicle 602 in order to enhance an in-vehicle experience for a driver and / or a passenger. Further, the in-vehicle computing system may be coupled to systems for providing autonomous vehicle control.

[0076] In-vehicle computing system or infotainment system 609 may include one or more processors including an operating system processor 714 and an interface processor 720. Operating system processor 714 may execute an operating system on the in-vehicle computing system, and control input / output, display, playback, and other operations of the in-vehicle computing system. Interface processor 720 may interface with a vehicle control system 730 via an inter-vehicle system communication module 722.

[0077] Inter-vehicle system communication module 722 may output data to one or more other vehicle systems 731 and / or one or more other vehicle control elements 761, while also receiving data input from other vehicle systems 731 and other vehicle control elements 761, e.g. by way of vehicle control system 730. When outputting data, inter-vehicle system communication module 722 may provide a signal via a bus corresponding to any status of the vehicle, the vehicle surroundings, or the output of any other information source connected to the vehicle. Vehicle data outputs may include, for example, analog signals (such as current velocity), digital signals provided by individual information sources (such as clocks, thermometers, location sensors such as GPS sensors, and so on), digital signals propagated through vehicle data networks (such as an engine CAN bus through which engine related information may be communicated, a climate control CAN bus through which climate control related information may be communicated, and a multimedia data network through which multimedia data is communicated between multimedia components in the vehicle). For example, vehicle data outputs may be output to vehicle control system 730, and vehicle controlAttorney Docket No. P240063WO system 730 may adjust vehicle control elements 761 based on the vehicle data outputs. As another example, the in-vehicle computing system or infotainment system 609 may retrieve from the engine CAN bus the current speed of the vehicle estimated by the wheel sensors, a power state of the vehicle via a batten and / or power distribution system of the vehicle, an ignition state of the vehicle, and so on. In addition, other interfacing means such as Ethernet may be used as well without departing from the scope of this disclosure.

[0078] A storage device 708 may be included in in-vehicle computing system or infotainment system 609 to store data such as instructions executable by operating system processor 714 and / or interface processor 720 in non-volatile form. The storage device 708 may store application data, including prerecorded sounds, video, and / or images, to enable the in- vehicle computing system or infotainment system 609 to run an application for connecting to a personal device of an occupant of the vehicle 602. The application may retrieve information gathered by vehicle systems / sensors, input devices (e.g., a user interface 718), data stored in one or more storage devices, such as a volatile memory 719A or a non-volatile memory 719B, devices in communication with the in-vehicle computing system (e.g., a mobile device connected via a Bluetooth® link), and so on. In-vehicle computing system or infotainment system 609 may further include a volatile memory 719A. V olatile memory 719A may be RAM. Non-transitory storage devices, such as and / or non-volatile memory 719B, may store instructions and / or code that, when executed by a processor (e.g., operating system processor 714 and / or interface processor 720). controls the in-vehicle computing system or infotainment system 609 to perform one or more of the actions described in the disclosure.

[0079] One or more microphones 702 (e.g., microphones 176) may be included in the in- vehicle computing system or infotainment system 609 to receive voice commands from a user, to measure ambient noise in the vehicle, to determine whether audio from speakers of the vehicle is tuned in accordance with an acoustic environment of the vehicle, and so on. A speech processing unit 704 may process voice commands, such as the voice commands received from the microphone 702. In some embodiments, in-vehicle computing system or infotainment system 609 may also be able to receive voice commands and sample ambient vehicle noise using a microphone included in an audio system 732 of the vehicle.

[0080] One or more additional sensors may be included in a sensor subsystem 710 of the in-vehicle computing system or infotainment system 609. For example, the sensor subsystem 710 may include a plurality of cameras 725. such as a rear view camera for assisting a user in parking the vehicle and / or other external cameras, radars, lidars, ultrasonic sensors, and the like. The sensor subsystem 710 may include one or more in-cabin cameras (e.g., DMS cameraAttorney Docket No. P240063WO181 and / or OMS cameras 180) for identifying a user (e.g., using facial recognition and / or user gestures). For example, an in-cabin camera may be used to identify one or more users of the vehicle via facial recognition software, and / or to detect whether a user is using a personal computing device. The in-cabin camera may also detect physical gestures of the user, which may be mapped to various functionalities of the personal computing device.

[0081] Sensor subsystem 710 of in-vehicle computing system or infotainment system 609 may communicate with and receive inputs from various vehicle sensors and may further receive user inputs. For example, the inputs received by sensor subsystem 710 may include transmission gear position, transmission clutch position, gas pedal input, brake input, transmission selector position, vehicle speed, engine speed, mass airflow through the engine, ambient temperature, intake air temperature, and so on, as well as inputs from climate control system sensors (such as heat transfer fluid temperature, antifreeze temperature, fan speed, passenger compartment temperature, desired passenger compartment temperature, ambient humidity, and so on), an audio sensor detecting voice commands issued by a user, a fob sensor receiving commands from and optionally tracking the geographic location / proximity of a fob of the vehicle, and so on.

[0082] One or more additional sensors may be included in and / or communicatively coupled to a sensor subsystem 710 of the in-vehicle computing system 609. For example, the sensor subsystem 710 may include and / or be communicatively coupled to a camera, such as a rear view camera for assisting a user in parking the vehicle, a cabin camera for identifying a user, and / or a front view' camera to assess quality of the route segment ahead.

[0083] While certain vehicle system sensors may communicate with sensor subsystem 710 alone, other sensors may communicate with both sensor subsystem 710 and vehicle control system 730, or may communicate with sensor subsystem 710 indirectly via vehicle control system 730. Sensor subsystem 710 may serve as an interface (e.g., ahardware interface) and / or processing unit for receiving and / or processing received signals from one or more of the sensors described in the disclosure.

[0084] A navigation subsystem 711 of in-vehicle computing system or infotainment system 609 may generate and / or receive navigation information such as location information (e.g.. via a GPS sensor and / or other sensors from sensor subsystem 710), route guidance, traffic information, point-of-interest (POI) identification, and / or provide other navigational services for the user. Navigation sub-system 711 may include inputs / outputs including analog to digital converters, digital inputs, digital outputs, network outputs, radio frequency transmittingAttorney Docket No. P240063WO devices, and so on. In some examples, navigation sub-system 711 may interface with vehicle control system 730.

[0085] An external device interface 712 of in-vehicle computing system or infotainment system 609 may be coupleable to and / or communicate with one or more external devices 650 located external to vehicle 602. While the external devices are illustrated as being located external to vehicle 602, it is to be understood that they may be temporarily housed in vehicle 602, such as when the user is operating the external devices while operating vehicle 602. In other words, the external devices 650 are not integral to vehicle 602. The external devices 650 may include a mobile device 628 (e.g., connected via a Bluetooth®, NFC, WI-FI Direct®, or other wireless connection) or an alternate Bluetooth®-enabled device 752.

[0086] Mobile device 628 may be a mobile phone, smart phone, wearable devices / sensors that may communicate with the in-vehicle computing system via wired and / or wireless communication, or other portable electronic device(s). Other external devices include one or more external services 746. For example, the external devices may include extra-vehicular devices that are separate from and located externally to the vehicle. Still other external devices include one or more external storage devices 754, such as solid-state drives, pen drives. USB drives, and so on. External devices 650 may communicate with in-vehicle computing system or infotainment system 609 either wirelessly or via connectors without departing from the scope of this disclosure. For example, external devices 650 may communicate with in-vehicle computing system or infotainment system 609 through the external device interface 712 over a network 760, a USB connection, a direct wired connection, a direct wireless connection, and / or other communication link.

[0087] The external device interface 712 may provide a communication interface to enable the in-vehicle computing system to communicate with mobile devices associated with contacts of the driver. For example, the external device interface 712 may enable phone calls to be established and / or text messages (e.g., Short Message Service (SMS), Multimedia Message Sendee (MMS), and so on) to be sent (e.g., via a cellular communications network) to a mobile device associated with a contact of the driver. The external device interface 712 may additionally or alternatively provide a wireless communication interface to enable the in- vehicle computing system to exchange and / or synchronize data with one or more devices in the vehicle (e.g., the driver’s mobile device) via Wi-Fi Direct®.

[0088] One or more applications 744 may be operable on mobile device 628. As an example, a mobile device application 744 may be operated to display video or audio content to a user of mobile device application 744. In addition, specific user data requests may be receivedAttorney Docket No. P240063WO at mobile device 628 from in-vehicle computing system or infotainment system 609 via the external device interface 712. The specific data requests may include requests for determining where the user is located within vehicle 602. Mobile device application 744 may send control instructions to components (e.g., microphone, amplifier, and so on) or other applications (e.g., navigational applications) of mobile device 628 to enable data to be collected on the mobile device or a requested adjustment made to the components. Mobile device application 744 may then relay the collected information back to in-vehicle computing system or infotainment system 609.

[0089] Likewise, one or more applications 748 may be operable on external services 746. As an example, external services applications 748 may be operated to aggregate and / or analyze data from multiple data sources. For example, external services applications 748 may aggregate data from one or more social media accounts of the user, data from the in-vehicle computing system (e.g., sensor data, log files, user input, and so on), data from an internet query (e.g., weather data, POI data), and so on. The collected data may be transmitted to another device and / or analyzed by the application to determine a context of the driver, vehicle, and environment and perform an action based on the context (e.g.. requesting / sending data to other devices).

[0090] Vehicle control system 730 may include controls for controlling aspects of various vehicle systems 731 involved in different in-vehicle functions. These may include, for example, controlling aspects of vehicle audio system 732 for providing audio entertainment to the vehicle occupants, aspects of a climate control system 734 for meeting the cabin cooling or heating needs of the vehicle occupants, as well as aspects of a telecommunication system 736 for enabling vehicle occupants to establish telecommunication linkage with others.

[0091] Audio system 732 may include one or more acoustic reproduction devices including electromagnetic transducers such as one or more speakers 735. Vehicle audio system 732 may be passive or active such as by including a power amplifier. In some examples, in-vehicle computing system or infotainment system 609 may be the only audio source for the acoustic reproduction device, or there may be other audio sources that are connected to the audio reproduction system (e.g.. external devices such as a mobile phone). The connection of any such external devices to the audio reproduction device may be analog, digital, or any combination of analog and digital technologies.

[0092] Climate control system 734 may be configured to provide a comfortable environment within the cabin or passenger compartment of vehicle 602. Climate control system 734 includes components enabling controlled ventilation such as air vents, a heater, an airAttorney Docket No. P240063WO conditioner, an integrated heater and air-conditioner system, and so on. Other components linked to the heating and air-conditioning setup may include a windshield defrosting and defogging system capable of clearing the windshield and a ventilation-air filter for cleaning outside air that enters the passenger compartment through a fresh-air inlet.

[0093] Vehicle control system 730 may also include controls for adjusting the settings of various vehicle control elements 761 (or vehicle controls, or vehicle system control elements) related to the engine and / or auxiliary elements within a cabin of the vehicle, such as one or more steering wheel controls 762 (e g., steering wheel-mounted audio system controls, cruise controls, windshield wiper controls, headlight controls, turn signal controls, and so on), instrument panel controls, microphone(s), accelerator / brake / clutch pedals, a gear shift, door / window controls positioned in a driver or passenger door, seat controls, cabin light controls, audio system controls, cabin temperature controls, and so on. Vehicle control elements 761 may also include internal engine and vehicle operation controls (e.g., engine controller module, actuator controls, valves, and so on) that are configured to receive instructions via the CAN bus of the vehicle to change operation of one or more of the engine, exhaust system, transmission, and / or other vehicle system. The control signals may also control audio output at one or more speakers 735 of the vehicle’s audio system 732. For example, the control signals may adjust audio output characteristics such as volume, equalization, audio image (e.g., the configuration of the audio signals to produce audio output that appears to a user to originate from one or more defined locations), audio distribution among a plurality of speakers, and so on. Likewise, the control signals may control vents, air conditioner, and / or heater of climate control system 734. For example, the control signals may increase delivery of cooled air to a specific section of the cabin. For example, the control signals may increase delivery of cooled air to a specific section of the cabin. Additionally, while operating in an autonomous mode, the autonomous vehicle control system may control some or all of the above vehicle controls.

[0094] Vehicle controls 761 may include a steering control system 762, a braking control system 763, and an acceleration control system 764. Vehicle controls 761 may include additional control systems. In some example, vehicle controls 761 may be operated autonomously, such as during autonomous vehicle operation. In other examples, vehicle controls 761 may be controlled by a user. Further, in some examples, a user may primarily control vehicle controls 761, while one or more ADAS 765 may intermittently adjust vehicle controls 761 in order to increase vehicle performance. For example, the one or more ADASAttorney Docket No. P240063WO765 may include a cruise control system, a lane departure warning system, a collision avoidance system, an adaptive braking system, and the like.

[0095] Steering control system 762 may be configured to control a direction of the vehicle. For example, during a non-autonomous mode of operation, steering control system 762 may be controlled by a steering wheel. For example, the user may turn the steering wheel in order to adjust a vehicle direction.

[0096] Braking control system 763 may be configured to control an amount of braking force applied to the vehicle. For example, during a non-autonomous mode of operation, braking control system 763 may be controlled by a brake pedal. For example, the user may depress the brake pedal in order to increase an amount of braking applied to the vehicle.

[0097] Acceleration control system 764 may be configured to control an amount of acceleration applied to the vehicle. For example, during a non-autonomous mode of operation, acceleration control system 764 may be controlled by an acceleration pedal. For example, the user may depress the acceleration pedal in order to increase an amount of torque applied to wheels of the vehicle, causing the vehicle to accelerate in speed.

[0098] Control elements positioned on an outside of a vehicle (e.g.. controls for a secunty system) may also be connected to in-vehicle computing system or infotainment system 609, such as via inter-vehicle system communication module 722. The control elements of vehicle control system may be physically and permanently positioned on and / or in the vehicle for receiving user input. In addition to receiving control instructions from in-vehicle computing system or infotainment system 609, vehicle control system 730 may also receive input from one or more external devices 650 operated by the user, such as from mobile device 628. This allows aspects of vehicle systems 731 and vehicle control elements 761 to be controlled based on user input received from the external devices 650.

[0099] In-vehicle computing system or infotainment system 609 may further include one or more antennas 706. The in-vehicle computing system may obtain broadband wireless internet access via antennas 706, and may further receive broadcast signals such as radio, television, weather, traffic, and the like. The in-vehicle computing system or infotainment system 609 may receive positioning signals such as GPS signals via antennas 706. The in- vehicle computing system may also receive wireless commands via radio frequency (RF such as via antennas 706 or via infrared or other means through appropriate receiving devices. In some embodiments, antenna 706 may be included as part of audio system 732 or telecommunication system 736. Additionally, antenna 706 may provide AM / FM radio signals to external devices 650 (such as to mobile device 628) via external device interface 712.Attorney Docket No. P240063WO

[0100] One or more elements of the in-vehicle computing system or infotainment system 609 may be controlled by a user via user interface 718. User interface 718 may include a graphical user interface presented on a touch screen, such as touch screen 608 and / or display screen 611 of FIG. 6, and / or user-actuated buttons, switches, knobs, dials, sliders, microphones, etc. For example, user-actuated elements may include steering wheel controls, door and / or window controls, instrument panel controls, audio system settings, climate control system settings, and the like. A user may also interact with one or more applications of the in-vehicle computing system or infotainment system 609 and mobile device 628 via user interface 718. In addition to receiving a user’s vehicle setting preferences on user interface 718, vehicle settings selected by in-vehicle control system may be displayed to a user on user interface 718. Notifications and other messages (e.g.. received messages), navigational assistance, advertisements, and / or other information may be displayed to the user on a display of the user interface. User preferences / information and / or responses to presented messages may be performed via user input to the user interface.

[0101] The in-vehicle computing system or infotainment system 609 may include a DMS or OMS 721. The DMS / OMS 721 may receive data from various sensors and / or systems of the vehicle (e g., sensor subsystem 710, cameras 725, microphone 702) and may monitor aspects of driver or passenger behavior to increase a performance of the vehicle and / or a driving experience of the driver or passenger. In some examples, one or more outputs of the DMS / OMS 721 may be inputs into a device interface manager 723. which may be a non-limiting example of device interface manager 208 of FIG. 2. The device interface manager may use images captured by DMS / OMS 721 to detect gestures of the driver or passenger, and map the gestures to actions performed on an external device 650, as described in reference to FIGS. 4 and 5.

[0102] Thus, via the systems and methods described herein, existing components of an incabin environment of a vehicle, such as an OMS including one or more cameras positioned to capture images of occupants of the vehicle, may be advantageously leveraged to provide additional options for interacting with a personal computing device while travelling in the vehicle. When an occupant of the vehicle is using a personal computing device in the vehicle, the occupant may optionally select to interact with the personal computing device via a gestural interface, where the occupant may perform physical gestures (e.g., hand gestures) in the air that are captured by the OMS system, and then mapped to functionalities of the personal computing device or actions supported by a UI of the personal computing device or an application running on the personal computing device. The gestural interface is maintained by a communication between a device interface manager of a controller of the vehicle and a gestural interfaceAttorney Docket No. P240063WO sen-ice running on the personal computing device. The device interface manager may rely on a trained ML model (e.g., a gestural detection model) to detect one of various supported physical gestures on which the ML model has been trained that are defined as a gestural control, and to disambiguate the physical gesture from other physical gestures that are not defined as gestural controls. For example, the gestural detection model may be trained to recognize a swipe of a hand of the user to the left, and the user may define a gestural control that maps the swipe of the hand to the left to a desired functionality of the personal computing device. In various examples, the user may define desired gestural controls from a library of a set of predefined physical gestures on which the gestural detection model is trained. The gestural controls may be used to perform actions in applications running on the personal computing device, or to perform other functionalities of the personal computing device.

[0103] For example, a child sitting in a rear seat of the vehicle may wish to play a first person shooter video game on a personal tablet. The child may position the tablet at a back side of a seat in front of the child. The child may open a UI of the gestural interface service on the personal tablet, and may indicate a seat of the child to the gestural interface service. The gestural interface service may display a library of different gestural control options for interacting with the personal tablet. Among the options, the child may see a “finger gun” gestural control including a short video clip taken by an OMS camera that illustrates a hand making a finger gun, where the gun is fired by moving a thumb of the hand. The child may select the finger gun gestural control, and may select a second control option for mapping the finger gun to an action of shooting a gun in the first person shooter video game. The gestural interface service may then send the finger gun gestural control to the device interface manager. The child may begin to play the first person shooter video game, and while the child is playing, the device interface manager may capture images of the child via an OMS camera positioned at the seat of the child. The images may be inputted into the trained gestural detection model, and the trained gestural detection model may detect gestural controls included in the library of gestural control options. The trained gestural detection model may detect the finger gun shooting gesture of the child as the child shoots a target in the video game. The device interface model may map the finger gun shooting gesture to the action of shooting the gun in the video game, based on the child’s defined gestural control. The device interface manager may notify the gestural interface service that the action of shooting the gun has been detected, and the gestural interface service may perform the action of shooting the gun in the video game. In this way, the child can play the video game in a more intuitive and natural manner than by selecting controls on a screen of the personal computing device.Attorney Docket No. P240063WO

[0104] The technical effect of using an in-cabin camera of a vehicle to capture images of an occupant of the vehicle using a personal computing device, detecting a physical gesture of the occupant in the images using a ML model, mapping the detected physical gesture to a functionality of the personal computing device or an action in a UI of the personal computing device, and performing the action in the personal computing device, is that the personal computing device may be controlled in a manner that is more efficient and intuitive than via the UI. and easier under driving conditions, but not possible using the limited capabilities of the personal computing device alone. That is, due to a lack of cameras and limited camera angles of the personal computing device, a gestural interface provided based on the functionalities of the personal computing device alone would be inaccurate and would rely on the user performing gestures within a narrow field of view of a camera provided on the device. By leveraging the cameras and / or in-cabin radar of the vehicle, a wider range of physical gestures may be detected and disambiguated, and the gestural controls may be more accurately identified. Additionally, an amount of memory7and processing power of the personal computing device may not be sufficient to support the gestural interface. The gesture detection model may consume more memory than is available at the personal computing device. By leveraging the additional processing power available at the vehicle for processing the images captured by the in-cabin camera, a specificity7of the gestural controls and a flexibility7of the gestural interface system may be increased, generating a robust and reliable alternative interface to the personal computing device that could not be provided based on the processing power of the personal computing device alone.

[0105] The disclosure also provides support for a method for a controller of a vehicle, the method comprising: detecting a personal computing device of an occupant of the vehicle via a radio frequency (RF) signal, detecting a physical gesture of the occupant via an in-cabin sensing device of the vehicle while the occupant is using the personal computing device, the physical gesture performed in the air without touching the personal computing device, mapping the physical gesture to an action, the action performed via a user interface (UI) of the personal computing device, and notifying a gestural interface service running on the personal computing device of the action, wherein the action is performed by the gestural interface service on the personal computing device. In a first example of the method, the physical gesture includes one of waving a hand, pointing or moving a finger, and swiping a hand or finger in a direction. In a second example of the method, optionally including the first example, the in-cabin sensing device is one of a camera of an occupant monitoring system (OMS) of the vehicle, a time-of- flight (TOF) camera of the vehicle, and an in-cabin radar of the vehicle. In a third example ofAttorney Docket No. P240063WO the method, optionally including one or both of the first and second examples, the method further comprises: in response to detecting the personal computing device, transmitting a signal to the gestural interface service to determine whether the gestural interface service is enabled at the personal computing device, in response to not receiving a response from the gestural interface service sending a notification to the personal computing device to prompt the occupant to enable the gestural interface service, in response to receiving a response from the gestural interface service, determining a seat location of the occupant or sending a notification to the gestural interface service to prompt the occupant to identify a seat location of the occupant, receiving a user input of the seat location from the gestural interface service, receiving one or more images outputted by the in-cabin sensing device, the in-cabin sensing device positioned at the seat location, and detecting the physical gesture of the occupant in the one or more images. In a fourth example of the method, optionally including one or more or each of the first through third examples, the method further comprises: receiving a set of gestural controls for interacting with the UI from the personal computing device via the gestural interface service, the set of gestural controls including a mapping of images of physical gestures and / or verbal commands to actions performed in the UI and / or in an application running on the personal computing device. In a fifth example of the method, optionally including one or more or each of the first through fourth examples, the set of gestural controls is received in response to the personal computing device being paired with the vehicle when the personal computing device and / or application is launched or running. In a sixth example of the method, optionally including one or more or each of the first through fifth examples, the set of gestural controls is requested from the gestural interface service by the controller. In a seventh example of the method, optionally including one or more or each of the first through sixth examples, detecting the physical gesture of the occupant via the one or more images further comprises inputting the one or more images into a machine learning (ML) model, and receiving as an output of the ML model, an encoding of an action performed in the UI associated with the physical gesture. In a eighth example of the method, optionally including one or more or each of the first through seventh examples, the ML model is trained to detect a plurality of physical gestures, and the set of gestural controls requested from the gestural interface service is selected from a library of gestural controls including the plurality of physical gestures.

[0106] The disclosure also provides support for a method for a personal computing device, the method comprising: transmitting a set of gestural controls stored in a memory of the personal computing device to a device interface manager component of a controller of a vehicle via a gestural interface service running on the personal computing device, each gestural controlAttorney Docket No. P240063WO of the set of gestural controls comprising one or more images of a physical gesture performed in the air by a user of the personal computing device, and an associated action performed in a user interface (UI) of the personal computing device, receiving a notification transmitted from the device interface manager to the gestural interface service indicating an action of a gestural control to be performed in the UI, and performing the action in the UI, wherein the notification is received as a result of the physical gesture of the gestural control being captured via an incabin sensing device of the vehicle. In a first example of the method, the personal computing device is one of a smart phone, a tablet, an eBook reader, and a laptop. In a second example of the method, optionally including the first example, the in-cabin sensing device is one of a camera of an occupant monitoring system (OMS) of the vehicle, a time-of-flight (TOF) camera of the vehicle, and an in-cabin radar of the vehicle. In a third example of the method, optionally including one or both of the first and second examples, the method further comprises: displaying a library of gestural controls for interacting with the personal computing device in the UI, the library of gestural controls including physical gestures that a machine learning (ML) model of the device interface manager has been trained to detect, receiving a user input of a selected gestural control of the library of gestural controls, receiving a mapping of the selected gestural control to a functionality of the personal computing device or an action performed in the UI, storing the selected, mapped gestural control in the set of gestural controls in the memory. In a fourth example of the method, optionally including one or more or each of the first through third examples, the method further comprises: in response to receiving a notification from the device interface manager, prompting the user via the UI to input a location of a seat of the vehicle that the user is positioned in, and in response to receiving the user input, transmitting the location of the seat to the device interface manager.

[0107] The disclosure also provides support for a gestural interface system of a vehicle, the gestural interface system comprising: a personal computing device of an occupant of the vehicle, the personal computing device having a first processor and a first memory storing instructions that when executed, cause the first processor to perform an action in a user interface (UI) of the personal computing device, the action indicated by a gestural interface service running on the personal computing device, a controller of the vehicle, the controller having a second processor and a second memory storing instructions that when executed, cause the second processor to: detect the personal computing device via a radio frequency (RF) signal, create a wireless connection with the personal computing device, detect a physical gesture of the occupant via an in-cabin sensing device of the vehicle while the occupant is using the personal computing device, the physical gesture performed in the air without touching theAttorney Docket No. P240063WO personal computing device, map the physical gesture to the action, and notify the gestural interface service of the action via the wireless connection. In a first example of the system, the in-cabin sensing device is one of a camera of an occupant monitoring system (OMS) of the vehicle, a time-of-flight (TOF) camera of the vehicle, and an in-cabin radar of the vehicle. In a second example of the system, optionally including the first example, further instructions are stored in the second memory that when executed, cause the second processor to: in response to detecting the personal computing device, transmit a signal to the gestural interface service via the wireless connection to determine whether the gestural interface sendee is running at the personal computing device, in response to not receiving a response from the gestural interface sen-ice, send a notification to the personal computing device via the wireless connection to prompt the occupant to enable the gestural interface service, in response to receiving a response from the gestural interface service, send a notification to the gestural interface sendee to prompt the occupant to identify a seat location of the occupant, receive a user input of the seat location from the gestural interface service, receive one or more images outputted by an in-cabin sensing device positioned at the seat location, and detect the physical gesture of the occupant in the one or more images. In a third example of the system, optionally including one or both of the first and second examples, the physical gesture of the occupant is detected in the one or more images by a machine learning (ML) gesture detection model. In a fourth example of the system, optionally including one or more or each of the first through third examples, further instructions are stored in the first memory that when executed, cause the first processor to: via the wireless connection, transmit a set of gestural controls for interacting with the UI from the personal computing device to the controller using the gestural interface service, the set of gestural controls including a mapping of images of physical gestures and / or verbal commands to actions performed in the UI and / or in an application running on the personal computing device. In a fifth example of the system, optionally including one or more or each of the first through fourth examples, further instructions are stored in the first memory that when executed, cause the first processor to: display a library- of gestural controls for interacting with the personal computing device in the UI, the library of gestural controls including physical gestures that the gesture detection model has been trained to detect, receive a user input of a selected gestural control of the library of gestural controls, receive a mapping of the selected gestural control to a functionality of the personal computing device or an action performed in the UI, store the selected, mapped gestural control in the set of gestural controls in the memory.

[0108] When introducing elements of various embodiments of the present disclosure, the articles “a,” ‘'an,’’ and '‘the” are intended to mean that there are one or more of the elements.Attorney Docket No. P240063WOThe terms “first,"’ “second,” and the like, do not denote any order, quantity, or importance, but rather are used to distinguish one element from another. The terms “comprising,” “including,” and “having” are intended to be inclusive and mean that there may be additional elements other than the listed elements. As the terms “connected to,” “coupled to,” etc. are used herein, one object (e.g., a material, element, structure, member, etc.) can be connected to or coupled to another object regardless of whether the one object is directly connected or coupled to the other object or whether there are one or more intervening objects between the one object and the other object. In addition, it should be understood that references to “one embodiment” or “an embodiment” of the present disclosure are not intended to be interpreted as excluding the existence of additional embodiments that also incorporate the recited features.

[0109] In addition to any previously indicated modification, numerous other variations and alternative arrangements may be devised by those skilled in the art without departing from the spirit and scope of this description, and appended claims are intended to cover such modifications and arrangements. Thus, while the information enhancement described above with particularity and detail in connection with what is presently deemed to be the most practical and preferred aspects, it will be apparent to those of ordinary skill in the art that numerous modifications, including, but not limited to, form, function, manner of operation and use may be made without departing from the principles and concepts set forth herein. Also, as used herein, the examples and embodiments, in all respects, are meant to be illustrative only and should not be construed to be limiting in any manner.

Claims

Attorney Docket No. P240063WOCLAIMS:

1. A method for a controller of a vehicle, the method comprising: detecting a personal computing device of an occupant of the vehicle via a radio frequency (RF) signal; detecting a physical gesture of the occupant via an in-cabin sensing device of the vehicle while the occupant is using the personal computing device, the physical gesture performed in the air without touching the personal computing device; mapping the physical gesture to an action, the action performed via a user interface (UI) of the personal computing device; and notifying a gestural interface service running on the personal computing device of the action; wherein the action is performed by the gestural interface service on the personal computing device.

2. The method of claim 1, wherein the physical gesture includes one of waving a hand, pointing or moving a finger, and swiping a hand or finger in a direction.

3. The method of claim 1, wherein the in-cabin sensing device is one of a camera of an occupant monitoring system (OMS) of the vehicle, a time-of-flight (TOF) camera of the vehicle, and an in-cabin radar of the vehicle.

4. The method of claim 1, further comprising: in response to detecting the personal computing device, transmitting a signal to the gestural interface service to determine whether the gestural interface service is enabled at the personal computing device; in response to not receiving a response from the gestural interface service sending a notification to the personal computing device to prompt the occupant to enable the gestural interface service; in response to receiving a response from the gestural interface service, determining a seat location of the occupant or sending a notification to the gestural interface sendee to prompt the occupant to identify a seat location of the occupant; receiving a user input of the seat location from the gestural interface service;Attorney Docket No. P240063WO receiving one or more images outputed by the in-cabin sensing device, the in-cabin sensing device positioned at the seat location; and detecting the physical gesture of the occupant in the one or more images.

5. The method of claim 1, further comprising receiving a set of gestural controls for interacting with the UI from the personal computing device via the gestural interface service, the set of gestural controls including a mapping of images of physical gestures and / or verbal commands to actions performed in the UI and / or in an application running on the personal computing device.

6. The method of claim 5. wherein the set of gestural controls is received in response to the personal computing device being paired with the vehicle when the personal computing device and / or application is launched or running.

7. The method of claim 5, wherein the set of gestural controls is requested from the gestural interface service by the controller.

8. The method of claim 7, wherein detecting the physical gesture of the occupant via the one or more images further comprises inputting the one or more images into a machine learning (ML) model, and receiving as an output of the ML model, an encoding of an action performed in the UI associated with the physical gesture.

9. The method of claim 8, wherein the ML model is trained to detect a plurality of physical gestures, and the set of gestural controls requested from the gestural interface sendee is selected from a library of gestural controls including the plurality of physical gestures.

10. A method for a personal computing device, the method comprising: transmitting a set of gestural controls stored in a memory of the personal computing device to a device interface manager component of a controller of a vehicle via a gestural interface service running on the personal computing device, each gestural control of the set of gestural controls comprising one or more images of a physical gesture performed in the air by a user of the personal computing device, and an associated action performed in a user interface (UI) of the personal computing device;Attorney Docket No. P240063WO receiving a notification transmitted from the device interface manager to the gestural interface service indicating an action of a gestural control to be performed in the UI; and performing the action in the UI; wherein the notification is received as a result of the physical gesture of the gestural control being captured via an in-cabin sensing device of the vehicle.

11. The method of claim 10, wherein the personal computing device is one of a smart phone, a tablet, an eBook reader, and a laptop.

12. The method of claim 10, wherein the in-cabin sensing device is one of a camera of an occupant monitoring system (OMS) of the vehicle, a time-of-flight (TOF) camera of the vehicle, and an in-cabin radar of the vehicle.

13. The method of claim 10, further comprising: displaying a library of gestural controls for interacting with the personal computing device in the UI, the library of gestural controls including physical gestures that a machine learning (ML) model of the device interface manager has been trained to detect; receiving a user input of a selected gestural control of the library of gestural controls; receiving a mapping of the selected gestural control to a functionality of the personal computing device or an action performed in the UI; storing the selected, mapped gestural control in the set of gestural controls in the memory.

14. The method of claim 10, further comprising: in response to receiving a notification from the device interface manager, prompting the user via the UI to input a location of a seat of the vehicle that the user is positioned in; and in response to receiving the user input, transmitting the location of the seat to the device interface manager.

15. A gestural interface system of a vehicle, the gestural interface system comprising: a personal computing device of an occupant of the vehicle, the personal computing device having a first processor and a first memory' storing instructions that when executed, cause the first processor to perform an action in a user interface (UI) of the personal computingAttorney Docket No. P240063WO device, the action indicated by a gestural interface service running on the personal computing device; a controller of the vehicle, the controller having a second processor and a second memory storing instructions that when executed, cause the second processor to: detect the personal computing device via a radio frequency' (RF) signal; create a wireless connection with the personal computing device; detect a physical gesture of the occupant via an in-cabin sensing device of the vehicle while the occupant is using the personal computing device, the physical gesture performed in the air without touching the personal computing device; map the physical gesture to the action; and notify the gestural interface service of the action via the wireless connection.

16. The gestural interface system of claim 15, wherein the in-cabin sensing device is one of a camera of an occupant monitoring system (OMS) of the vehicle, a time-of-flight (TOF) camera of the vehicle, and an in-cabin radar of the vehicle.

17. The gestural interface system of claim 16, wherein further instructions are stored in the second memory' that when executed, cause the second processor to: in response to detecting the personal computing device, transmit a signal to the gestural interface service via the wireless connection to determine whether the gestural interface service is running at the personal computing device; in response to not receiving a response from the gestural interface service, send a notification to the personal computing device via the wireless connection to prompt the occupant to enable the gestural interface service; in response to receiving a response from the gestural interface service, send a notification to the gestural interface service to prompt the occupant to identify a seat location of the occupant; receive a user input of the seat location from the gestural interface service; receive one or more images outputted by an in-cabin sensing device positioned at the seat location; and detect the physical gesture of the occupant in the one or more images.

18. The gestural interface system of claim 17, where the physical gesture of the occupant is detected in the one or more images by a machine learning (ML) gesture detection model.Attorney Docket No. P240063WO19. The gestural interface system of claim 18. wherein further instructions are stored in the first memory that when executed, cause the first processor to: via the wireless connection, transmit a set of gestural controls for interacting with the UI from the personal computing device to the controller using the gestural interface service, the set of gestural controls including a mapping of images of physical gestures and / or verbal commands to actions performed in the UI and / or in an application running on the personal computing device.

20. The gestural interface system of claim 19. wherein further instructions are stored in the first memory that when executed, cause the first processor to: display a library of gestural controls for interacting with the personal computing device in the UI, the library of gestural controls including physical gestures that the gesture detection model has been trained to detect; receive a user input of a selected gestural control of the library’ of gestural controls; receive a mapping of the selected gestural control to a functionality of the personal computing device or an action performed in the UI; store the selected, mapped gestural control in the set of gestural controls in the memory.

Images