Audio arbitration for audio devices

Abstract

The invention provides audio arbitration for an audio device. A computer may control audio output provided to a second user via a second user audio device based on a user location of a first user.

Description

Technical Field

[0001] The audio system described in this article enhances scenarios where multiple users can listen to different audio data streams simultaneously. Background Technology

[0002] A computer can output one or more audio streams via appropriate audio devices (such as speakers, headphones, etc.), thereby providing audio content to listeners. For example, a computer in a vehicle can output audio streams to the vehicle occupants. Audio streams can include a wide range of content, such as vehicle data, vehicle service notifications, navigation maps, entertainment data (such as podcasts, movies, video games, and the Internet via a web browser), and / or cellular data (such as text messages). Summary of the Invention

[0003] For example, the techniques described herein can provide control over one or more audio streams associated with one or more users, such as vehicle occupants. Control over the delivery of audio data as described herein can facilitate in-vehicle communication (e.g., between users) and / or between devices such as vehicle human-machine interfaces (HMIs) and one or more users. The respective user (such as a vehicle occupant) may be associated with an audio device, such as a speaker or headphones (e.g., headphones with wireless earbuds) pointing to the user's location or position (i.e., the position where the user is seated in the vehicle compartment (hereinafter referred to as the user's "compartment position")). Audio data can be provided to one or more corresponding audio devices based on parameters and / or states (e.g., vehicle status, such as parked or moving) of the user and / or events (e.g., the vehicle being approached by a second vehicle such as a law enforcement vehicle, medical vehicle, or fire response vehicle).

[0004] Therefore, this disclosure includes a system comprising a computer, the computer including a processor and a memory, the memory storing instructions executable by the processor to: control audio output provided to a second user via a second user's audio device based on the user location of a first user.

[0005] User position can include the posture of the first user's head, hands, and / or arms.

[0006] User location can be determined by eye movement.

[0007] Eye movement can be used to adjust the first user's eye to a viewing direction facing the vehicle components.

[0008] Eye movement can adjust the first user's eyes to face the second user's viewing direction.

[0009] In addition to the user's location, audio output can also be controlled based on external events.

[0010] An external event could be the approach of another vehicle.

[0011] External events can be traffic conditions.

[0012] Audio output can be controlled based on predefined permissions.

[0013] Permissions can be based on input from a second user.

[0014] Permissions can be based on the input of the first user.

[0015] Controlling audio output may include adjusting the volume of the audio output.

[0016] Controlling audio output can include pausing audio output.

[0017] Controlling the audio output may include changing the audio output from a first audio program to a second audio program.

[0018] Audio output can only be controlled when the volume of the audio output is higher than the specified volume threshold.

[0019] One method includes controlling audio output provided to a second user via a second user's audio device based on the user location of a first user.

[0020] User position can include the posture of the first user's head, hands, and / or arms.

[0021] User location can be determined by eye movement.

[0022] In addition to the user's location, audio output can also be controlled based on external events.

[0023] Audio output can be controlled based on predefined permissions. Attached Figure Description

[0024] Figure 1 This is a block diagram of an example vehicle system.

[0025] Figure 2 This is a top-down illustrative view of the vehicle, exposing the passenger compartment for explanation.

[0026] Figure 3 This is a flowchart of an example process for registering an audio device.

[0027] Figure 4 This is a flowchart of an example process for outputting audio data. Detailed Implementation

[0028] Exemplary system components

[0029] refer to Figure 1Vehicle system 100 includes vehicle 102. Vehicle system 100 is presented as an example environment. However, it should be understood that this disclosure is applicable to other architectures and / or systems, such as home entertainment systems, aircraft infotainment systems, etc. Vehicle system 100 includes computer 104 having memory including instructions executable by a processor of computer 104 to perform processes and operations (including those described herein). For example, the memory stores processor-executable instructions, including instructions for controlling audio output provided to a second user via a second user's audio device based on the position of a first user (e.g., the direction or movement of the first user's eyes).

[0030] Computer 104 can be communicatively coupled to sensors 106, displays 108, audio devices 110, and other components in vehicle 102 via vehicle communication network 114. Vehicle 102 can be any passenger or commercial vehicle, such as sedans, trucks, SUVs, crossovers, vans, minivans, taxis, buses, ICE (internal combustion engine), BEV (battery electric vehicle), hybrid vehicles, PHEV (plug-in hybrid electric vehicle), etc.

[0031] As mentioned above, vehicle computer 104 includes a processor and memory. The memory includes one or more forms of computer-readable medium and stores instructions executable by computer 104 to perform various operations, including those disclosed herein. For example, computer 104 may be a general-purpose computer having a processor and memory as described above, and / or may include electronic control units (ECUs) or controllers for specific functions or sets of functions, and / or dedicated electronic circuitry including ASICs (Application-Specific Integrated Circuits) manufactured for specific operations (e.g., ASICs for processing and / or transmitting sensor data). In another example, computer 104 may include an FPGA (Field-Programmable Gate Array), which is manufactured as a user-configurable integrated circuit. Typically, hardware description languages ​​such as VHDL (Very High Speed ​​Integrated Circuit Hardware Description Language) are used in electronic design to describe digital and mixed-signal systems such as FPGAs and ASICs. For example, an ASIC is manufactured based on VHDL programming provided before manufacturing, while the logic components within an FPGA may be configured based on VHDL programming (e.g., stored in memory electrically connected to the FPGA circuitry). In some examples, computer 104 may include a combination of processor, ASIC, and / or FPGA circuitry. Computer 104 can be multiple computers coupled together.

[0032] The memory can be of any type (e.g., hard disk drive, solid-state drive, server, or any volatile or non-volatile media). The memory can store collected data transmitted from sensor 106. The memory can be a separate device from computer 104, and computer 104 can retrieve the data stored in the memory via network 114 in vehicle 102 (e.g., via CAN bus, wireless network, etc.). Alternatively or additionally, the memory can be part of computer 104 (e.g., as memory of computer 104).

[0033] Computer 104 may include programs to: operate one or more of the vehicle components, such as propulsion (e.g., controlling the speed of vehicle 102 by controlling one or more of an internal combustion engine, an electric motor, a hybrid engine, etc.), steering, interior and / or exterior lights, display 108, audio device 110, etc.; and determine whether and when computer 104 (not a human operator) controls such operations.

[0034] Computer 104 is typically arranged for communication over a vehicle communication network 114, which may include buses in vehicle 102, such as Controller Area Network (CAN), and / or other wired and / or wireless mechanisms. Alternatively or additionally, where computer 104 actually comprises multiple devices, vehicle communication network 114 may be used for communication between the devices represented herein as computer 104. Furthermore, as mentioned below, various controllers and / or sensors 106 may provide data to computer 104 via vehicle communication network 114.

[0035] Via vehicle network 114, computer 104 can transmit messages to and / or receive messages (e.g., CAN messages) from various devices and / or components in vehicle 102 (e.g., sensors 106, ECUs, etc.). Alternatively or additionally, where computer 104 actually comprises multiple devices, vehicle communication network 114 can be used for communication between devices represented herein as computer 104. Furthermore, as mentioned below, various controllers and / or sensors 106 can provide data to computer 104 via vehicle communication network 114.

[0036] Sensor 106 can provide data about the occupants of vehicle 100. Sensor 106 may be a camera and can detect electromagnetic radiation within a certain wavelength range. For example, the sensor may detect visible light, infrared radiation, ultraviolet light, or a range of wavelengths including visible light, infrared light, and / or ultraviolet light. For example, the camera may be a charge-coupled device (CCD), complementary metal-oxide-semiconductor (CMOS), or any other suitable type.

[0037] Vehicle 102 may include a display 108. Display 108 displays visual data (e.g., two-dimensional visual data) to the occupants of vehicle 102. The display may be any suitable type for displaying content clearly visible to the respective occupants (e.g., light-emitting diode (LED)), organic light-emitting diode (OLED), liquid crystal display (LCD), plasma, digital light processing technology (DLPT), etc. Display 108 may display the visual data via a screen in monochrome or color, and the visual data may be updated at a certain frame rate, such as 60 frames per second. The displayed visual data may be a static image, in which most of the two-dimensional area does not change with each frame, or it may be a dynamic image, in which most of the two-dimensional area changes with each frame. The visual data to be displayed on display 108 may be generated by a display controller. The display controller is a computing device such as an ECU, which may receive data to be displayed on display 108 in a visual format from computer 104, other vehicle ECUs, or from external computing device 118 via server 116.

[0038] Display 108 allows for user interaction. For example, display 108 can be a conventional touchscreen display, allowing the user to provide input to computer 104 via the display (e.g., selecting a dataset of content to be output by display 108 via display 108). The touchscreen can be any suitable type for receiving input from the user (e.g., resistive, capacitive, infrared, etc.).

[0039] Now besides Figure 1 In addition to referencing Figure 2 The audio device 110 can be any suitable device configured to output sound to a user (such as a first user 200a and a second user 200b as occupants of vehicle 102). For example, the audio device 110 can be a speaker, a personal device (e.g., headphones), etc. A speaker is an electroacoustic transducer that converts electrical signals into sound. A speaker can be any suitable type for producing sound that is audible (e.g., dynamic) to the respective user 200. A portable device can be any suitable device for emitting sound to a single user 200 (e.g., headphones such as in-ear or over-ear headphones, portable speakers, etc.). The personal device can be connected via a wired connection such as an audio jack and / or via a connection such as Bluetooth. TM The wireless connection is connected to the vehicle network 114.

[0040] Computer 104 can identify one or more audio devices 110. In cases where the audio device 110 is not a vehicle audio device but a portable device that can be paired / unpaired and / or plugged / unplugged into a user device and / or computer 104 (e.g., the audio device is a personal headset), computer 104 can identify the audio device 110 in the vehicle based on detecting that the audio device 110 is paired with or plugged into the computer and / or based on sensor data. For example, sensor 106 can collect one or more images of user 200, which include the audio device 110. Computer 104 can then input the images into a neural network trained to recognize the audio device 110 in the images. In cases where the audio device 110 is a vehicle audio device, vehicle 102 can identify the audio device 110 based on stored data about the vehicle audio device 110 (such as stored data about the speakers and their positions within vehicle 102).

[0041] Computer 104 can control the audio output of audio device 110. That is, computer 104 can select a set of parameters for configuring the audio stream, including the volume (i.e., decibel level) of the audio stream and the output location of the audio stream (i.e., the specifications of one or more audio devices 110 for outputting the audio stream). Computer 104 can control one or more parameters for delivering audio content, wherein one or more other parameters may remain unchanged. For example, to control audio device 110, computer 104 can reduce the volume of the audio stream and / or change which audio devices 110 are outputting the audio stream. Thus, computer 104 can control the audio output by adjusting the volume of the audio output, pausing the audio output, and changing the audio output from a first audio stream to a second audio stream (each of which will be described sequentially below). Computer 104 can control the audio output, for example, based on a lookup table specifying how the audio output should be controlled under a given external event and / or vehicle state. As another example, computer 104 can control the audio output based on user input (e.g., computer 104 may be instructed to always pause the audio output while the first user 200a is speaking). Computer 104 can control the audio output of audio device 110 based on user location or external events, as described below.

[0042] Figure 2The vehicle compartment 202 is shown. A vehicle user 200 may be associated with an audio device 110 in the compartment 202. Associating the audio device 110 with a user 200 means that the audio device 110 is designated for use by that user 200 and is assumed to be used by that user. A computer 104 may store user associations, such that an audio device 110 is assigned to a corresponding user 200. The computer 104 may associate the audio device 110 with the user 200 based on the user wearing the device 110 (e.g., headphones), as can be determined by any suitable image recognition technology (e.g., such as neural networks as mentioned above).

[0043] The computer can associate the audio device 110 with the user 200 based on the placement of the audio device 110 relative to the user 200's compartment location (e.g., front and rear areas of the compartment, headphones placed on the ears, etc.). The computer 104 can store the compartment locations of the vehicle audio devices 110 that are typically located in the nearest passenger compartment location (e.g., as specified by the vehicle manufacturer and / or as indicated by user input). For example, if the first user 200a is located in the operator compartment location of vehicle 102, it is any audio device 110 (e.g., a speaker) supported by the surface of vehicle 102 that is closer to the operator compartment location than any device 110 in any other compartment location. Alternatively or additionally, a device 110 oriented to emit sound toward a compartment location can be associated with that compartment location, and therefore with the user 200 at that compartment location.

[0044] As in Figure 2 As can be seen, the vehicle compartment 202 may include multiple audio devices 110. The compartment 202 may accommodate occupants or users 200 of the vehicle 102. The compartment 202 includes one or more compartment positions (e.g., one or more compartment positions located in the front row of the compartment 202 and one or more compartment positions located in the second row behind the front row). The compartment 202 may also include a third row (not shown) of compartment positions located at the rear of the compartment 202. The position and orientation of the compartment positions and their components may be selected by the user 200. In the current example, user 200a is located in the operator compartment position, and user 200b is located in the right rear compartment position.

[0045] Audio device 110 may be a portable user device (e.g., a smartphone, etc.) and / or may be installed in the vehicle compartment 202. For example, audio device 110 may be installed in the dashboard, console, etc. Audio device 110 may be a personal audio device (e.g., a wireless headset) or may be a dedicated in-vehicle audio device (e.g., a vehicle speaker) as shown. Audio device 110 may be supported by vehicle 102 in locations around the vehicle compartment 202 such that audio output can be provided to users 200 at different compartment locations at similar volumes (e.g., by outputting the same decibel level). For example, audio device 110 (e.g., those audio devices installed in vehicle 102) may be positioned on the dashboard, console, sides of vehicle 102, etc.

[0046] Audio device 110 can be configured for audio isolation, meaning that audio is output from audio device 110 such that a first user hears the audio at a first volume, where the first volume is intended to present audio content that the first user 200a can clearly hear, and a second user 200b will not hear the audio or will hear the audio at a second, lower, and generally less audible volume. For example, audio isolation can be achieved when an audio stream is output to a personal audio device 110 (such as headphones), where user 200 receives sound from the audio stream and prevents or reduces other sounds from reaching the user's ears. In some examples, audio isolation can be achieved by changing the audio output mode of the audio stream to output the audio stream via a designated audio device 110 (e.g., a vehicle speaker) arranged to cancel or interfere with sounds from each other. For example, if audio device 110 is a speaker, computer 104 can output the audio stream at a volume that the user can generally hear and that other users 200 cannot hear or at least cannot clearly hear, which can be facilitated by the speaker being directed towards the passenger compartment position 208 where user 200 is seated. Additionally, computer 104 may execute a suitable active noise cancellation algorithm to generate canceled audio data that cancels (i.e., reduces) audio data streams from another user 200. Active noise cancellation algorithms can cause sound waves contained in the audio stream to be out of phase or phase-shifted, thereby generating canceled audio data. The output canceled audio data can cause destructive interference to the sound that generated the input audio data. For example, canceled audio data used to cancel a first audio stream can be added to a second audio stream to isolate a second user 200 (i.e., minimize or reduce interference to the second user). Such techniques can be useful in cases where user 200 cannot access a personal audio device 110 (e.g., a Bluetooth headset), which would typically perform audio isolation by blocking audio streams that user 200 does not want to hear. In examples where user 200 does have access to the personal audio device 110, computer 104 can associate those personal audio devices 110 with a specific user 200 (e.g., where the user connects a headset to computer 104 via Bluetooth and designates the headset as belonging to user 200).

[0047] Computer 104 can control the audio output provided to second user 200b based on the position of first user 200a. As used herein, user position encompasses movement, i.e., the user positioning themselves, such as the posture of the first user 200a's head, hands, and / or arms. Computer 104 can control the audio output based on the determination that the intention of the first user 200a's position is for the first user 200a to attract the attention of the second user 200b. That is, the computer can detect the user position and control the audio output when the first user 200a moves to attract the attention of the second user 200b (e.g., to make it possible for the first user 200a to converse with the second user 200b). Computer 104 can detect the user position based on data collected by sensor 106, such as image data provided by camera sensor 106 in vehicle compartment 202. To detect and interpret the position of user 200 (which may include movement), computer 104 can input the image data into any suitable image recognition algorithm, such as an algorithm including a deep neural network trained to determine posture from image data. In the example, the neural network is trained to detect and identify locations (i.e., classify user locations, such as movement or eye gaze direction). For example, location classification could be a movement or gesture by which a first user 200a intends to attract the attention of a second user 200b. For example, the neural network could be trained to identify a location where user 200 moves, including raising a user's fist to their mouth (e.g., coughing), as not intended to attract the attention of another user and therefore not requiring control of the audio output. As another example, the neural network could be trained to identify a first user 200a waving their hand toward the second user 200b as intended to attract the attention of user 200, which would require control of the audio output. The computer 104 may store lookup tables, etc., specifying whether user locations output by the image recognition algorithm (e.g., waving at another user, nodding toward another user, adjusting headphones, etc.) require control of the audio output or do not require control of the audio output.

[0048] The location of user 200, as classified by the image recognition algorithm, can be eye movement. Computer 104 can detect eye movement, including the gaze direction of user 200. Computer 104 can first use any suitable eye detection algorithm (e.g., shape-based techniques using an elliptical eye model or a complex eye model; feature-based techniques, such as detecting local features, detecting filter responses, or detecting the pupil and iris; appearance-based techniques; hybrid techniques of the foregoing; etc.) to detect the gaze direction of user 200. Computer 104 can then use any suitable gaze tracking algorithm (e.g., model-based techniques, interpolation-based techniques, appearance-based techniques, visible light-based techniques, etc.) to detect the gaze direction of user 200. For example, when computer 104 detects that first user 200a is looking directly at second user 200b (i.e., when first user 200a directs their gaze toward the body of second user 200b), the computer can control the audio output. Eye movement can also be adjusting the eyes of first user 200 to a viewing direction toward vehicle components (e.g., viewing second user 200 via a rearview mirror). Computer 104 can determine the user's gaze direction. For example, a visible light or infrared image depicting the user from camera 115 can be input into a machine learning program trained to identify the gaze direction from the image. The gaze direction can be specified by a line drawn from a point on the user's eye (e.g., the center point of the pupil, where the line is typically perpendicular to a plane tangent to the point on the user's eye). Therefore, when the line defining the gaze direction intersects another user 200 or vehicle component, it can be determined that the user's gaze direction is pointing towards the other user 200 or vehicle component. Furthermore, when the angle between the line defining the gaze direction and a second line defined by the point on the user's eye and a point on the surface of the other user 200 or vehicle component is less than a specified threshold (e.g., 5°, 10°, etc.), the gaze direction can be specified as being towards the other user 200 or vehicle component. As described above, computer 104 can store lookup tables, etc., specifying whether control of audio output is needed for the detected user position.

[0049] As mentioned above, computer 104 can control audio output in response to detecting a user's location. That is, when computer 104 detects a user location where audio output needs to be controlled (e.g., as specified by a stored lookup table), computer 104 can control the audio output in any of the aforementioned ways (e.g., switching from a first audio stream to a second audio stream, adjusting the volume of an audio stream, pausing the audio stream, etc.). Computer 104 can also control audio output according to another lookup table. That is, depending on the detected user location, computer 104 can control the audio differently (e.g., user 200a tapping their ear causes computer 104 to switch the audio output to a second audio stream, while waving their hand pauses the audio stream).

[0050] Computer 104 can only control the audio output when the volume of the audio output exceeds a specified volume threshold. That is, computer 104 can only control the audio output in response to detecting the user's location or an event requiring control if the volume of the audio stream being listened to by user 200 exceeds a specified decibel level. For example, if the threshold is 20 decibels, computer 104 will only control the audio output to user 200 if the audio device 110 associated with user 200 is outputting audio at or above 20 decibels. The threshold decibel level is typically specified based on empirical testing or simulation as a decibel level believed to be likely to interfere with user 200's hearing and understanding of other sounds (e.g., the speech of other users). Alternatively or additionally, the threshold can be established through user input. For example, the computer may output sounds at different decibel levels, and when another user speaks or when a sound simulating another user's voice at a specified decibel level is played, the user 200 may provide input to select one of the decibel levels of the threshold, and / or the user may simply input the decibel level of the threshold, such as "soft," "medium," or "loud," where the user's selection will then be associated with the selected decibel level.

[0051] Computer 104 may register audio device 110 before controlling the audio output from audio device 110. Registering audio device 110 means storing its identity (e.g., network identifier, etc.) along with an indicator or flag authorizing computer 104 to control the audio output of audio device 110 in computer 104's memory or data storage. Registration may be performed by computer 104 for detecting audio device 110 (e.g., detecting pairing via Bluetooth, etc., and then prompting user input to perform registration). For example, user 200 of audio device 110 may be prompted to provide input authorizing computer 104 to control the audio output of device 110. Alternatively or additionally, computer 104 may register device 110 without user input. For example, computer 104 may register device 110 when sensor 106 detects device 110 (e.g., device 110 may be identified in Bluetooth pairing and registered based on the device's stored identifier and past registrations and user settings).

[0052] Registration can also establish different permissions for computer 104 to control audio device 110. As used herein, "permission" refers to the ability granted by computer 104 to control the audio output of device 110. Permissions are typically device-specific, i.e., how computer 104 manages device 110 in vehicle compartment 202. Furthermore, the permissions for device 110 can vary depending on various scenarios of device 110 in compartment 202. A device 110 scenario refers to one or more attributes used by device 110, such as the identity of user 200 using device 110, the user's location on device 110, the type of content streaming through device 110, etc. User 200 can specify permissions to specify scenarios in which computer 104 can control the audio output of device 110. For example, user 200a can specify that their audio device 110 (e.g., personal headphones) is not controlled by computer 104 in response to the location of another user 200b, but can be controlled in response to the location of a different user 200. As another example, user 200a (e.g., the operator of vehicle 102) may specify that another user 200b has no permissions for audio device 110 and that its audio output can be controlled by computer 104 without restriction. In addition to user input, permissions can also be specified by the vehicle compartment location of audio device 110, the type of audio device 110 (e.g., personal headphones or vehicle speakers), the identity of the user 200 of audio device 110, etc. As an example, audio device 110 associated with user 200 in the driver's compartment may have permissions that allow for different audio control than other users (e.g., computer 104 may control the output based on more than one internal / external event). As another example, audio device 110, as a personal audio device (e.g., headphones), may have different permissions than non-personal audio devices 110 (e.g., vehicle speakers). The permissions of device 110, or a set of permissions, may be stored in the memory of computer 104 (e.g., in a table, etc.).

[0053] Computer 104 can detect external events. As used throughout this disclosure, an "external event" is an event, occurrence, or activity other than the location of user 200 detectable by sensor 106. Computer 104 can determine events that require control of audio output, or factors that, in combination with one or more other events, require control of audio output. External events may include events such as the approach of another vehicle (e.g., law enforcement vehicle, fire response vehicle, medical vehicle, any vehicle with right-of-way, etc.), traffic conditions, vehicle 102 approaching a designated destination on a navigation route, etc. Computer 104 can detect external events via any suitable mechanism, such as object detection based on data from sensor 106, vehicle-to-vehicle or vehicle-to-infrastructure communication, communication from remote device 118, etc.

[0054] Computer 104 can detect internal events. In this disclosure, an "internal event" is the location of user 200 that can be detected by sensor 106, as described above.

[0055] Computer 104 can control audio output based on vehicle status. "Vehicle status" as used herein means the state of vehicle 102 as described by physical measurements of vehicle 102 and / or the environment surrounding vehicle 102. For example, vehicle status may include one or more of the following: ambient temperature surrounding vehicle 102, passenger compartment temperature of vehicle 102, speed of vehicle 102, and the movement state of vehicle 102 (e.g., moving or not moving and / or parked). Computer 104 can determine vehicle status based on data collected by sensor 106.

[0056] Computer 104 can determine one or more vehicle states in real time or near real time based on physical measurements of vehicle 102 and / or the environment in and around vehicle 102. For example, when vehicle 102 is moving on a road, computer 104 can determine that vehicle 102 is in a "moving state". As another example, when the vehicle's transmission is in park, vehicle 102 can be in a "parked" state. As yet another example, when computer 104 is displaying instructions for route navigation to user 200 (e.g., the user has requested guidance to a destination), the vehicle can be in a "navigation" state.

[0057] Computer 104 can control audio output based on one or more control parameters. In this context, "control parameters" (or simply "parameters") are specifications of how computer 104 can control the audio output of a particular audio device 110. That is, what control computer 104 can have over the audio output in response to external or internal events and / or permissions (i.e., in the current example, parameters may include internal events, external events, vehicle status, and / or permissions). Parameters may specify permitted controls (e.g., stopping or pausing output, controlling volume, etc.), system states (such as vehicle status), and / or external events that permit control (e.g., vehicle parts, vehicle movement, vehicle speed exceeding a threshold, detection of an approaching vehicle, etc.) and internal events on which computer 104 is permitted to control device 110 (e.g., user 200 position or gesture) (e.g., waving, eye movement, etc.). Table 1 below shows an example lookup table specifying parameters (e.g., categories of user position, permissions, external events, and vehicle status) and explanations of the corresponding example parameters:

[0058]

[0059] Table 1

[0060] Table 1 provides examples of how computer 104 can control audio output based on parameters. That is, computer 104 can detect one or more events (e.g., internal and / or external events), vehicle status, and / or one or more permissions, and can determine the appropriate control output for device 110 based on the events (e.g., based on a lookup table, such as Table 1). For example, the first row of Table 1 shows that an operator waving towards a user can cause computer 104 to provide a control output to reduce the volume of the user's audio device 110. In another example, the last row of Table 1 shows that a combination of "Do Not Disturb" and "Operator Override" permissions and the internal event "Looking Direction Towards User" can cause computer 104 to provide a control output to reduce the volume of the user's audio device 110. In this example, the "Do Not Disturb" permission indicates that the audio device 110 should not be interfered with or controlled by computer 104, but the "Operator Override" permission takes precedence over the "Do Not Disturb" permission, such that when the internal event "Looking Direction Towards User" is detected, computer 104 can reduce the volume of the user's device 110.

[0061] Therefore, as shown in the figure, computer 104 can consider one or more parameters as shown in Table 1 to determine the control output of audio device 110. In addition to specifying the control output based on the occurrence of each parameter, lookup tables or the like can also specify the control output based on a combination of parameters detected together. For example, an entry in the lookup table can specify that if sensor 106 detects two user positions (e.g., gesture and gaze direction), computer 104 mutes the audio output, while if only one of the internal events occurs, computer 104 simply reduces the volume of the audio output.

[0062] Permissions can be specified through input from a first user and / or based on input from a second user. As an example, if user 200a of audio device 110 (e.g., via input through audio device 110, display 108, etc.) specifies to computer 104 that they do not wish to be disturbed by a specific user 200b, and / or if user 200b specifies that user 200b should be ignored for possible control of device 110, then computer 104 may ignore any location of user 200b to control the audio output of user 200a's audio device 110. As another example, user 200b (e.g., a passenger) may specify that they should not be disturbed by any other user 200. Furthermore, even if user 200b has specified "Do Not Disturb," it is possible that user 200a (e.g., a vehicle operator) specifies that user 200b's permission is overridden when user 200a attempts to attract their attention; in such cases, computer 104 will control the audio output of audio device 110 when user 200a makes a movement intended to attract user 200b's attention.

[0063] Permissions can be based on audio device 110. That is, computer 104 can assign permissions to a specific audio device 110 based on stored instructions. As an example, computer 104 can disable all permissions from a group of headphones belonging to a teenager. Users can assign permissions to a specific audio device 110 (e.g., via user input), or the assigned permissions can be pre-stored by computer 104.

[0064] In addition to the location of user 200, control parameters may also include external events. That is, computer 104 may determine whether control of audio device 110 is needed based on the detection of external events. A control parameter lookup table (e.g., Table 1) may specify whether control of audio output is needed for different external events, and to what extent the audio output can be controlled if so (e.g., pause, adjust volume, etc.).

[0065] Control parameters may also include vehicle status. A control parameter lookup table (e.g., Table 1) may specify how each vehicle status may affect whether control of the audio output is needed and to what extent the computer 104 can control the audio output. For example, a "moving" status may have no effect on the computer 104's control of the audio output. However, a "parked" status may require the computer 104 to override the permissions of all users 200 so that the driver of vehicle 102 can speak to each user 200. The effect of vehicle status on the control of audio output may be specified during the development of the computer 104 or by user 200 via user input. As an example, user 200a may specify that when the vehicle is in a "highway" status, the computer 104 should control the audio output normally, while when the vehicle is in a "lane warning" status (e.g., lane crossing detected), the computer 104 will override all user permissions.

[0066] Example process

[0067] refer to Figures 1 to 2 Description Figure 3 An example process 300 for registering an audio device with a computer 104 is shown. Process 300 can be executed according to program instructions executed by the computer 104.

[0068] The process begins at box 310, where computer 104 identifies one or more audio devices 110 in vehicle 102. As described above, computer 104 may identify audio device 110 via stored data (e.g., where audio device 110 is a vehicle audio device 110), image data, or communication established between the user and computer 104 (e.g., pairing via Bluetooth, etc.).

[0069] Next, in box 315, computer 104 requests registration of device 110. As an example, computer 104 may require audio device 110 to be registered before it can be used in vehicle 102. That is, before audio device 110 can utilize any vehicle system (e.g., vehicle infotainment system), device 110 is typically registered by computer 104 to make its audio output controllable.

[0070] In box 320, computer 104 registers device 110. As described above, registering audio device 110 means storing its identity (e.g., network identifier, etc.) together with an indicator or flag that authorizes computer 104 to control the audio output of audio device 110 in the memory or data storage of computer 104.

[0071] Next, in box 325, computer 104 stores permissions associated with audio device 110. For example, user 200 of audio device 110 may specify permissions (if any) via touchscreen display 108 or audio device 110. As another example, permissions stored in computer 104 may be based on carriage location, user 200's identity, or type of audio device 110. As described above, permissions specify the scenarios in which computer 104 is permitted to control the audio output of audio device 110 (e.g., permissions may prohibit control when a particular user 200b attempts to attract user 200a's attention).

[0072] Next, in box 330, computer 104 determines whether any additional unregistered audio devices 110 exist. If an additional unregistered audio device 110 exists, the process returns to box 315. Otherwise, process 300 then ends.

[0073] refer to Figures 1 to 2 Description Figure 4 An example process 400 is shown for monitoring internal and external events and determining whether control over audio device 110 is needed based on these events. Process 400 can be executed according to program instructions executed by computer 104, and typically executes after registration process 300 has been completed. Process 400 can monitor multiple users 200 and multiple devices 110 simultaneously or concurrently. That is, computer 104 can execute the process for any number of audio devices 110.

[0074] Process 400 begins in decision box 410, where computer 104 monitors internal events. That is, computer 104 monitors the user position of user 200. Computer 104 actuates sensor 106 to collect image data of user 200 and analyzes the image data to obtain the user position (e.g., via a neural network). Whenever computer 104 detects user 200 in vehicle 102, computer 104 can actuate sensor 106 to collect image data at a specified frame rate (e.g., 60 frames per second).

[0075] In box 415, which is depicted as following box 410 but may also be executed simultaneously or in parallel with box 410, computer 104 monitors external events. As described above, computer 104 may detect external events via any suitable mechanism, such as object detection based on data from sensor 106, vehicle-to-vehicle or vehicle-to-infrastructure communication, communication from remote device 118, etc.

[0076] Next, in decision box 420, computer 104 determines whether a parameter triggering the control output of audio device 110 is detected in either box 410 or 415. That is, if computer 104 detects an internal or external event in box 410 or 415, computer 104 determines that a parameter has been detected. If no parameter is detected, process 400 returns to box 410, and computer 104 continues to monitor internal and external events, etc. If computer 104 determines that a parameter has been detected, process 400 continues to box 425.

[0077] In block 425, computer 104 determines how to control the output of audio device 110. Computer 104 may determine how to control audio device 110 based on the output specified for parameters in a lookup table (e.g., Table 1). Computer 104 may consider one or more occurring parameters and control the audio output as specified in the table. The table may specify that audio control is not required for a particular event or combination of events. If audio control is not required, process 400 returns to block 410 to continue monitoring the parameters. Otherwise, process 400 continues to block 430.

[0078] In box 430, computer 104 monitors the volume of audio device 110. The volume can be measured in decibels. The volume can be detected and measured by any suitable means, such as audio sensor 106 (e.g., microphone), electroacoustic measurement, etc.

[0079] In decision box 435, computer 104 determines whether the volume of the audio output of audio device 110, measured in box 430, meets or exceeds the volume threshold described above. The threshold may be pre-stored by computer 104 or selected by user 200. If the volume of the audio output does not meet or exceed the threshold (e.g., the volume may be low enough not to prevent user 200 from hearing other users 200b), process 400 returns to box 410 to continue monitoring the parameter. Otherwise, process 400 continues to box 440.

[0080] In box 440, computer 104 has determined that the audio output meets or exceeds a volume threshold. Next, computer 104 controls the audio output of audio device 110 as described above (e.g., reducing the volume of the audio output, pausing the audio output, etc.).

[0081] Next, in box 450, computer 104 determines whether to continue process 400. For example, once process 400 is initiated, computer 104 can return to box 410 to continue monitoring parameters. However, process 400 may end after an input or event terminating process 400, such as user 200 stopping operation of vehicle 102 (e.g., shutting off a propulsion system such as the engine), user 200 providing input to end process 400, etc. If process 400 is to continue, the process returns to box 410; otherwise, process 400 ends.

[0082] Computing devices such as those discussed herein typically each include commands that can be executed by one or more computing devices, such as those identified above, and used to carry out blocks or steps of the processes described above. For example, the process blocks discussed above can be embodied as computer-executable commands.

[0083] Computer-executable commands can be compiled or interpreted by computer programs created using a variety of programming languages ​​and / or technologies, including but not limited to single or combined forms of the following: Java TM C, C++, Python, Julia, SCALA, Visual Basic, JavaScript, Perl, HTML, etc. Typically, a processor (i.e., a microprocessor) receives (i.e., from memory, computer-readable media, etc.) commands and executes these commands, thereby performing one or more procedures, including those described herein. Such commands and other data may be stored in files and transferred using various computer-readable media. Files in a computing device are typically collections of data stored on computer-readable media such as storage media, random access memory, etc.

[0084] Computer-readable media (also known as processor-readable media) include any non-transitory (i.e., tangible) medium that participates in providing data (i.e., instructions) that can be read by computer 104 (i.e., by the processor of computer 104). Such media can take many forms, including but not limited to non-volatile and volatile media. Instructions can be transmitted via one or more transmission media, including optical fibers, wires, wireless communications, and internals constituting a system bus coupled to the processor of computer 104. Common forms of computer-readable media include, for example, RAM, PROM, EPROM, flash EEPROM, any other memory chip or magnetic tape, or any other medium from which computer 104 can read.

[0085] Unless otherwise expressly indicated herein, all terms used in the claims are intended to have the ordinary and common meaning as understood by those skilled in the art. In particular, unless the claims explicitly limit the recitation to the contrary, the use of singular articles such as “a,” “the,” or “the” should be interpreted as one or more of the elements indicated by the recitation.

[0086] In the accompanying drawings, the same reference numerals indicate the same elements. Furthermore, some or all of these elements may be changed. Regarding the media, processes, systems, methods, etc., described herein, it should be understood that although the steps of such processes, etc., are described as occurring in a specific sequence, such processes can be practiced by performing the described steps in an order other than that described herein. It should also be understood that some steps may be performed simultaneously, other steps may be added, or some steps described herein may be omitted. In other words, the description of processes herein is provided for illustrative purposes and should in no way be construed as limiting the claimed invention.

[0087] The use of “in response to,” “based on,” and “after determining” in this document indicates a causal relationship, not just a temporal one. Unless otherwise expressly stated, “based on” or “in response to” may mean at least partially based on or at least partially in response to.

[0088] Examples are contemplated herein. Any example embodiments or features described herein are not necessarily to be construed as preferred or advantageous over other embodiments or features. Furthermore, the example embodiments described herein are not intended to be limiting. It should be readily understood that certain aspects of the disclosed systems and methods can be arranged and combined in a variety of different configurations, all of which are contemplated herein. Additionally, the specific arrangements shown in the figures should not be considered limiting. It should be understood that other embodiments may include more or fewer of each element shown in a given figure. Additionally, some of the shown elements may be combined or omitted. Furthermore, exemplary embodiments may include elements not shown in the figures.

[0089] This disclosure has been described in an illustrative manner, and it should be understood that the terminology used is intended to be descriptive in nature and not restrictive. In light of the foregoing teachings, many modifications and variations of this disclosure are possible, and this disclosure may be practiced in ways other than those specifically described. The adjectives “first” and “second” are used throughout this document as identifiers and are not intended to indicate importance, order, or quantity. The use of “in response to,” “after determining…,” etc., indicates a causal relationship, not just a temporal one. The operations, systems, and methods described herein should always be implemented and / or performed in accordance with applicable user manuals and / or safety guidelines.

[0090] According to the present invention, a system is provided having a computer including a processor and a memory, the memory storing instructions executable by the processor to: control audio output provided to a second user via a second user's audio device based on the user location of a first user.

[0091] According to an embodiment, the user position includes the posture of the first user's head, hands, and / or arms.

[0092] According to an embodiment, the user's position is eye movement.

[0093] According to an embodiment, the eye movement refers to the first user adjusting their eyes to a viewing direction facing the vehicle component.

[0094] According to an embodiment, the eye movement is the adjustment of the first user's eyes to face the viewing direction of the second user.

[0095] According to an embodiment, the instructions further include instructions for performing the following operations: controlling the audio output based on external events in addition to the user's location as described by the first user.

[0096] According to an embodiment, the external event is the approach of another vehicle.

[0097] According to an embodiment, the external event is traffic conditions.

[0098] According to an embodiment, the instructions further include instructions for performing the following operation: controlling the audio output based on predetermined permissions.

[0099] According to an embodiment, the permission is based on the input of the second user.

[0100] According to an embodiment, the permission is based on the input of the first user.

[0101] According to an embodiment, controlling the audio output includes adjusting the volume of the audio output.

[0102] According to an embodiment, controlling the audio output includes pausing the audio output.

[0103] According to an embodiment, controlling the audio output includes changing the audio output from a first audio program to a second audio program.

[0104] According to an embodiment, the instructions further include instructions for performing the following operation: controlling the audio output only when the volume of the audio output is higher than a specified volume threshold.

[0105] According to the present invention, a method includes: controlling audio output provided to a second user via a second user's audio device based on the user location of a first user.

[0106] In one aspect of the invention, the user position includes the posture of the first user's head, hands, and / or arms.

[0107] In one aspect of the invention, the user position is eye movement.

[0108] In one aspect of the invention, the method includes controlling the audio output based on external events in addition to the user location of the first user.

[0109] In one aspect of the invention, the method includes controlling the audio output based on predetermined permissions.

Claims

1. A method, the method comprising: The audio output provided to the second user via the second user's audio device is controlled based on the first user's location.

2. The method of claim 1, wherein the user position includes the posture of the first user's head, hands, and / or arms.

3. The method of claim 1, wherein the user position is eye movement.

4. The method of claim 3, wherein the eye movement is the adjustment of the first user's eyes to a viewing direction toward the vehicle component.

5. The method of claim 3, wherein the eye movement is the adjustment of the first user's eyes to face the viewing direction of the second user.

6. The method of claim 1, further comprising controlling the audio output based on external events in addition to the user location of the first user.

7. The method of claim 6, wherein the external event is the approach of another vehicle.

8. The method of claim 6, wherein the external event is traffic conditions.

9. The method of claim 1, further comprising controlling the audio output based on predetermined permissions.

10. The method of claim 9, wherein the permission is based on the input of the second user.

11. The method of claim 9, wherein the permission is based on the input of the first user.

12. The method of claim 1, wherein controlling the audio output includes adjusting the volume of the audio output.

13. The method of claim 1, wherein controlling the audio output includes pausing the audio output.

14. The method of claim 1, wherein controlling the audio output includes changing the audio output from a first audio program to a second audio program.

15. A remote computer comprising a processor and a memory, the memory storing instructions executable by the processor to perform the method as claimed in any one of claims 1 to 14.

Images