System and method for communicating with visually and hearing impaired vehicle occupants

Abstract

A method and system for controlling an occupant output system associated with a vehicle is provided. The method and system receive vehicle or occupant context data from a vehicle context data source, generate occupant message data based on the vehicle or occupant context data, and determine whether an occupant associated with the occupant output system is visually or hearing impaired. When the occupant is determined to be visually or hearing impaired, the method and system determines an output modality to assist the occupant, and generates output for the occupant on an output device in the output modality based on the occupant message data.

Description

Technical Field

[0001] The present disclosure relates generally to systems and methods associated with vehicles, and more particularly to systems and methods for communicating with hearing and visually impaired vehicle occupants.

[0002] Situational Technology

[0003] An autonomous vehicle is a vehicle that is able to sense its environment and navigate with little or no user input. Autonomous vehicles use sensing devices such as radar, lidar, image sensors, etc. to sense their environment. Autonomous vehicle systems also use information from global positioning system (GPS) technology, navigation systems, vehicle-to-vehicle communications, vehicle-to-infrastructure technology, and / or drive-by-wire systems to navigate the vehicle.

[0004] Vehicle automation has been categorized into numerical levels ranging from zero to five, with zero corresponding to automation with no full human control and five corresponding to full automation with no human control. Various automated driver assistance systems (ADAS) such as cruise control, adaptive cruise control, and parking assist systems correspond to lower levels of automation, while truly "driverless" vehicles correspond to higher levels of automation.

[0005] A visually impaired vehicle occupant may find some vehicle maneuvers (e.g., sudden turns) disconcerting and may not be fully aware of the external scene, which may also be disconcerting. Additionally, boarding and exiting a vehicle (e.g., a rideshare vehicle) may be difficult for a visually impaired occupant. A hearing-impaired vehicle occupant may not be able to receive the full range of information conveyed by the vehicle, such as upcoming stops or mapped navigation directions, to name two of many possible examples.

[0006] Therefore, it is desirable to provide systems and methods that assist hearing and vision impaired vehicle occupants in receiving communications from the vehicle. In addition, it is desirable to provide various information to vision and hearing impaired occupants to assist in the comfort and safety of these occupants. Moreover, other desirable features and characteristics of the present invention will become apparent from the following detailed description and the appended claims, taken in conjunction with the accompanying drawings and the foregoing technical and contextual art. Summary of the invention

[0007] A system associated with a vehicle. The system includes an occupant output system including an output device, a vehicle context data source, and a processor in operable communication with the occupant output system and the vehicle context data source. The processor is configured to execute program instructions to cause the processor to receive vehicle or occupant context data from the vehicle context data source, generate occupant message data based on the vehicle or occupant context data, determine whether an occupant associated with the occupant output system is visually or hearing impaired, determine an output modality to assist the occupant when the occupant is determined to be visually or hearing impaired, and generate output for the occupant in the output modality on the output device based on the occupant message data.

[0008] In an embodiment, the vehicle or occupant context data includes images or other sensory data from the scene outside the vehicle.

[0009] In an embodiment, the vehicle or occupant context data includes the position of the occupant relative to the vehicle.

[0010] In an embodiment, the program instructions are configured to cause the processor to determine the position of the occupant relative to the vehicle based on one or more transmissions from the occupant device.

[0011] In an embodiment, the vehicle or occupant context data includes vehicle behavior information based on vehicle control commands.

[0012] In an embodiment, the vehicle or occupant context data includes map data.

[0013] In an embodiment, the occupant message data includes a message describing the environment external to the vehicle.

[0014] In an embodiment, the occupant message data includes instructions for occupants to board and exit the vehicle or any other occupant actions related to the vehicle.

[0015] In an embodiment, the occupant message data includes a description of an action taken by the vehicle.

[0016] In an embodiment, a determination is made as to whether the occupant is visually impaired or hearing impaired based on stored occupant data.

[0017] In an embodiment, determining whether the occupant is visually or hearing impaired includes retrieving stored occupant data based on the occupant identifier.

[0018] In an embodiment, the occupant identifier is determined based on a biometric identification of the occupant or based on a communication from an occupant device.

[0019] In an embodiment, a determination is made through automatic detection whether the occupant is visually or hearing impaired.

[0020] In an embodiment, generating the output for the occupant includes translating a message included in the message data into a preferred language of the occupant.

[0021] In an embodiment, generating the output for the occupant includes generating a sign language animation and / or a text output when the occupant is determined to be hearing impaired, and displaying the sign language animation on a visual display device of the output device.

[0022] In an embodiment, generating the output for the occupant includes generating an audio output on an audio output device of the occupant output system.

[0023] In an embodiment, vehicle or occupant contextual data includes an occupant position and an occupant identifier relative to the vehicle, the occupant position and occupant identifier being determined based on wireless communications from an occupant device, occupant message data including guidance regarding approaching, entering, exiting, or leaving the vehicle determined based on the occupant position, and whether an occupant associated with an occupant output system is visually or auditory impaired based on the occupant identifier.

[0024] In an embodiment, vehicle or occupant context data is generated by a processor by fusing context data from a perception system with vehicle motion data derived from vehicle control commands generated by an autonomous driving system of the vehicle.

[0025] In another aspect, a method for controlling an occupant output system associated with a vehicle is provided. The method includes receiving, by a processor, vehicle or occupant context data from a vehicle context data source; generating, by the processor, occupant message data based on the vehicle or occupant context data, determining, by the processor, whether an occupant associated with the occupant output system is visually or hearing impaired, determining, by the processor, an output modality to assist the occupant when the occupant is determined to be visually or hearing impaired, and generating, by the processor, an output for the occupant on an output device of the occupant output system based on the occupant message data.

[0026] In an embodiment, the method includes generating a sign language animation and / or a text output as an output to the occupant when it is determined that the occupant is hearing impaired. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Exemplary embodiments will be described below in conjunction with the following drawings, wherein like reference numerals represent like elements, wherein:

[0028] Figure 1 is a functional block diagram illustrating a vehicle associated with a communication system for communicating with blind and visually impaired passengers according to various embodiments;

[0029] Figure 2 An autonomous driving system associated with a communication system according to various embodiments is shown;

[0030] Figure 3 is a functional block diagram of a communication system according to various embodiments;

[0031] Figure 4is a functional block diagram of a detection system for automatically detecting a visually or hearing impaired person that may be associated with a communication system in accordance with various embodiments;

[0032] Figure 5 is a flow chart illustrating a method of communicating with a visually and hearing impaired passenger of a vehicle according to various embodiments; and

[0033] Figure 6 is a flow chart illustrating a method of automatically detecting a person who is visually or hearing impaired. DETAILED DESCRIPTION

[0034] The following detailed description is merely exemplary in nature and is not intended to limit application and use. In addition, there is no intention to be bound by any express or implied theory proposed in the previous technical field, context, brief overview or detailed description below. As used herein, the term module refers to any hardware, software, firmware, electronic control component, processing logic and / or processor device, individually or in any combination, including but not limited to: application-specific integrated circuits (ASICs), electronic circuits, processors (shared, dedicated or grouped) and memories that execute one or more software or firmware programs, combinational logic circuits and / or other suitable components that provide the functions.

[0035] Embodiments of the present disclosure may be described herein in terms of functional and / or logical block components and various processing steps. It should be understood that such block components may be implemented by any number of hardware, software, and / or firmware components configured to perform a specified function. For example, embodiments of the present disclosure may employ various integrated circuit components, such as memory elements, digital signal processing elements, logic elements, lookup tables, etc., which may perform various functions under the control of one or more microprocessors or other control devices. In addition, those skilled in the art will appreciate that embodiments of the present disclosure may be practiced in conjunction with any number of systems, and the systems described herein are merely exemplary embodiments of the present disclosure.

[0036] For the sake of brevity, conventional techniques related to signal processing, data transmission, signaling, control, and other functional aspects of the system (as well as the various operating components of the system) may not be described in detail herein. In addition, the connecting lines shown in the various figures included herein are intended to represent example functional relationships and / or physical couplings between the various elements. It should be noted that many alternative or additional functional relationships or physical connections may exist in the embodiments of the present disclosure.

[0037] As used herein, the term "occupant" or "passenger" includes persons located in a vehicle (eg, in a seat of the vehicle) as well as persons intended or to board a vehicle.

[0038] refer to Figure 1According to various embodiments, a communication system, generally shown at 200, is associated with a vehicle 10. Typically, the communication system 200 provides information to a visually or hearing impaired passenger of the vehicle. The information may include assisting a visually impaired person in entering the vehicle. The communication system 200 may use a personal application that may communicate with the vehicle 10 and provide guidance to the visually and hearing impaired person in the form of output preferred by the visually or hearing impaired person, including precise real-time motion guidance. The information communicated by the communication system 200 may include information about actions taken by the vehicle and other contextual information to be communicated to the hearing or visually impaired passenger. The communication information may be determined by fusing environmental data and vehicle control data. The communication system 200 may generate output using neural machine translation into the passenger's preferred language. The output may include sign language animations (and / or text output) generated for hearing impaired passengers and audio information generated for visually impaired passengers. The communication system 200 may automatically detect whether an occupant is visually impaired or hearing impaired by monitoring eye movements and responses to sounds. In Figure 1 In the embodiment of the present invention, the communication system 200 is shown as being included within the vehicle 10, but the distributed system may be configured with some components included in a passenger's personal electronic device (e.g., a smartphone or tablet), some components located in a cloud processing system, and some components located on the vehicle 10, or any combination thereof.

[0039] like Figure 1 As shown, the vehicle 10 generally includes a chassis 12, a body 14, front wheels 16, and rear wheels 18. The body 14 is disposed on the chassis 12 and substantially surrounds the components of the vehicle 10. The body 14 and the chassis 12 may together form a frame. The wheels 16-18 are each rotatably connected to the chassis 12 near a corresponding corner of the body 14.

[0040] In various embodiments, the vehicle 10 is an autonomous vehicle, and the communication system 200 is associated with the vehicle 10 (hereinafter referred to as the vehicle 10). The vehicle 10 is, for example, a vehicle that is automatically controlled to transport passengers from one location to another. In the illustrated embodiment, the vehicle 10 is described as a passenger car, but it should be understood that any other vehicle including motorcycles, trucks, sport utility vehicles (SUVs), recreational vehicles (RVs), marine vessels, airplanes, shared buses, coaches, etc. can also be used. In an exemplary embodiment, the vehicle 10 is a so-called level 4 or level 5 automation system. The level 4 system indicates "high automation", which refers to the driving mode-specific performance of the autonomous driving system for all aspects of dynamic driving tasks, even if the human driver does not respond appropriately to the intervention request. The level 5 system indicates "full automation", which refers to the full-time performance of the autonomous driving system for all aspects of dynamic driving tasks under all road and environmental conditions that the human driver can manage. However, in other embodiments, the vehicle 10 is less automated and includes an advanced driver assistance system (ADAS).

[0041] As shown, the vehicle 10 generally includes a propulsion system 20, a transmission system 22, a steering system 24, a braking system 26, a sensor system 28, an actuator system 30, at least one data storage device 32, at least one controller 34 and a connection system 36. In various embodiments, the propulsion system 20 may include an internal combustion engine, an electric motor such as a traction motor and / or a fuel cell propulsion system. The transmission system 22 is configured to transmit power from the propulsion system 20 to the wheels 16-18 according to a selectable speed ratio. According to various embodiments, the transmission system 22 may include a stepped automatic transmission, a continuously variable transmission or other suitable transmission. The braking system 26 is configured to provide braking torque to the wheels 16-18. In various embodiments, the braking system 26 may include friction brakes, line brakes, regenerative braking systems such as electric motors and / or other suitable braking systems. The steering system 24 affects the position of the wheels 16-18. Although depicted as including a steering wheel for illustrative purposes, in some embodiments contemplated within the scope of the present invention, the steering system 24 may not include a steering wheel.

[0042] The sensor system 28 includes one or more sensing devices 40a-40n that sense observable conditions of the external environment and / or the internal environment of the vehicle 10. The sensing devices 40a-40n may include, but are not limited to, radar, lidar, global positioning system, optical camera, thermal camera, ultrasonic sensor, and / or other sensors. The actuator system 30 includes one or more actuator devices 42a-42n that control one or more vehicle features, such as, but not limited to, the propulsion system 20, the transmission system 22, the steering system 24, and the braking system 26. In various embodiments, the vehicle features may further include interior and / or exterior vehicle features, such as, but not limited to, doors, trunks, and cabin features, such as air, music, lighting, etc. (not numbered).

[0043] The connected system 36 is configured to wirelessly transmit information to and from other entities 48, such as, but not limited to, other vehicles ("V2V" communications), infrastructure ("V2I" communications), remote systems, and / or personal devices. In an exemplary embodiment, the connected system 36 is a wireless communication system that is configured to communicate via a wireless local area network (WLAN) using the IEEE 802.11 standard or by using cellular data communications. However, additional or alternative communication methods such as dedicated short-range communication (DSRC) channels are also considered to be within the scope of the present disclosure. A DSRC channel refers to a one-way or two-way short-range to medium-range wireless communication channel designed specifically for automotive applications, and a corresponding set of protocols and standards.

[0044] The data storage device 32 stores data for automatically controlling the vehicle 10. In various embodiments, the data storage device 32 stores a defined map of the navigable environment. In various embodiments, the defined map may be predefined by a remote system (e.g., a cloud processing system) and obtained from the remote system. For example, the defined map may be assembled by the remote system and transmitted to the vehicle 10 (wirelessly and / or wiredly) and stored in the data storage device 32. It will be appreciated that the data storage device 32 may be part of the controller 34, separate from the controller 34, or part of the controller 34 and part of an independent system.

[0045] The controller 34 includes at least one processor 44 and a computer readable storage device or medium 46. The processor 44 can be any custom or commercially available processor, a central processing unit (CPU), a graphics processing unit (GPU), a secondary processor of several processors associated with the controller 34, a semiconductor-based microprocessor (in the form of a microchip or chipset), a macroprocessor, any combination thereof, or generally any device for executing instructions. The computer readable storage device or medium 46 can include volatile and non-volatile storage in, for example, read-only memory (ROM), random access memory (RAM), and keep-alive memory (KAM). KAM is a permanent or non-volatile memory that can be used to store various operating variables when the processor 44 is powered off. The computer readable storage device or medium 46 can be implemented using any of a variety of known storage devices, such as PROMs (programmable read-only memory), EPROMs (electrical PROMs), EEPROMs (electrically erasable PROMs), flash memory, or any other electrical, magnetic, optical, or combination storage device capable of storing data, some of which represent executable instructions used by the controller 34 in controlling the vehicle 10.

[0046] The instructions may include one or more separate programs, each of which includes an ordered list of executable instructions for implementing logical functions. When executed by the processor 44, the instructions receive and process signals from the sensor system 28, execute logic, calculations, methods and / or algorithms for automatically controlling components of the vehicle 10, and generate control signals to the actuator system 30 to automatically control components of the vehicle 10 based on the logic, calculations, methods and / or algorithms. Although Figure 1 Only one controller 34 is shown, but embodiments of the vehicle 10 may include any number of controllers 34 that communicate via any suitable communication medium or combination of communication mediums and collaborate to process sensor signals, execute logic, calculations, methods and / or algorithms, and generate control signals to automatically control features of the vehicle 10.

[0047] In various embodiments, one or more instructions of the controller 34 are included in the communication system 200 and, when executed by the processor 44, perform Figure 3 and Figure 4 The system describes the functions and Figure 5 and Figure 6The steps of the method described are described. In particular, the processor 44 is configured by instructions to allow the personal application to communicate with the vehicle 10 and translate the relevant information into the communication form desired by the individual, including sign language animation. The wireless connection technology allows the communication system to provide accurate location information during entry and exit from the vehicle 10. In addition, the processor 44 is configured by instructions to interpret environmental and vehicle context information to the passengers and output the interpretation in a manner suitable for hearing and visually impaired passengers. These interpretations may be suitable for vehicles equipped with ADAS or vehicles with a higher degree of automation. In addition, the processor 44 is configured by instructions to automatically determine when a communication modality for the visually or hearing impaired is required without prior interaction with the individual.

[0048] Reference Figure 1 The described vehicle 10 may be adapted for use in a taxi, bus or shuttle system in a particular geographic area (e.g., a city, a school or business park, a shopping mall, an amusement park, an activity center, etc.), or may simply be managed by a remote system. For example, the vehicle 10 may be associated with a remote transportation system based on an autonomous vehicle. In various embodiments, the operating environment also includes one or more user devices (not shown) that communicate with the vehicle 10 and / or the remote transportation system (not shown) via a communication network (not shown). A registered user of the remote transportation system may create a ride request via a user device. The ride request will typically indicate the passenger's desired pickup location (or current GPS location), the desired destination location (which may identify a predefined vehicle stop and / or a user-specified passenger destination), and the pickup time. The remote transportation system receives the ride request, processes the request, and dispatches the selected autonomous vehicle to pick up the passenger at the specified pickup location and at the appropriate time.

[0049] Reference now Figure 2 , and continue to refer to Figure 1 According to an exemplary autonomous driving application, the controller 34 implements the following Figure 2 An autonomous driving system (ADS) 70 is shown. That is, suitable software and / or hardware components of the controller 34 (eg, the processor 44 and the computer readable storage device 46 ) are used to provide an autonomous driving system 70 for use in conjunction with the vehicle 10 .

[0050] In various embodiments, the instructions for the autonomous driving system 70 may be organized by functions, modules, or systems. Figure 3 As shown, the autonomous driving system 70 may include a computer vision system 74, a positioning system 76, a guidance system 78, and a vehicle control system 80. As may be appreciated, as the present disclosure is not limited to the present example, in various embodiments, the instructions may be organized into any number of systems (e.g., combined, further divided, etc.).

[0051] In various embodiments, the computer vision system 74 synthesizes and processes sensor data and predicts the presence, location, classification, and / or path of features and objects in the environment of the vehicle 10. In various embodiments, the computer vision system 74 may combine information from multiple sensors, including but not limited to cameras, lidar, radar, and / or any number of other types of sensors. The computer vision system 74 provides perception data that the communication system 200 may use to derive environmental context data that may be used to generate passenger messages that describe the environment of the vehicle 10 to the user.

[0052] The positioning system 76 processes the sensor data and other data to determine the location of the vehicle 10 relative to the environment (e.g., local location relative to a map, precise location relative to a road lane, vehicle heading, speed, etc.). The guidance system 78 processes the sensor data and other data to determine a path for the vehicle 10 to follow. The vehicle control system 80 generates control signals for controlling the vehicle 10 based on the determined path. The location of the vehicle in the environment can also be used to derive context data to generate passenger messages describing the environment of the vehicle 10 to the user. The control signals used to control the vehicle 10 provide the user with vehicle context data for generating passenger messages to the user describing the actions taken by the vehicle 10.

[0053] Environmental context data and vehicle context data may also be generated in a vehicle equipped with an ADAS system using data from the sensor system 28 and the ADAS vehicle control commands.

[0054] In various embodiments, the controller 34 implements machine learning techniques to assist in the functions of the controller 34 , such as feature detection / classification, obstacle mitigation, route traversal, mapping, sensor integration, ground truth determination, etc.

[0055] As briefly mentioned above, Figure 1 The communication system 200 is associated with the autonomous driving system 70 and may also be associated with additional computer-implemented devices and cloud processing systems.

[0056] Now refer to Figure 3 and Figure 4 The communication system 200 is described in more detail. The communication system 200 includes vehicle and passenger context data sources, including a driving control source 202, an external camera 204, an internal camera 206, a mapping module 208, a V2X module 210, and other environmental data sources 212. Any suitable information source that allows the communication system 200 to input information about the vehicle's internal and external environment and vehicle motion may be included. The driving control source 202 may be Figure 2The vehicle control system 80, ADAS control system or other source of vehicle control or status allows the communication system 200 to determine what action the vehicle 10 has taken or will take, and use this information to generate messages to be conveyed to passengers. The external camera 204 (which can be part of the sensor system 28) can image the external scene around the vehicle to allow the communication system 200 to determine the environment of the vehicle 10. Other external perception systems may be included, such as lidar, radar, ultrasound, etc. In some embodiments, the fusion of perception data from different sources can be used to generate data describing the vehicle environment, which in turn allows the communication system 200 to generate relevant messages describing the environment. In addition, the external camera 204 or other perception sensor can allow the location of passengers who wish to board the vehicle 10 to be determined. The internal camera 206 allows images of occupants in the cabin of the vehicle 10 to be captured, which is useful when obtaining passenger information, particularly for automatically detecting whether the passenger is hearing or visually impaired. The mapping module 208 can be similar to the positioning system 76 described above or a more standard navigation tool. Typically, the mapping module will include detailed information about the environment of the vehicle 10, including the names and types of places, which provides useful environmental contextual data for generating descriptive messages to users of the environment of the vehicle 10. The V2X module 210 allows information to be obtained from the environment including other vehicles, construction equipment, pedestrian equipment, advertising equipment, infrastructure, advisory equipment, etc. This information can be a rich source of environmental information for generating relevant information to the passengers of the vehicle 10. Many other environmental data sources 212 can be used in the communication system 200 to provide contextual data for generating messages for the passengers. Figure 3 In the embodiment of FIG. 1 , various types of data sources have been categorized to provide passenger context data 214 and vehicle context data 216. Passenger context data 214 includes passenger identification information or information related to identifying a passenger, and vehicle context data includes information describing actions taken by vehicle 10 and external environment information.

[0057] exist Figure 3 In an exemplary embodiment of the present invention, the communication system 200 includes a passenger message generation module 217 and a passenger output system 224. The passenger message generation module 217 receives the vehicle context data 216 and the passenger context data 214. The passenger message generation module 217 includes a vehicle context interpretation submodule 218 for generating a message describing the action and environmental context of the vehicle 10. The passenger message generation module 217 includes a passenger location submodule 222 and a passenger guidance submodule 220 for locating passengers (potential passengers) inside or outside the vehicle 10 and for providing movement directions to passengers to safely take seats, board, get off, etc.

[0058] The vehicle context interpretation submodule 218 receives vehicle context data 216 that provides information about the external environment and also provides information about the actions taken by the vehicle 10. The information about the external environment may include map information from the map module 208, such as location names, landmarks, stores, amenities, facilities, planned actions, upcoming stops, traffic information, remaining travel time, etc. Visually impaired people may be interested in this to provide context for their journey. The information may include information about any noteworthy maneuvers, such as sudden stops, sudden or sharp turns, lane changes, speed changes, etc. This information originates from the drive control source 202 and may be a filtered version of the control command when the vehicle acceleration indicated by, for example, an inertial measurement unit exceeds a certain threshold. Other information obtained from the drive control source 202 may be vehicle status information, such as fuel / battery levels, horn usage, signals, etc. The vehicle context data 216 may include perception information, such as information obtained from the external camera 204. The perception information may be based on a labeled version of the external environment perceived by the sensor system 28 of the vehicle 10 after processing by one or more machine learning recognition algorithms, which may be included in the computer vision system 74. Other sources of environmental data 212 may include sources of weather information, which may be advantageously explained to passengers, particularly when there are changes in the weather.

[0059] The vehicle context interpretation submodule 218 may parse the vehicle context data 216 into information. The vehicle context data 216 may only provide information for a limited upcoming time or distance window to ensure relevance. Alternatively, the vehicle context interpretation submodule 218 may filter the received information to an upcoming time or distance limited window. For example, information related to an upcoming 200-foot window (which may be a certain time period based on the current speed) may be processed. The vehicle context interpretation submodule 218 may algorithmically process the vehicle context data 216 into discrete messages contained in message data 238, which are sent to the passenger output system 224. Exemplary messages may include: "Entering [name of town / city]", "Bank name on the right", "Swerve to avoid vehicle ahead", "Hail storm", "Due to traffic, arrival time has been delayed by [X] minutes", "Turn left", "Change lanes", "On the highway", "Icy conditions", "Stop at traffic light", "Parking lot on the right", etc. Any information that may be sensed, known, or derived from various data sources regarding the external environment and actions taken by the vehicle may be condensed into a relevant message and sent to the occupant output system 224 in the message data 238 .

[0060] In an exemplary embodiment, a passenger location submodule 222 is provided to locate a visually or hearing-impaired passenger relative to the vehicle 10. The passenger may be located outside or inside the vehicle 10. The passenger location submodule may receive an indication that the passenger (or potential passenger) is hearing or visually impaired. This indication may be provided in the form of a notification from an application on the passenger device 240, or may be provided as described in reference to FIG. Figure 4 The occupant context data 214 may be provided as part of the occupant context data 214. The occupant context data 214 may additionally include information that enables the location of a person who is hearing or visually impaired to be determined. The information may be a transmission signal from the passenger device 240 that allows triangulation (or other processing algorithms to locate the device based on the received signal) by a signal receiver (and associated processing software) included in the connection system 36 of the vehicle 10. The transmission signal may be Bluetooth, ultra-wideband, etc. Alternatively, once the passenger is identified based on the signal from the passenger device 240, image processing may be performed to roughly locate the passenger within the image data using a projection process from real-world space to image space, which requires knowledge of the camera inherent data. The image data may be provided by an external camera 204 or an internal camera 206 and as part of the occupant context data 214. Image processing may then be used to refine the rough estimate of the passenger within the image data to identify the passenger within the image. By performing the inverse process of the projection from image space to real-world space, the precise coordinates of the passenger's position may be obtained. The coordinate system may be in the reference system of the vehicle 10. Regardless of the particular algorithm used to locate the passenger relative to the vehicle, three-dimensional (3D) coordinates of the passenger are generated by the passenger location submodule 222 and sent to the passenger guidance submodule 220. Other data modalities may be included to determine the position of the passenger, such as processing lidar, radar, or other imaging modalities to locate the passenger relative to the vehicle.

[0061] The passenger guidance submodule 220 receives the 3D coordinates of the passenger's position relative to the vehicle and accesses the vehicle model 242, which includes information about the door position and seat position. The passenger guidance submodule 220 is therefore able to use the precise 3D position of the passenger relative to the vehicle and the detailed vehicle model 242 to generate a detailed guidance message. The guidance message can describe the actions to be taken to get on and off the vehicle 10. For example, the number of steps required, the direction of the steps, where to go to find any vehicle features (e.g., door handles, seat belts, etc.), etc. In addition, the passenger positioning submodule 222 can track the position of the passenger in real time so that the passenger movement guidance can be dynamically updated by the passenger guidance submodule 220. The guidance message can be included in the message data 238 and output to the passenger output system 224.

[0062] In an exemplary embodiment, the passenger output system 224 includes various output devices, including a tactile device for providing tactile feedback to the passenger, an audio output device 244 (e.g., a speaker, a headset, earphones, etc.) for providing audio information to the passenger, and a visual display device 228 for displaying information to the passenger. The various output devices can be included as part of the passenger device 240 or as part of the vehicle (e.g., a vehicle speaker or a vehicle display). The passenger device 240 can be a laptop, a tablet, a smartphone, or any other portable computer device. The passenger output system 224 includes a neural machine translation module 230, an output modality decision module 236, an auxiliary determination module 232, and a sign language animation generation module 234. The various modules can be included in an application on the passenger device 240, or as part of the controller 34 of the vehicle 10.

[0063] The assistance determination module 232 is configured to determine whether the passenger requires special assistance for the visually or hearing impaired, and what type of assistance is required. The assistance determination module 232 may be based on reference Figure 4 The automatic detection process described may make such a determination based on information included in the passenger device 240 or based on information included in the database of passenger profiles 246. For example, the passenger device 240 may include a status notification that is communicated internally to the passenger output system 224 when the passenger output system 224 is part of the passenger device 240, or communicated to the passenger output system 224 via wireless transmission when the passenger output system 224 is part of the vehicle 10. The status notification may be a setting of the passenger device 240 selected by the passenger. In another example, the passenger device 240 may transmit the passenger identifier 248 to the passenger output system 224 included in the vehicle. The passenger output system 224 may retrieve a record corresponding to the passenger identifier 248 from the database of passenger profiles 246 to determine whether assistance is needed and what type of assistance is needed. In another example, the passenger profile database 246 may include biometric data identifying the passenger in each record. The passenger output system 224 may access the interior camera 206 of the vehicle 10 or the camera of the passenger device 240 to obtain biometric data of each passenger in the vehicle to see if a record is included in a database of passenger profiles 246 indicating whether special assistance can be provided and what type of special assistance can be provided. The passenger profile database 246 may be stored in a data storage device of the vehicle 10 (e.g., information about Figure 1 The data storage device 32 described above, the storage device of the passenger device 240, or the cloud storage.

[0064] The assistance determination module 232 provides information about the required assistance type to the output mode decision module 236, and the output mode decision module 236 sets the output mode to be provided to each passenger (if any). The output form can be provided with different complexity. In one embodiment, the output mode is set to visually impaired or hearing impaired. In another embodiment, the output mode can include more detailed passenger preferences (obtained from the passenger profile database 246). For example, a hearing-impaired person may prefer text messages rather than sign language messages. If the visually impaired person is not completely lost, they may prefer large text rather than sound. In another example, the hearing or visually impaired preference can be selected from different levels of details in the message. For example, a hearing-impaired person may not want detailed information about the external scene, and different visually impaired people may prefer different levels of details.

[0065] The neural machine translation module 230 may convert the messages included in the message data 238 into the passenger's preferred language, which may be set in the database of the passenger profile 246. The passenger output system 224 may select a default preferred language based on the language of the country in which the vehicle 10 is sold.

[0066] When the output modality is set for the visually impaired, the translated (if necessary) message from the neural machine translation module 230 may be converted to speech (using text-to-speech processing) and provided to the audio output device 244. When the output modality is set for the hearing impaired, the translated message may be provided to the sign language animation generation module 234. As described above, the output message may not be all of the messages included in the message data 238, as user preferences and different assistance requirements between hearing and visually impaired persons may result in filtering the provided messages to ensure relevance.

[0067] The sign language animation generation module 234 receives the translated (if necessary) message and generates an animated form of sign language to represent the entire message. The sign language used (such as American Sign Language, British Sign Language, Chinese Sign Language, German Sign Language, etc.) can depend on the preferred message language according to the passenger profile database 246. The generated sign language animation is output to the passenger on the visual display device 228.

[0068] It should be understood that the functionality of the vehicle context interpretation submodule 218 and the passenger guidance submodule 220 need not necessarily all be provided. That is, both functions are useful independently. Thus, a system can be provided that either provides interpretation of vehicle motion and the external environment, or provides movement guidance to passengers in and around the vehicle 10. However, the combination of these functions provides enhanced utility for people with visual and hearing impairments.

[0069] See also Figure 4According to one embodiment, the detection system 300 is used to automatically detect whether any passenger of the vehicle 10 has visual or hearing impairment. The detection system 300 includes a passenger monitoring system 302, and the passenger monitoring system 302 includes a camera 304, a sound generator 306, a passenger identification module 308, and an image analysis module. The camera 304 can be Figure 3 The camera 304 provides an image data frame (ie, video data) of each passenger to the image analysis module 310 for analysis to determine whether the passenger has hearing or visual impairment.

[0070] The passenger identification module 308 determines whether the communication system 200 already knows the passenger of the vehicle 10. Specifically, the passenger identification module 308 may receive the passenger identifier 248 from the passenger device 240, which may be used to look up the database of passenger profiles 246 to see if a record already exists. Alternatively, the passenger identification module 308 may receive image data captured by the passenger's camera 304 and perform facial or other biometric recognition processing thereon. The biometric data may then be compared to the biometric data included in the passenger profile database to see if the passenger is already known to the communication system 200. If a record already exists, the passenger's output modality may be determined without automatic detection, and the process proceeds to output a message according to the stored output modality, such as reference. Figure 3 If a record does not exist, the process moves on to analyzing the image data from camera 304 to automatically detect vision or hearing impairment.

[0071] The image analysis module receives image data frames from the camera 304 and can perform various algorithmic processes to determine whether the passenger is visually or hearing impaired. In one process, eye movements are extracted from the image data, which can indicate a visually impaired person. In another process, the sound generator 306 provides sound, and the image data is analyzed to measure any reaction to the sound. In one example, the sound is directional (e.g., from one side of the stereo speaker system of the vehicle 10), and the head movement toward the sound is detected by analysis. Sound-based detection can indicate a hearing-impaired person. In another example, the lights can be activated in the vehicle 10. For example, a flash can be used. The image data can be analyzed to measure any reaction to the light, which can indicate whether a person is visually impaired. In another example, the image analysis module 310 can detect visually impaired people by analyzing how the passengers move around the car, which may be different from non-visually impaired people. A combination of these techniques can also be used. In one embodiment, a machine learning algorithm is included in the image analysis module 310, which is trained based on a reserve of labeled (e.g., hearing-impaired, visually impaired, non-hearing-impaired, non-visually impaired) input image data. Although the present embodiment has been described in the context of image analysis, other perceptual modalities are also applicable. For example, lidar data may be provided in addition or in lieu of image data, which may be monitored (processed) for characteristic motion of a visually impaired person or in response to motion of generated sound of a hearing impaired person.

[0072] The detection of a visually or hearing impaired person by the image analysis module 310 may be sent to the occupant output system 224 so that a message is provided to the occupant according to the appropriate output modality. In addition, the detection result may be stored in a database of the occupant profile along with the occupant identifier 248 or biometric data extracted from the image data captured by the camera 304.

[0073] Understandably, Figure 4 The detection system is 300 Figure 3 The communication system 200 is more suitable. That is, Figure 4 The automatic detection system 300 has utility in various applications where automatic detection of hearing or visually impaired persons is required. The output modality may be set based on the automatic detection, although the content of the output may be of various types and need not be limited to the content of the output modality provided by the user. Figure 3 Message generated by the communication system 200.

[0074] In an embodiment of the present disclosure, the communication system 200 stores the user's preferred communication mode (including audio / sign language animation and language preference) in the passenger profile database 246. The communication system 200 translates and converts any notifications from the vehicle 10 into the user's preferred communication mode. The communication system 200 uses wireless communication to accurately determine the user's position relative to the vehicle 10. The communication system 200 provides user guidance for approaching, entering, exiting, and leaving the vehicle 10 based on wireless communication and / or sensory information. Therefore, the communication system 200 is able to utilize the user's saved communication mode to provide information from the vehicle 10 to the user in a personalized manner. The communication system 200 is also able to utilize connectivity and sensory technologies to provide precise positioning information.

[0075] In additional or alternative embodiments of the present disclosure, the communication system 200 combines environmental information and vehicle action information to create a contextual interpretation of the action for output. The communication system 200 uses neural machine translation to generate audio in a selected auditory language, uses neural machine translation to generate sign language in the selected sign language, then generates sign language animation (and / or text output), and finally outputs the generated audio and sign language animation. Therefore, the present disclosure enables hearing and visually impaired passengers to understand the actions taken by an automated driver assistance system (ADAS) or a more fully autonomous vehicle by combining relevant environmental details with interpretations of the actions of the vehicle 10, and provides this information in a form that is targeted to hearing and visually impaired individuals.

[0076] In additional or alternative embodiments of the present disclosure, the communication system 200 identifies the appropriate communication modality with the individual by identifying irregular eye movements, monitoring reactions to the generated tones, and storing the modality for later retrieval. Thus, hearing or visually impaired individuals can be identified without requiring prior interaction with the individual to initiate communication through an alternative modality appropriate for that individual.

[0077] Reference now Figure 5 and 6 , and continue to refer to Figure 1-4 , the flowchart shows that according to the present disclosure, the Figure 1-4 The communication system 200 described herein performs the method 500 and 600. It can be understood from the present disclosure that the order of operations in the method is not limited to the following. Figure 5 and Figure 6 The methods 500, 600 are not performed in the order shown, but may be performed in one or more different orders as applicable and in accordance with the present disclosure. In various embodiments, the methods 500, 600 may be scheduled to run based on one or more predetermined events, and / or may run continuously during operation of the vehicle 10. Exemplary predetermined events may be each time a passenger is stopped to be picked up, each time the vehicle 10 is started, each time a new passenger (potential passenger) approaches the vehicle, or each time a door of the vehicle 10 is opened.

[0078] Figure 5 A method of communicating with a visually or hearing impaired person according to a preferred output modality is shown. In step 502, vehicle and / or passenger context data 214, 216 are received from various data sources of the communication system 200. The vehicle context data 216 may include environmental information about the scene outside the vehicle 10 and information about actions taken by the vehicle 10 (e.g., resulting from an autonomous driving command). The occupant context data 214 may specifically relate to information that enables the precise 3D position of potential occupants to be determined.

[0079] At step 504, message data 238 is generated. Message data 238 may include interpretations of vehicle actions and / or external environment based on vehicle context data 216. Message data 238 may also or alternatively include motion guidance based on occupant context data 214 to help hearing or visually impaired people board the vehicle, find a seat, fasten their seat belts, and / or exit the vehicle 10.

[0080] In step 506, it is determined whether the passenger is hearing or visually impaired. This can be performed by looking up a database of passenger profiles 246 or by automatic detection, such as by referring to Figure 6 Described. Depending on whether the passenger is hearing or visually impaired, the appropriate output mode is set. In some embodiments, passengers can also save the preferred standards for the language of the message and the sign language in their preference storage in the passenger profile database.

[0081] In step 508, an output is generated for the passenger according to the set output modality and the preferred language of the message and the sign language standard (e.g., American Sign Language). The neural machine translation module 230 can translate the message into the preferred language. In the case of a hearing-impaired person, the sign language animation generation module 234 can convert the message into the desired type of animated sign language and output it through the visual display device 228. In the case of a visually impaired person, the translated message can be converted into speech and output through the audio output device 244.

[0082] refer to Figure 6, an exemplary method 600 for automatically detecting visually and hearing impaired passengers in a vehicle 10. In step 602, frames of image data (e.g., video data) are received from a camera 304. In step 604, the image data is analyzed to determine whether a passenger of the vehicle 10 is hearing or visually impaired. In one embodiment, the analysis includes processing by a machine learning algorithm that has been trained to detect visually impaired and hearing impaired passengers. The analysis may include detecting irregular eye movements, which may indicate a visually impaired person. The analysis may include irregular movements inside the vehicle cab, which may indicate a visually impaired person. The analysis may include detecting irregular or absent responses to sounds produced by an audio output device 244, which may indicate a hearing impaired person. Whether the passenger is hearing or visually impaired will indicate the appropriate output modality. In step 606, messages, notifications, suggestions, etc. are output according to the set output mode. When the passenger is visually impaired, the output of step 606 may be processed via speech to text and audio is output through the audio output device 244. When the passenger is hearing impaired, the output of step 606 may be achieved by displaying a sign language animation (and / or text output) on the visual display device 228 .

[0083] Although at least one exemplary embodiment has been introduced in the foregoing detailed description, it should be understood that there are a large number of variations. It should also be understood that one or more exemplary embodiments are merely examples and are not intended to limit the scope, applicability, or configuration of the present disclosure in any way. On the contrary, the foregoing detailed description will provide a convenient roadmap for those skilled in the art to implement one or more exemplary embodiments. It should be understood that various changes may be made to the functions and arrangements of the elements without departing from the scope of the present disclosure as set forth in the attached claims and their legal equivalents.

Claims

1. A system associated with a vehicle, comprising: an occupant output system including at least one output device; at least one vehicle context data source; and at least one processor in operable communication with the occupant output system and the at least one vehicle context data source, wherein the at least one processor is configured to execute program instructions, wherein the program instructions are configured to cause the at least one processor to: receiving vehicle context data from at least one vehicle context data source; determining whether an occupant associated with the occupant output system is at least one of visually and auditorily impaired; generating occupant message data based on the vehicle context data; When it is determined that the occupant is at least one of visually and hearing impaired, determining an output modality to assist the occupant, wherein when it is determined that the occupant is hearing impaired, the output modality is at least one of sign language animation and text data on a visual display device of the at least one output device, and wherein when it is determined that the occupant is visually impaired, the output modality is an audio output on an audio output device of the at least another output device; and generating an output for the passenger on at least one output device based on the determined output modality and based on the passenger message data, in: The vehicle context data includes a passenger position relative to the vehicle and a passenger identifier; determining the occupant location and occupant identifier based on at least one of: wireless communications and processing of sensory data from an occupant device; The occupant message data includes guidance regarding at least one of approaching, entering, exiting, and leaving the vehicle determined based on the occupant's position; and A determination is made based on the occupant identifier whether an occupant associated with the occupant output system is at least one of visually and hearing impaired.

2. The system according to claim 1, wherein: The vehicle context data includes images or other sensory data from the scene outside the vehicle. 3 . The system of claim 1 , wherein the vehicle context data includes a position of an occupant relative to the vehicle.

4. The system according to claim 3, wherein: The program instructions are configured to cause the at least one processor to determine the position of the occupant relative to the vehicle based on at least one of: one or more transmissions from occupant equipment; and Processing of sensory data. 5 . The system of claim 1 , wherein the vehicle context data includes vehicle behavior information based on vehicle control commands.

6. The system according to claim 1, characterized in that The vehicle context data includes map data.

7. The system according to claim 1, wherein: The occupant message data includes information describing the environment outside the vehicle.

8. The system according to claim 1, characterized in that The passenger message data includes guidance for passengers to board or alight from the vehicle or any other passenger behavior related to the vehicle.

9. The system according to claim 1, characterized in that The occupant message data includes a description of an action taken by the vehicle.

10. A method for controlling an occupant output system associated with a vehicle, the method comprising: receiving, by at least one processor, vehicle context data from at least one vehicle context data source; determining, via the at least one processor, whether an occupant associated with the occupant output system is at least one of visually and auditorily impaired; generating, via the at least one processor, occupant message data based on the vehicle context data; determining, by at least one processor, an output modality to assist the occupant when it is determined that the occupant is at least one of visually and hearing impaired, wherein when it is determined that the occupant is hearing impaired, the output modality is at least one of a sign language animation and text data on a visual display device of at least one output device, and wherein when it is determined that the occupant is visually impaired, the output modality is an audio output on an audio output device of at least one input device; and Based on the output modality and the occupant message data, an output is generated by at least one processor for the passenger on at least one output device of the passenger output system. in: The vehicle context data includes a passenger position relative to the vehicle and a passenger identifier; determining the occupant location and occupant identifier based on at least one of: wireless communications and processing of sensory data from an occupant device; The occupant message data includes guidance regarding at least one of approaching, entering, exiting, and leaving the vehicle determined based on the occupant's position; and A determination is made based on the occupant identifier whether an occupant associated with the occupant output system is at least one of visually and hearing impaired.

Images