Language detection system for vehicle

Abstract

The language detection system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include issuing a vehicle cue and determining responsiveness to the issued vehicle cue, the responsiveness including an execution response and a non-responsive response. A language monitoring module is activated in response to a non-responsive response of the determined responsiveness and a spoken language is monitored within an interior cabin of the vehicle. A driver state is estimated based on the determined language, and a translation function is performed using a translation module in response to the estimated driver state.

Description

Technical Field

[0001] The information provided in this section is for the purpose of generally presenting the context of the present disclosure. The work of the presently named inventors is neither explicitly nor implicitly admitted as prior art to the present disclosure to the extent described in this section and in aspects of the description that may not have otherwise been identified as prior art at the time of filing.

[0002] The present disclosure generally relates to a language detection system for a vehicle. Background Art

[0003] Vehicles often incorporate assisted driving features. The assisted driving features may be configured with a voice function to provide assistance to the driver of the vehicle. The voice function may be programmed using a predetermined language as part of the vehicle controller. In some scenarios, the voice function may be used to guide the driver through the operation of the vehicle. The vehicle's controller may assess the driver's readiness to perform an operation that may be performed using a predetermined language. In some cases, the predetermined language of the controller may be different from the driver's primary language. Summary of the invention

[0004] In some aspects, a computer-implemented method, when executed by data processing hardware, causes the data processing hardware to perform operations. The operations include issuing a vehicle prompt and determining a responsiveness to the issued vehicle prompt. The responsiveness includes performing a response and a non-responsive response. A language monitoring module is activated in response to the determined responsiveness and non-responsiveness response. Spoken language is monitored within an interior cabin of the vehicle, and a language of the monitored spoken language is determined. A driver state is estimated based on the determined language, and a translation function is performed using a translation module in response to the estimated driver state.

[0005] In some examples, the language of the monitored spoken language may be determined by receiving a language translation from a server. The translation function may be performed by updating the vehicle prompt with the language translation. In some examples, the translation function may be performed by providing on-board training using the language translation. Operations may include determining the severity of the vehicle event and issuing an auxiliary vehicle prompt using the determined language. A communication server may be contacted, and the translation service may be performed via the communication server. The severity of the vehicle event may be determined through the collected in-cabin data.

[0006] In other aspects, a language detection system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include issuing a vehicle prompt and determining a responsiveness to the issued vehicle prompt, the responsiveness including an execution response and a non-responsive response. A language monitoring module is activated in response to the determined responsiveness and a non-responsive response, and a spoken language is monitored within an interior cabin of the vehicle. A driver state is estimated based on the determined language, and a translation function is performed using a translation module in response to the estimated driver state.

[0007] In some examples, the language of the monitored spoken language may be determined by receiving a language translation from a server. The translation function may be performed by updating the vehicle prompt with the language translation. In some examples, the translation function may be performed by providing on-board training using the language translation. Operations may include determining the severity of the vehicle event and issuing an auxiliary vehicle prompt using the determined language. A communication server may be contacted, and the translation service may be performed via the communication server. The severity of the vehicle event may be determined through the collected in-cabin data.

[0008] In other aspects, a computer-implemented method causes data processing hardware to perform operations when executed by data processing hardware. The operations include issuing a vehicle prompt and determining a responsiveness to the issued vehicle prompt. The responsiveness includes executing a response and a non-responsive response. Spoken language is monitored within an interior cabin of the vehicle, and a language monitoring module is activated in response to the monitored spoken language and the determined responsiveness (including a non-responsive response). A driver state is estimated based on the determined language, and a language translation is generated on a server. The operations also include executing a translation module using the generated language translation in response to the estimated driver state.

[0009] In some examples, generating a language translation may include utilizing artificial intelligence to generate a language translation. The driver state may be estimated by collecting in-cabin data. In some examples, as part of performing the translation function, the generated language translation may be used to display one or more auxiliary vehicle prompts. In some configurations, the generated language translation may be used to provide in-vehicle training. The estimated driver state may include an estimated emergency state, and a third-party server may be contacted as part of the performed translation function. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The drawings described herein are for illustrative purposes only of selected configurations and are not intended to limit the scope of the present disclosure.

[0011] Figure 1 is a partial perspective view of the interior of a vehicle according to the present disclosure;

[0012] Figure 2 is a partial perspective view of the interior of a vehicle having a speech detection system according to the present disclosure;

[0013] Figure 3 is a functional block diagram of a language detection system according to the present disclosure;

[0014] Figure 4 is an example flow chart of a language detection system according to the present disclosure that monitors spoken language in a cabin;

[0015] Figure 5 yes Figure 4 An example flow chart of a language detection system for detecting a language type and a driver status;

[0016] Figure 6 yes Figure 5 An example flow chart of a language detection system utilizing language translation technology;

[0017] Figure 7 yes Figure 6 An example flow chart of a language detection system for updating a vehicle language;

[0018] Figure 8 yes Figure 6 An example flow chart of a language detection system for connecting a driver to a translator; and

[0019] Fig. 9 yes Figure 6 An example flow chart of a language detection system for detecting events based on a driver state.

[0020] Corresponding reference characters indicate corresponding parts throughout the several views of the drawings. DETAILED DESCRIPTION

[0021] Example configurations will now be described more fully with reference to the accompanying drawings. Example configurations are provided so that the present disclosure will be thorough and will fully convey the scope of the present disclosure to those of ordinary skill in the art. Specific details, such as examples of specific components, devices, and methods, are set forth to provide a thorough understanding of the configurations of the present disclosure. It will be apparent to those of ordinary skill in the art that specific details need not be employed, that example configurations may be embodied in many different forms, and that the specific details and example configurations should not be construed as limiting the scope of the present disclosure.

[0022] The terms used herein are only used for the purpose of describing specific exemplary configurations and are not intended to be limiting. As used herein, the singular articles "a", "an" and "the" may also be intended to include plural forms unless the context clearly indicates otherwise. The terms "comprises", "comprising", "including" and "having" are inclusive and therefore specify the presence of features, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, steps, operations, elements, parts and / or groups thereof. The method steps, processes and operations described herein should not be interpreted as necessarily requiring them to be performed in the specific order discussed or shown, unless specifically identified as an execution order. Additional or alternative steps may be adopted.

[0023] When an element or layer is referred to as being "on another element or layer," "engaged to," "connected to," "attached to," or "coupled to" another element or layer, it may be directly on, directly engaged with, connected to, attached to, or coupled to another element or layer, or there may be intermediate elements or layers. Conversely, when an element is referred to as being "directly on another element or layer," "directly engaged to," "directly connected to," "directly attached to," or "directly coupled to" another element or layer, there may be no intermediate elements or layers. Other words used to describe the relationship between elements should be interpreted in a similar manner (e.g., "between" versus "directly between," "adjacent" versus "directly adjacent," etc.). As used herein, the term "and / or" includes any and all combinations of one or more associated listed items.

[0024] The terms "first", "second", "third", etc. can be used here to describe various elements, components, regions, layers and / or parts. These elements, components, regions, layers and / or parts should not be limited by these terms. These terms can only be used to distinguish an element, component, region, layer or part from another region, layer or part. Unless the context clearly indicates, terms such as "first", "second" and other numerical terms do not imply order or sequence. Therefore, without departing from the teaching of the example configuration, the first element, component, region, layer or part discussed below can be referred to as the second element, component, region, layer or part.

[0025] In this application, including the definitions below, the term "module" may be replaced with the term "circuit". The term "module" may refer to, be part of, or include: an application specific integrated circuit (ASIC); a digital, analog, or mixed analog / digital discrete circuit; a digital, analog, or mixed analog / digital integrated circuit; a combinational logic circuit; a field programmable gate array (FPGA); a processor (shared, dedicated, or grouped) that executes code; a processor (shared, dedicated, or grouped) that stores code executed by a memory; other suitable hardware components that provide the described functionality; or a combination of some or all of the above, such as in a system on a chip.

[0026] The term "code" as used above may include software, firmware and / or microcode, and may refer to a program, a routine, a function, a class and / or an object. The term "shared processor" covers a single processor that executes some or all codes from multiple modules. The term "group processor" covers a processor that executes some or all codes from one or more modules in combination with an additional processor. The term "shared memory" covers a single memory that stores some or all codes from multiple modules. The term "group memory" covers a memory that stores some or all codes from one or more modules in combination with an additional memory. The term "memory" may be a subset of the term "computer-readable medium". The term "computer-readable medium" does not cover transient electrical signals and electromagnetic signals propagated through the medium, and therefore can be considered to be tangible and non-transient memory. Non-limiting examples of non-temporary memory include tangible computer-readable media, which include non-volatile memory, magnetic memory, and optical memory.

[0027] The devices and methods described in this application may be implemented in part or in whole by one or more computer programs executed by one or more processors. The computer programs include processor-executable instructions stored on at least one non-transitory tangible computer-readable medium. The computer programs may also include and / or rely on stored data.

[0028] A software application (i.e., software resource) may refer to computer software that enables a computing device to perform tasks. In some examples, a software application may be referred to as an "application," "app," or "program." Example applications include, but are not limited to, system diagnostic applications, system management applications, system maintenance applications, word processing applications, spreadsheet applications, messaging applications, media streaming applications, social networking applications, and gaming applications.

[0029] Non-transitory memory can be a physical device used to temporarily or permanently store programs (e.g., sequences of instructions) or data (e.g., program state information) for use by a computing device. Non-transitory memory can be volatile and / or non-volatile addressable semiconductor memory. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., commonly used for firmware, such as bootloaders). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0030] These computer programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in high-level procedural and / or object-oriented programming languages ​​and / or assembly / machine languages. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., disk, optical disk, processor, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0031] Various implementations of the systems and techniques described herein can be realized in digital electronic and / or optical circuits, integrated circuits, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementations in one or more computer programs executable and / or interpretable on a programmable system that includes at least one programmable processor, which may be special-purpose or general-purpose, coupled to receive data and instructions from a storage system, at least one input device, and at least one output device, and to send data and instructions to the storage system, at least one input device, and at least one output device.

[0032] The process and logic flow described in this specification can be performed by one or more programmable processors (also referred to as data processing hardware), which execute one or more computer programs to perform functions by operating on input data and generating outputs. The process and logic flow can also be performed by a dedicated logic circuit (e.g., FPGA (field programmable gate array) or ASIC (application-specific integrated circuit)). As an example, a processor suitable for executing a computer program includes both general-purpose and special-purpose microprocessors, and any one or more processors of any type of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Typically, a computer will also include one or more large-capacity storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data, or be operably connected to receive data from it or transmit data to it or both. However, a computer does not need to have such a device. Computer-readable media suitable for storing computer program instructions and data include all forms of nonvolatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0033] To provide interaction with a user, one or more aspects of the present disclosure may be implemented on a computer having a display device (e.g., a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen) for displaying information to the user and optionally a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other kinds of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in any form, including sound, voice, or tactile input. In addition, the computer may interact with the user by sending documents to and receiving documents from a device used by the user; for example, by sending a web page to a web browser on a user's client device in response to a request received from the web browser.

[0034] refer to Figure 1-9, the speech detection system 10 is integrated with the vehicle 100. The speech detection system 10 is configured to interconnect the vehicle processor 12 and the server 200, as described below. The vehicle processor 12 is configured to execute the speech monitoring module 14 described herein. The vehicle 100 can be equipped with a sensor array 102, which is configured to capture audio signals 16 within the interior cabin 104 of the vehicle 100.

[0035] The vehicle processor 12 includes data processing hardware 18 and memory hardware 20 in communication with the data processing hardware 18. The data processing hardware 18 is configured to receive the audio signal 16 from the sensor array 102. It is generally contemplated that the language detection system 10 includes a computer-implemented method that is executed by the data processing hardware 18 and causes the data processing hardware 18 to perform the various operations described herein. Additionally or alternatively, the memory hardware 20 may store the computer-implemented method as instructions that, when executed on the data processing hardware 18, cause the data processing hardware 18 to perform the operations described herein.

[0036] refer to Figure 1-3 As described herein, the speech monitoring module 14 is configured to monitor spoken language within the interior cabin 104 of the vehicle 100 via the sensor array 102. For example, the audio signal 16 may include spoken language 16a detected from within the vehicle 100 and may be communicated to the vehicle processor 12 as sensor data 106. The sensor data 106 also includes in-cabin data 108 and operational data 110. In addition to the audio signal 16, the in-cabin data 108 may also include image data that may be captured via an image sensor of the sensor array 102. The operational data 110 may include movement of the steering wheel, propulsion of the vehicle 100, lane changes, and any other operational functions of the vehicle 100. The operational data 110 may also be detected by the sensor array 102. Although the operational data 110 may be detected by the sensor array 102, it is also contemplated that the vehicle processor 12 may independently determine at least some of the operational data 110 in addition to the operational data 110 received from the sensor array 102.

[0037] As described above, the in-cabin data 108 includes the audio signal 16 captured by the sensor array 102. The sensor array 102 may include, but is not limited to, a microphone array that captures the audio signal 16 and transmits the audio signal 16 to the language detection system 10. The sensor array 102 receives the audio signal 16 and transmits the audio signal 16 to the vehicle processor 12. In response to receiving the audio signal 16, the vehicle processor 12 may initiate the language monitoring module 14 and may, for example, execute a computer-implemented method of the language detection system 10.

[0038] The vehicle processor 12 includes an in-cabin assessment system 22 that is configured to execute the language monitoring module 14. The in-cabin assessment system 22 is also configured with a translation module 24. The translation module 24 is described in more detail below, and receives the audio signal 16 monitored by the language monitoring module 14. The in-cabin assessment system 22 can communicate a language translation 26 with an occupant and / or driver 112 of the vehicle 100. The vehicle processor 12 is configured to issue various vehicle prompts 28 during operation of the vehicle 100. As described herein, the language monitoring module 14 monitors spoken language 16a within the vehicle 100, and the language detection system 10 is configured to utilize the language translation 26 to reconfigure the vehicle prompts 28 to improve communication with the driver 112.

[0039] Still reference Figure 1-3 , the language detection system 10 interconnects the vehicle processor 12 with the server 200. The server 200 includes a communication server 202 and a translation module 204. The language detection system 10 is configured to utilize the translation module 204 of the server 200 if the translation confidence 30 of the translation module 24 of the in-cabin assessment system 22 is below a confidence threshold 32. For example, the confidence threshold 32 may be approximately ninety-five percent (95) confidence to utilize the in-cabin assessment system 22 to translate the vehicle prompt 28. The language detection system 10 is configured to determine the translation confidence 30 based on the sensor data 106 to determine the responsiveness of the translated vehicle prompt 28.

[0040] When the language detection system 10 determines that the translation confidence 30 of the in-cabin assessment system 22 is below the confidence threshold 32, the language detection system 10 utilizes the translation module 204 of the server 200. The translation module 204 of the server 200 can utilize a generative artificial intelligence (AI) 206 to generate a language translation 208. The translation module 204 receives the audio signal 16, and the generative AI 206 uses the generated language translation 208 to generate a translated vehicle prompt 28. In some cases, the server 200 can also utilize the communication server 202 to connect a third-party translator 210 with the driver 112 of the vehicle 100 to help deliver the vehicle prompt 28, as described in more detail below.

[0041] Further references Figure 1-3, the language detection system 10 may also interconnect the server 200 with a third-party server 300. The third-party server 300 is configured to provide emergency assistance to the occupant 112 of the vehicle 100. The server 200 may provide the generated language translation 208 to the third-party server 300 to facilitate communication between the third-party server 300 and the occupant 112. The language detection system 10 is configured to distinguish between the languages ​​spoken within the interior cabin 104 of the vehicle 100, so that the language detection system 10 can identify the location of each respective speaker within the vehicle 100.

[0042] For example, the language detection system 10 is configured to monitor the audio signal 16 and identify the speaker, which may be the driver 112. The language detection system 10 may also distinguish between multiple speakers 112 speaking simultaneously to identify spoken language 16a from the audio signal 16. The language detection system 10 may also identify the spoken language 16a if the speaker 112 is utilizing a voice call feature of the vehicle 100 or otherwise utilizing a voice activated function within the vehicle 100. The language detection system 10 interconnects the vehicle processor 12, the server 200, and in some cases, the third party server 300 to maximize the translation assistance provided to the speaker 112.

[0043] Reference now Figure 2-5 , the vehicle processor 12 is configured to issue various vehicle prompts 28 during operation of the vehicle 100. The vehicle prompts 28 may be issued via audio within the interior cabin 104 and / or via text on the infotainment device of the vehicle 100. The vehicle prompts 28 may include, but are not limited to, messages 28a, on-board training 28b, and / or instructions 28c. The vehicle processor 12 is configured to monitor the responsiveness 40 of the driver 112 to the vehicle prompts 28 via the language monitoring module 14 and the in-cabin assessment system 22. The responsiveness 40 may include an execution response 40a and a non-response response 40b. If the vehicle processor 12 detects an execution response 40a, the language detection system 10 may continue to monitor the vehicle processor 12 and operate in the background of the vehicle processor 12. However, if a non-response response 40b is detected, the language detection system 10 may trigger the vehicle processor 12 to execute the in-cabin assessment system 22.

[0044] It is generally contemplated that the language monitoring module 14 of the vehicle processor 12 and the language detection system 10 as a whole are operable during operation of the vehicle 100 regardless of the responsiveness 40. The language detection system 10 can collect data including the audio signal 16 for future use. For example, the processor hardware 20 of the vehicle memory 12 can store a language setting 42 that can be refined based on the background operation of the language detection system 10. The language setting 42 is based on the detected audio signal 16, which can be classified as a stored language 44, a new language 46, a language mix 48, and / or an unrecognizable language 50. The language setting 42 can be updated using the audio signal 16 and the translation module 24 of the in-cabin assessment system 22. In some examples, the language setting 42 can be updated by the server 200 via the translation module 204. In particular, when the audio signal 16 is classified as an unrecognizable language 50 by the in-cabin assessment system 22, the language detection system 10 can utilize either of the translation modules 24, 204.

[0045] When the vehicle processor 12 receives the audio signal 16, the vehicle processor 12 executes the language monitoring module 14. The language detection system 10 can utilize the language monitoring module 14 to identify the keyword 52. For example, the language monitoring module 14 can identify the keyword 52 in response to the vehicle prompt 28 to determine whether the occupant 112 understands the vehicle prompt 28. The keyword 52 can help the vehicle processor 12 determine the responsiveness 40 of the occupant 112. Therefore, the language monitoring module 14 can operate continuously during operation of the vehicle 100. The language detection system 10 can selectively execute the in-cabin assessment system 22 in response to the non-responsive response 40b to the vehicle prompt 28.

[0046] Continue to refer Figure 2-5 , the language detection system 10 can be activated in response to the non-responsive response 40b. In other examples, the language monitoring module 14 can run continuously in the background and can be actively activated in response to the non-responsive response 40b. The language monitoring module 14 is configured to detect spoken language and monitor keywords 52. The keywords 52 may include a request for help, or may be an indicator of confusion in response to the vehicle prompt 28. In some examples, the in-cabin assessment system 22 estimates the driver state 60 using the detected language from the language monitoring module 14. For example, the language monitoring module 14 identifies the spoken language 16a based on the language setting 42. The in-cabin assessment system 22 may also utilize non-verbal input from the in-cabin data 108 received as part of the sensor data 106 to further assist in estimating the driver state 60. Therefore, the driver state 60 can be estimated by collecting the in-cabin data 108.

[0047] The in-cabin assessment system 22 is configured to perform a translation based on one or more of the audio signal 16 and the keyword 52. In the event that the translation confidence 30 of the in-cabin assessment system 22 is low, the server 200 can be used to supplement and enrich the confidence probability of the in-cabin assessment system 22. Thus, the language detection system 10 can utilize a hybrid arrangement of the vehicle processor 12 and the server 200 operating in collaboration. It is further contemplated that in some examples, the server 200 can utilize crowdsourcing to further adjust and refine the confidence threshold 32 over time to continuously improve the language detection system 10. As described herein, the language detection system 10 ultimately provides an updated vehicle prompt 28 in a language determined to be understood by the occupant 112.

[0048] Reference Figures 2 to 6 , the language detection system 10 is configured to perform a translation function 62 in response to the estimated driver state 60. The translation function 62 may include a language translation 26 from the in-cabin assessment system 22 and / or the server 200. For example, the vehicle processor 12 may receive the language translation 26 from the server 200, which may be generated using the generation AI 206. The driver state 60 is estimated using various sensor data 106 to determine why the responsiveness 40 of the occupant 112 is a non-responsive response 40b. In some examples, the occupant 112 or the driver 112 may not understand the vehicle prompt 28. In response, the language detection system 10 may update the vehicle prompt 28 with the language translation 26 and issue the translated vehicle prompt 28.

[0049] In other examples, the language detection system 10 can use the language translation 26 to provide one or more secondary vehicle prompts 28. The secondary vehicle prompts 28 can include, but are not limited to, on-board training 28b in the language translation. For example, the on-board training 28b can help the driver 112 further understand the vehicle prompts 28 by also presenting the on-board training 28b in the language translation. As described above, the secondary vehicle prompts 28 can also utilize displayed messages 28a and / or instructions 28c that utilize language translation.

[0050] Reference again Figure 1-9 , the language detection system 10 may determine that the driver 112 has a non-responsive response 40b to the critical command issued as part of the vehicle prompt 28. In addition to projecting the vehicle prompt 28 in the translated language, the language detection system 10 may also display additional auxiliary vehicle prompts 28 in the translated language. If the language detection system 10 still determines a non-responsive response 40b after the additional vehicle prompts 28, the language detection system 10 initiates contact between the vehicle processor 12 and the communication server 202. The communication server 202 may connect a third-party translator 210 with the driver 112 to assist in delivering the information presented as part of the vehicle prompt 28.

[0051] The language detection system 10 uses the audio signal 16 and the language translation to identify the third party translator 210 who speaks the identified spoken language 16a. The server 200 is configured to connect the third party translator 210 via the communication server 202 to help the driver 112 understand the vehicle prompt 28. In some examples, the vehicle processor 12 is configured to determine the severity of the vehicle event 120, and can issue the auxiliary vehicle prompt 28 based on the severity of the vehicle event 120.

[0052] The severity of the vehicle event 120 may be determined at least in part by the in-cabin data 108 received from the sensor array 102. Based on the severity of the vehicle event 120, the language detection system 10 may contact the communication server 202 to perform translation services 212 via the third-party translator 210. In some cases, the severity of the vehicle event 120 may be determined to be an emergency event 120a. The language detection system 10 may contact the third-party server 300 via the server 200 to provide assistance to the driver 112.

[0053] In the example of emergency event 120a, language detection system 10 can communicate the detected language and translated language data with third-party server 300. Third-party server 300 can utilize the translated data to coordinate emergency services to help communicate with driver 112. In some examples, third-party server 300 can cooperate with communication server 202 to utilize third-party translator 210 to communicate with driver 112. Thus, server 200 can share the translated data with third-party server 300.

[0054] Still reference Figures 1 to 9 , with special reference Figures 4 to 9 An example of a computer-implemented method of the language detection system 10 is described. At 500, the vehicle processor 12 issues a vehicle prompt 28, and at 502, the vehicle processor 12 determines whether the driver 112 responds to the vehicle prompt 28. If the driver 112 responds, the language detection system 10 is not activated. If the driver 112 does not respond to the vehicle prompt 28, the vehicle processor 12 monitors the language spoken in the vehicle 100 at 504. At 506, the language monitoring module 14 is subsequently or simultaneously activated by the vehicle processor 12. At 508, the language monitoring module 14 detects the language, and at 510 monitors the keywords 52. At 512, the language monitoring module 14 determines whether the detected language is a stored language 44, a new language 46, a language mix 48, and / or an unrecognizable language 50. Similarly, at 514, the language monitoring module 14 may determine whether the detected keywords 52 include a request for help or detection confusion in response to the vehicle prompt 28.

[0055] At 516, the language detection system 10 can use the detected language and any potential keywords to estimate the driver state 60. At 518, the language detection system 10 can receive non-language input as part of the sensor data 106, which can also be used to estimate the driver state. At 520, the language detection system 10 can generate the language translation 208 using the generated AI 206, and at 522, the vehicle processor 12 can receive the language translation 208 from the server 200.

[0056] Specific reference Figure 7-9 , the language detection system 10 may perform different steps depending on the detected scenario. For example, the driver 112 may not understand the vehicle prompt 28 at 530, and the language detection system 10 may update the message 28a and the vehicle prompt 28 with the language translation 208 at 532. At 534, the language detection system 10 may also provide in-vehicle training 28b using the language translation 208. In other examples, at 540, the driver 112 may not understand the key command. At 542, the language detection system 10 may display additional auxiliary vehicle prompts 28 using the language translation 208. At 544, the language detection system 10 may also contact the communication server 202 of the third-party translator 210. In another example, at 550, the language detection system 10 may detect an emergency 120. At 552, the language detection system 10 may project the instruction 28c using the language translation 208, and at 554, the in-cabin data 108 may be collected from the sensor array 102. At 556 , the language detection system 10 may also contact the third party server 300 for assistance.

[0057] Further references Figures 1 to 9 , the language detection system 10 advantageously assists in monitoring and translating spoken language to assist the driver 112 or other occupants 112 within the vehicle 100. The language detection system 10 provides improved communication capabilities between the occupant 112 and the vehicle processor 12, such that the language detection system 10 can modify the pre-programmed language in response to a detected unresponsive response 40b to the vehicle prompt 28. Thus, it is contemplated that the functionality of the vehicle processor 12 can be provided to the occupant 112 and / or driver 112 regardless of potential differences in language. The language detection system 10 advantageously provides adaptive language assistance and translation during various assistance functions to maximize the utility of the vehicle assistance system.

[0058] Many embodiments have been described. However, it should be understood that various modifications can be made without departing from the spirit and scope of the present disclosure. Therefore, other embodiments are within the scope of the appended claims.

[0059] The foregoing description is provided for the purpose of illustration and description. It is not intended to be exhaustive or to limit the present disclosure. The individual elements or features of a particular configuration are generally not limited to that particular configuration, but are interchangeable where applicable and can be used in a selected configuration, even if not specifically shown or described. It can also be varied in many ways. Such variations should not be considered as departing from the present disclosure, and all such modifications are intended to be included within the scope of the present disclosure.

Claims

1. A language detection system for a vehicle, the language detection system comprising: Data processing hardware; as well as Memory hardware in communication with the data processing hardware, the memory hardware storing instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations comprising: Issue vehicle alerts; determining, by the processing hardware, a responsiveness to the issued vehicle prompt, the responsiveness comprising an executed response and a non-response response; activating a language monitoring module in response to the determined responsive non-responsive response; monitoring spoken language within an interior cabin of said vehicle; Determine the language of the oral language being monitored; estimating a driver state based on the determined language; and A translation function is performed in response to the estimated driver state. 2 . The language detection system of claim 1 , wherein determining the language of the monitored spoken language comprises receiving a language translation from a server. 3 . The language detection system of claim 2 , wherein performing the translation function comprises updating the vehicle prompt using the language translation.

4. The language detection system of claim 2, wherein performing the translation function comprises using the language translation to provide on-board training.

5. The language detection system of claim 1 further comprising determining a severity of the vehicle event and issuing an auxiliary vehicle prompt using the determined language. 6 . The language detection system of claim 5 , wherein determining the severity of the vehicle event includes contacting a communication server and performing a translation service via the communication server.

7. The language detection system of claim 5, wherein determining the severity of the vehicle event includes collecting in-cabin data.

8. The language detection system of claim 1, further comprising generating a language translation on a server.

9. The language detection system of claim 8, wherein generating the language translation comprises utilizing artificial intelligence to generate the language translation.

10. The language detection system of claim 1, wherein estimating the driver state includes estimating an emergency state, and performing the translation function includes contacting a third-party server.

Images