Vehicle interface control

Abstract

The invention provides vehicle interface control. An audio system for a vehicle includes a microphone concentrated on a first designated location of a first person relative to the vehicle; the camera is provided with a field of view covering the first specified position; the output device points to a second designated position of a second person relative to the vehicle; and a computer communicatively coupled to the microphone, the camera, and the output device. The computer is programmed to generate first text in a first language based on input audio data from the microphone and video data from the camera, translate the first text into second text in a second language, and instruct the output device to output the second text.

Description

Technical Field

[0001] The present disclosure relates to a vehicle interface control system. Background Art

[0002] In some cases, an occupant of a vehicle may wish to communicate with another person who may be located inside or outside the vehicle. Many factors may affect such communication and may impair and / or prevent a vehicle occupant from communicating with another person. Summary of the invention

[0003] The present disclosure describes a system for communication between an occupant of a vehicle and another person, and can support communication in which the other person speaks a different language than the occupant. The other person can be another occupant, or can be a person outside the vehicle, such as a pedestrian. The system includes: a microphone, the microphone is centered on a first specified position of a first person relative to the vehicle; a camera, the camera having a field of view covering the first specified position; an output device, the output device is directed to a second specified position of a second person relative to the vehicle; and a computer, the computer is communicatively coupled to the microphone, the camera, and the output device. The output device can be, for example, a speaker or a display. The use of the specified position can specify a location where a person may be present, for example, in a seat of the vehicle or outside the vehicle near a door of the vehicle. The computer is programmed to generate a first text in a first language based on input audio data from the microphone and video data from the camera, translate the first text into a second text in a second language, and instruct the output device to output the second text. The computer can use the video data to supervise the generation of the first text from the speech detected in the input audio data, thereby improving the accuracy of the speech-to-text conversion.

[0004] A system for a vehicle includes: a microphone focused on a first designated position of a first person relative to the vehicle; a camera having a field of view covering the first designated position; an output device directed to a second designated position of a second person relative to the vehicle; and a computer communicatively coupled to the microphone, the camera, and the output device. The computer is programmed to generate a first text in a first language based on input audio data from the microphone and video data from the camera, translate the first text into a second text in a second language, and instruct the output device to output the second text.

[0005] In one example, the first designated location may be outside the vehicle and adjacent to the vehicle. In another example, the microphone may be mounted to the exterior of the vehicle, and the field of view of the camera may encompass an area outside the vehicle.

[0006] In one example, the first designated location may be in a passenger compartment of the vehicle. In another example, the first designated location may be in a seat of the vehicle, the microphone may be positioned to receive audio from the first person sitting in the seat, and the field of view of the camera may encompass the seat.

[0007] In one example, the second designated location may be in a passenger compartment of the vehicle. In another example, the output device may be a speaker, the second designated location may be in a seat of the vehicle, and the speaker may be mounted to the seat.

[0008] In one example, the second designated location may be external to and adjacent to the vehicle.

[0009] A computer includes a processor and a memory, and the memory stores instructions executable by the processor to generate a first text in a first language based on input audio data from a microphone and video data from a camera, translate the first text into a second text in a second language, and instruct an output device to output the second text. The microphone is centered on a first designated position of a first person relative to a vehicle, and the camera has a field of view covering the first designated position. The output device is directed to a second designated position of a second person relative to the vehicle.

[0010] In one example, the instructions for generating the first text may include instructions for performing the following operations: generating an audio-based text based on the input audio data, generating a video-based text based on the video data, and combining the audio-based text with the video-based text into the first text. In another example, the instructions may also include instructions for performing the following operations: generating an audio-based confidence level for the audio-based text and a video-based confidence level for the video-based text, and the instructions for combining the audio-based text with the video-based text into the first text may include instructions for performing the following operations: combining the audio-based text with the video-based text into the first text based on the audio-based confidence level and the video-based confidence level. In yet another example, the audio-based text may include an audio-based text sequence, the video-based text may include a video-based text sequence, the audio-based confidence level may include a sequence of audio-based confidence values ​​for the corresponding audio-based text, and the video-based confidence level may include a sequence of video-based confidence values ​​for the corresponding video-based text. In yet another example, instructions for combining the audio-based text with the video-based text into the first text may include instructions for selecting text of the first text from the audio-based text or the video-based text based on the larger of the corresponding audio-based confidence value or the video-based confidence value.

[0011] In one example, the instructions for generating the first text may include instructions for performing the following operations: performing a speech-to-text algorithm on the input audio data.

[0012] In one example, the instructions for generating the first text may include instructions for performing the following operations: performing a lip reading algorithm on the video data. In another example, the lip reading algorithm may output video-based text, and the instructions for generating the first text may include instructions for performing the following operations: performing a speech-to-text algorithm on the input audio data to output audio-based text, and combining the audio-based text with the video-based text into the first text.

[0013] In one example, the output device may be a speaker, and the instructions may further include instructions for performing the following operations: generating output audio data from the second text and instructing the speaker to play the output audio data. In another example, the instructions may further include instructions for performing the following operations: generating elimination audio data based on the input audio data, and instructing the speaker to play the elimination audio data. In yet another example, the instructions may further include instructions for performing the following operations: instructing the speaker to play the output audio data and the elimination audio data simultaneously.

[0014] A method includes generating a first text in a first language based on input audio data from a microphone and video data from a camera, translating the first text into a second text in a second language, and instructing an output device to output the second text. The microphone is centered on a first designated position of a first person relative to a vehicle, and the camera has a field of view that encompasses the first designated position. The output device is directed toward a second designated position of a second person relative to the vehicle. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 is a perspective view of an example vehicle.

[0016] Figure 2 is a top view of the vehicle with the passenger compartment exposed for illustration purposes.

[0017] Figure 3 is a block diagram of an example system for a vehicle.

[0018] Figure 4 is a flow chart of an example process for controlling a system. DETAILED DESCRIPTION

[0019] Referring to the drawings, wherein like numerals refer to like parts throughout the several views, a system 105 for a vehicle 100 includes: a microphone 115, 215 centered on a first designated location 110 of a first person relative to the vehicle 100; a camera 120, 220 having a field of view encompassing the first designated location 110; an output device 125, 135, 210, 225 directed toward a second designated location 110 of a second person relative to the vehicle 100; and a computer 300 communicatively coupled to the microphone 115, 215, the camera 120, 220, and the output device 125, 135, 210, 225. The computer 300 is programmed to generate a first text in a first language based on input audio data from the microphone 115, 215 and video data from the camera 120, 220, translate the first text into a second text in a second language, and instruct the output device 125, 135, 210, 225 to output the second text.

[0020] refer to Figure 1 , the vehicle 100 may be any passenger or commercial automobile, such as a car, truck, sport utility vehicle, crossover, van, minivan, taxi, bus, etc.

[0021] The vehicle 100 includes a body 130. The vehicle 100 may be a unibody structure, in which the frame of the vehicle 100 and the body 130 are a single component. Alternatively, the vehicle 100 may be a body-on-frame structure, in which the frame supports the body 130, which is a separate component from the frame. The frame and body 130 may be formed of any suitable material (e.g., steel, aluminum, etc.).

[0022] The system 105 of the vehicle 100 may include at least one external camera 120. The external camera 120 is mounted on the vehicle 100, for example, mounted to the body 130 of the vehicle 100. The external camera 120 is aimed at the exterior of the vehicle 100, i.e., has a field of view covering an area outside the vehicle 100, for example, is oriented away from the vehicle 100, and is aimed, for example, transversely relative to the vehicle 100, i.e., has a field of view covering a line or plane orthogonal to the front-to-back length or axis of the vehicle. For example, the external camera 120 may be mounted to a roof rack, a center pillar, a door handle, a door inner sill trim, etc. The field of view of the external camera 120 covers one of the designated locations 110 of the vehicle 100, specifically a designated location 110 outside the vehicle 100. This orientation can detect pedestrians outside the vehicle 100 who intend to speak to the occupants of the vehicle 100, for example, pedestrians who have approached the door of the vehicle 100 or personnel at a toll booth or order window that the vehicle 100 has approached.

[0023] The external camera 120 can detect electromagnetic radiation within a range of wavelengths. For example, the external camera 120 can detect visible light, infrared radiation, ultraviolet light, or a range of wavelengths that include visible light, infrared light, and / or ultraviolet light. For example, the external camera 120 can be a charge coupled device (CCD), a complementary metal oxide semiconductor (CMOS), or any other suitable type.

[0024] The system 105 of the vehicle 100 may include at least one external microphone 115. The external microphone 115 may be mounted outside of the passenger compartment 200 of the vehicle 100, for example, attached to an outward-facing component of the vehicle 100. The external microphone 115 may be directed from the outside toward the vehicle 100, i.e., arranged to detect sounds originating from a source spaced apart from the vehicle 100. For example, the external microphone 115 may be mounted to the exterior of the vehicle 100, for example, may be part of an exterior door panel. The external microphone 115 may be mounted on a side of the vehicle 100, for example, the driver's side of the vehicle 100 as shown. The external microphone 115 is centered on one of the designated locations 110 of the vehicle 100, specifically a designated location 110 outside the vehicle 100, for example, a designated location 110 covered by the field of view of the external camera 120.

[0025] The external microphone 115 is a transducer that converts sound into an electrical signal. The external microphone 115 can be any suitable type for receiving sound from a person speaking outside the vehicle 100, for example, a dynamic microphone, a condenser microphone, a piezoelectric microphone, a transducer microphone on glass, a transducer microphone on trim, etc. If the external microphone 115 is a transducer microphone on trim, the external microphone 115 can be part of the door panel. An advantage of the external microphone 115 being a transducer microphone on trim is that it is more difficult for environmental factors to interfere with the performance of the transducer microphone on trim. The same is true for a transducer on glass. A single piece of debris (e.g., dirt, mud, ice, snow) may significantly block or attenuate the sampling of sound by microphones other than the transducer microphone on trim or the transducer microphone on glass.

[0026] The system 105 of the vehicle 100 may include at least one external speaker 125. The external speaker 125 may be mounted outside of the passenger compartment 200 of the vehicle 100, e.g., attached to an outward-facing component of the vehicle 100, mounted in the passenger compartment 200, or both. The external speaker 125 may be directed from the outside toward the vehicle 100, i.e., arranged to project sound away from the vehicle 100, even if mounted inside the passenger compartment 200. The external speaker 125 is directed toward one of the designated locations 110 of the vehicle 100. For example, the external speaker 125 may be mounted to a door panel directly outside of a pillar of the vehicle 100. The external speaker 125 may be mounted to the same side of the vehicle 100 as the external microphone 115 and / or the external camera 120, e.g., directed toward the same designated location 110 as the external microphone 115 and / or the external camera 120. For another example, when mounted inside the passenger compartment 200, the exterior speaker 125 may be used when the windows have been rolled down to augment or replace another exterior speaker 125 mounted outside the vehicle 100 to provide higher fidelity audio to support improved clarity.

[0027] The external speaker 125 can be any suitable type of speaker that is audible when a person is relatively close to the vehicle 100. Specifically, the external speaker 125 can be a panel exciter, that is, sound is generated by vibrating a rigid panel. For example, an electric motor can be adhered to the inside of a door panel and vibrations are applied to the door panel to generate sound. Alternatively or in addition, an electric motor (e.g., a piezoelectric transducer) can be adhered to the inside of a rigid glass panel and vibrations are applied to the glass panel to generate sound. The advantage of the external speaker 125 being a panel exciter or glass panel exciter rather than a point speaker is that it is more difficult for environmental factors to interfere with the performance of the external speaker 125. A single piece of debris (e.g., dirt, mud, ice, snow) can significantly block or attenuate sound from a point speaker but not from a panel exciter.

[0028] The system 105 may include at least one display screen 135. The display screen 135 is positioned to be displayed on the exterior of the vehicle 100, and therefore on the exterior of the passenger compartment 200 of the vehicle 100. For example, the display screen 135 may be mounted to the body 130 of the vehicle 100, for example, a center pillar of the body 130, or mounted to a door panel of a door of the vehicle 100. The display screen 135 on the door panel may be directly outside the center pillar. Mounting the display screen 135 next to the center pillar may position the display screen 135 next to the position of an occupant in the passenger compartment 200, regardless of whether the occupant is in front of or behind the vehicle 100. Thus, the display screen 135 is easily visible to a person approaching one of the doors of the vehicle 100. For another example, the display screen 135 may include a projector, such as a digital light projector (DLP), mounted in the passenger compartment 200, which projects a display image on a window of the vehicle 100. The window may include a film that reacts to a specific wavelength of light (e.g., light projected onto the window by the projector). For example, the film may include a phosphor coating or a quantum dot coating.

[0029] The display screen 135 may be of any suitable type for displaying content readable by a person standing outside the vehicle 100, such as a light emitting diode (LED), an organic light emitting diode (OLED), a liquid crystal display (LCD), plasma, digital light processing technology (DLPT), etc. The display screen 135 may be a touch screen and may accept input.

[0030] refer to Figure 2 The vehicle 100 includes a passenger cabin 200 to accommodate the occupants (if any) of the vehicle 100. The passenger cabin 200 may include a plurality of seats 205, for example, one or more of the seats 205 disposed in the front row of the passenger cabin 200 and one or more of the seats 205 disposed in the second row behind the front row. The passenger cabin 200 may also include seats 205 in a third row (not shown) at the rear of the passenger cabin 200.

[0031] The system 105 may include a user interface 210. The user interface 210 presents information to and receives information from an occupant of the vehicle 100. The user interface 210 may be located, for example, on a dashboard in the passenger compartment 200 of the vehicle 100, or anywhere else where an occupant may easily see it. The user interface 210 may include a dial, a digital readout, a screen, etc., for providing information to the occupant, such as, for example, known human-machine interface (HMI) elements. The user interface 210 may include buttons, knobs, a keyboard, etc., for receiving information from the occupant.

[0032] The area in and around the vehicle 100 contains a plurality of designated locations 110. Each designated location 110 is a preset location in space defined relative to the vehicle 100 (e.g., defined as a three-dimensional coordinate in the reference frame of the vehicle 100). The designated locations 110 are selected as possible locations of occupants of the vehicle 100 or pedestrians who are in conversation with occupants of the vehicle 100. Some of the designated locations 110 are in the passenger compartment 200 of the vehicle 100, for example, one designated location 110 in each seat 205 of the vehicle 100. For example, the vehicle 100 may include four designated locations 110, each located at the location of the head of the 50th percentile occupant sitting in one of the seats 205. One or more of the designated locations 110 may be located outside the vehicle 100 and adjacent to the vehicle, for example, adjacent to a door of the vehicle 100. For example, the vehicle 100 may include two designated locations 110 outside the vehicle 100, each at an average lateral distance and height relative to the vehicle 100 at a toll booth window or a drive-thru window. The designated locations 110 provide reference points for the placement of cameras 120, 220, microphones 115, 215, and speakers 125, 225. Each designated location 110 may have at least one camera 120, 220, at least one microphone 115, 215, and at least one speaker 125, 225 covering the designated location 110.

[0033] The system 105 includes at least one interior camera 220. The interior camera 220 has a field of view that encompasses at least one of the designated locations 110 (e.g., encompasses at least one of the seats 205). For example, the interior camera 220 can be mounted to the dashboard in front of the seat 205 and facing rearward. The interior camera 220 can have the same functionality as described above for the exterior camera 120.

[0034] The system 105 includes at least one interior microphone 215. Each interior microphone 215 is centered on a respective designated location 110 of the vehicle 100, for example, positioned to receive audio from an occupant sitting in a respective seat 205. For example, the interior microphone 215 may be mounted to a respective seat 205, for example, to a headrest of the seat 205. The interior microphone 215 is a transducer that converts sound into an electrical signal. The interior transducer 215 may be of any suitable type for receiving sound from an occupant of the vehicle 100, for example, a dynamic microphone, a condenser microphone, a piezoelectric microphone, etc.

[0035] The system 105 includes at least one interior speaker 225. Each interior speaker 225 is directed toward a respective designated location 110 of the vehicle 100, e.g., positioned to emit audio to an occupant seated in a respective seat 205. For example, the interior speaker 225 may be mounted to the respective seat 205, e.g., to a headrest of the seat 205. The interior speaker 225 may be any suitable type of speaker for outputting sound to an occupant of the passenger compartment 200, e.g., a dynamic loudspeaker.

[0036] refer to Figure 3 , the system 105 includes a computer 300. The computer 300 is a microprocessor-based computing device, for example, a general-purpose computing device (which includes a processor and a memory, an electronic controller, etc.), a field programmable gate array (FPGA), an application-specific integrated circuit (ASIC), a combination of the foregoing, etc. Typically, hardware description languages ​​such as VHDL (VHSIC (Very High Speed ​​Integrated Circuit) Hardware Description Language) are used in electronic design automation to describe digital and mixed-signal systems such as FPGAs and ASICs. For example, an ASIC is manufactured based on VHDL programming provided before manufacturing, and the logic components inside the FPGA can be configured based on, for example, VHDL programming stored in a memory electrically connected to the FPGA circuit. Therefore, the computer 300 may include a processor, a memory, etc. The memory of the computer 300 may include a medium for storing instructions executable by the processor and for electronically storing data and / or a database, and / or the computer 300 may include a structure such as the foregoing structure that provides programming. The computer 300 may be a plurality of computers coupled together.

[0037] The system 105 may include a communication network 305. The computer 300 may transmit and receive data via the communication network 305. The communication network 305 may be, for example, a controller area network (CAN) bus, Ethernet, Wi-Fi, a local interconnect network (LIN), and / or any other wired or wireless communication network. The computer 300 may be communicatively coupled to the microphones 115, 215, the cameras 120, 220, the speakers 125, 225, the display screen 135, the user interface 210, and other components via the communication network 305.

[0038] Return to Figure 1 to Figure 2 , the system 105 can facilitate a conversation between occupants and / or pedestrians in different designated locations 110 of the vehicle 100. The following description is for one side of the conversation, where one person is speaking in a first designated location 110 and one person is listening in a second designated location 110. For the other side of the conversation, the following description can be reversed at the same time, i.e., from the second designated location 110 to the first designated location 110. Alternatively, the system 105 can work unidirectionally, where one person broadcasts from a first designated location 110 to one or more occupants in one or more second designated locations 110, without reverse transmission.

[0039] The computer 300 may be programmed to receive, for example, via the user interface 210, a selection of one or more designated locations 110 including a first designated location 110 and at least one second designated location 110. The first designated location 110 refers to a designated location 110 of a person who is speaking, and the second designated location 110 refers to a designated location 110 of a person who is listening. For example, the selection may be for two designated locations 110, and the computer 300 may perform the following steps for each of the two designated locations 110 to the other designated location 110, so as to facilitate a conversation between the persons in the two designated locations 110. As another example, the selection may be for a first designated location 110 in which a person is to speak and one or more second designated locations 110 in which a person is to listen.

[0040] The computer 300 is programmed to receive input audio data from one of the microphones 115, 215 (i.e., the microphones 115, 215 centered on the first designated location 110). The microphones 115, 215 generate the input audio data. The input audio data is sound data recorded in any suitable format, such as a standard audio file format, such as .wav. The audio data includes at least sounds that vary over time. For example, the audio data can be represented as a spectrogram that shows amplitude as a function of time and frequency.

[0041] The computer 300 is programmed to receive video data from one of the cameras 120, 220 (i.e., the camera 120, 220 having a field of view covering the first specified position 110). The camera 120, 220 generates video data. The video data is a series of image frames depicting a scene contained in the field of view of the camera 120, 220. Each image frame is a two-dimensional matrix of pixels. The brightness and / or color of each pixel is represented as one or more numerical values, for example, a scalar unitless value of photometric light intensity between 0 (black) and 1 (white), or a value for each of red, green, and blue, for example, each with an 8-bit scale (0 to 255) or a 12-bit or 16-bit scale. The pixel can be a mixture of representations, such as a repeating pattern of scalar values ​​of the intensity of three pixels and a fourth pixel having three numerical color values, or some other pattern. A location in an image frame (i.e., a location in the field of view of the sensor when the image frame was recorded) can be specified in terms of pixel dimensions or coordinates, for example, a pair of ordered pixel distances, such as a number of pixels from the top edge of the field of view and a number of pixels from the left edge of the field of view.

[0042] The input audio data is associated with the video data, for example, recorded at the same time as the video data is recorded, in a scene depicted in the video data. For example, the input audio data may be synchronized in time with the video data.

[0043] The computer 300 can be programmed to generate audio-based text based on input audio data. For example, the computer 300 can perform a speech-to-text algorithm on the input audio data to output audio-based text. The computer 300 can use any suitable speech-to-text algorithm to convert speech to text, such as a hidden Markov model, speech recognition based on dynamic time warping, neural networks, end-to-end speech recognition, etc. The computer 300 can generate audio-based text without using video data. Audio-based text includes a sequence of words, which will be referred to as audio-based text.

[0044] The computer 300 can be programmed to generate video-based text based on video data. For example, the computer 300 can perform a lip reading algorithm on the video data to output video-based text. The computer 300 can use any suitable lip reading algorithm to generate video-based text, for example, a sequence of a convolutional neural network (CNN) trained to locate lips in an image, a CNN trained to extract features related to the shape of the lips, and a machine learning algorithm (such as a long short-term memory (LSTM) algorithm) for selecting text based on the extracted features. The LSTM algorithm can be used to select text based on the context of the surrounding text. The computer 300 can generate video-based text without using input audio data. The video-based text includes a sequence of text, which will be referred to as video-based text.

[0045] The computer 300 can be programmed to generate confidence levels for audio-based text and video-based text. For the purposes of this disclosure, a "confidence level" is defined as a measure of the likelihood that the generated text matches the corresponding actual spoken text. The confidence level for audio-based text will be referred to as an audio-based confidence level, and the confidence level for video-based text will be referred to as a video-based confidence level. The audio-based confidence level can include a sequence of audio-based confidence values ​​for the corresponding audio-based text, and the video-based confidence level can include a sequence of video-based confidence values ​​for the corresponding video-based text. The audio-based confidence value can be generated by a speech-to-text algorithm, for example, as a score accompanying the corresponding audio-based text. For example, the speech-to-text algorithm can generate audio-based text by generating scores for multiple candidate texts and selecting the candidate text with the highest score. The highest score of the selected audio-based text can be regarded as the audio-based confidence value of the audio-based text. The video-based confidence value can be generated in a similar manner by a lip-reading algorithm (e.g., by an LSTM algorithm within a lip-reading algorithm).

[0046] The computer 300 is programmed to generate a first text based on the input audio data from the microphones 115, 215 and the video data from the cameras 120, 220. The computer 300 can combine the audio-based text generated from the input audio data with the video-based text generated from the video data into the first text, for example, based on the audio-based confidence level and the video-based confidence level. For example, the computer 300 can select the text of the first text from the audio-based text or the video-based text according to the larger of the corresponding audio-based confidence value or the video-based confidence value. The audio-based text and the video-based text can be associated in a paired sequence according to the time synchronization of the input audio data and the video data, wherein each pair includes an audio-based text and a video-based text. The computer 300 selects a text from each pair. For a specific example, if the audio-based confidence values ​​of a sequence with five pairs of words are {0.80, 0.84, 0.71, 0.56, 0.75} and the video-based confidence values ​​are {0.65, 0.73, 0.67, 0.71, 0.69}, the computer 300 selects the first audio-based word in the first three pairs, the video-based word in the fourth pair, and the audio-based word in the fifth pair as the words in the first text. One or both of the audio-based confidence value and the video-based confidence value can be scaled so that the confidence values ​​are commensurate.

[0047] The first text is in a first language. The computer 300 may be programmed to receive or determine the first language. For example, an occupant at the first designated location 110 may provide input to the user interface 210 to select a first language. Unless the occupant has already made a selection, the first language may be a default language. For another example, the computer 300 may identify the occupant in the first designated location 110, for example based on a login provided by the occupant, facial recognition using video data, pairing of a mobile device belonging to the occupant with the user interface 210, and the like, and then the computer 300 may identify the language stored in the profile of the identified occupant. As another example, the computer 300 may use a language recognition algorithm to identify the first language based on the first text. Speech-to-text algorithms and lip-reading algorithms may be programmed or trained for the first language.

[0048] The computer 300 may be programmed to receive the second language of the second designated location 110. For example, the occupant at the second designated location 110 may provide the user interface 210 with input to select the second language of the second designated location 110. Unless the occupant has made a selection, the second language may be a default language. As another example, the computer 300 may select a language based on the GPS location and / or heading of the vehicle 100. The computer 300 may determine that the vehicle 100 is located inside a geo-fenced area or is approaching a geo-fenced area. For a specific example, the computer 300 may select French in response to the vehicle 100 traveling from the United States to an intersection near Monterey, Canada. For another example, the computer 300 may identify the occupant at the second designated location 110, for example, based on a login provided by the occupant, facial recognition using video data, pairing of a mobile device belonging to the occupant with the user interface 210, and the like, and then the computer 300 may identify the language stored in the profile of the identified occupant.

[0049] The computer 300 is programmed to translate a first text in a first language into a second text in a second language. The computer 300 can use any suitable machine translation algorithm to translate from the first language to the second language, for example, rule-based machine translation, such as conversion-based machine translation, interlingual machine translation, or dictionary-based machine translation; statistical machine translation, such as example-based machine translation; hybrid machine translation combining rule-based and statistical; neural machine translation, for example using deep learning, etc. The machine translation algorithm can receive the first text, the first language, and the second language as input, and provide the second text as output. Alternatively, the computer 300 can select from different machine translation algorithms based on the selection of the first language and the second language according to their performance in translating from the first language to the second language, and the selected machine translation algorithm receives the first text as input and provides the second text as output.

[0050] The computer 300 is programmed to instruct the output device 125, 135, 210, 225 to output the second text. The output device 125, 135, 210, 225 can be one of the speakers 125, 225 and / or a visual output device (such as a display screen 135 or a user interface 210). For example, the computer 300 can generate output audio data from the second text and instruct one of the speakers 125, 225 to play the output audio data, as will be described below. Alternatively or in addition, the computer 300 can instruct one of the visual output devices 135, 210 to display the second text, as will be described below.

[0051] The computer 300 is programmed to generate output audio data from the second text. For example, the computer 300 may execute a text-to-speech algorithm on the second text to output the output audio data. The computer 300 may use any suitable text-to-speech algorithm, for example, concatenative synthesis, such as unit selection synthesis, diphone synthesis, or domain-specific synthesis; formant synthesis; articulator parameter synthesis; hidden Markov model; deep learning-based synthesis; etc.

[0052] The computer 300 may be programmed to generate cancellation audio data based on the input audio data. The computer 300 may execute an active noise cancellation algorithm to generate cancellation audio data that cancels the input audio data. The active noise cancellation algorithm may invert or phase shift the sound waves contained in the input audio data to generate the cancellation audio data. Playing the cancellation audio data may cause destructive interference to the sound that generates the input audio data.

[0053] The computer 300 is programmed to instruct the speakers 125, 225 of the second designated location 110 to play the output audio data. The computer 300 can instruct the speakers 125, 225 to play the output audio data at a volume that is generally audible to the occupants at the second designated location 110 and inaudible or at least inaudible to the occupants in other designated locations 110 (e.g., the first designated location 110), which is facilitated by the speakers 125, 225 being directed toward the second designated location 110. Thus, the system 105 provides a zone output, i.e., different audio output for different areas of the vehicle 100.

[0054] The computer 300 can be programmed to instruct the speakers 125, 225 to play the cancellation audio data, for example, to play the output audio data and the cancellation audio data simultaneously. The computer 300 can instruct the speakers 125, 225 to play the cancellation audio data at a volume proportional to the volume of the sound that produced the input audio data to promote destructive interference. Noise cancellation helps the system 105 provide a zone output by reducing the sounds from other designated locations 110 that are perceptible at the second designated location 110, thereby helping the person at the second designated location 110 to more easily understand the output audio data.

[0055] The computer 300 may be programmed to instruct the visual output device 135, 210 to display the second text, i.e., to output a visual representation of the second text. For example, the computer 300 may instruct the visual output device 135, 210 to display the second text as text on the visual output device 135, 210, thereby providing a text translation of the speech of the person at the first designated location 110. As another example, the computer 300 may instruct the visual output device 135, 210 to display a sign language representation of the second text, for example, as a holographic display of a hand gesturing to the text of the second text.

[0056] Figure 4 4 is a flow chart illustrating an example process 400 for controlling the system 105. The memory of the computer 300 stores executable instructions for performing the steps of the process 400, and / or can be programmed in a structure such as described above. As a general overview of the process 400, the computer 300 receives input audio data from a first designated location 110, receives video data of the first designated location 110, generates audio-based text from the input audio data, generates video-based text from the video data, generates a first text from the audio-based text and the video-based text, receives a first language and a second language, translates the first text in the first language into a second text in the second language, generates output audio data from the second text, generates elimination audio data from the input audio data, and instructs the output device 125, 135, 210, 225 to output the second text to the second designated location 110, for example, play the output audio data and the elimination audio data or display the second text.

[0057] The process 400 begins in block 405, where the computer 300 receives input audio data for a first designated location 110 from a microphone 115, 215, as described above.

[0058] Next, in block 410 , the computer 300 receives video data of the first designated location 110 from the camera 120 , 220 , as described above.

[0059] Next, in block 415, the computer 300 generates audio-based text based on the input audio data, as described above.

[0060] Next, in block 420 , computer 300 generates video-based text based on the video data, as described above.

[0061] Next, in block 425 , the computer 300 generates a first text based on the input audio data and the video data (eg, by combining the audio-based text with the video-based text into a first text), as described above.

[0062] Next, in block 430 , the computer 300 receives and / or determines the first language and the second language, as described above.

[0063] Next, in block 435, computer 300 translates the first text in the first language into a second text in the second language, as described above.

[0064] Next, in block 440, the computer 300 generates output audio data from the second text, as described above.

[0065] Next, in block 445, the computer 300 generates cancellation audio data based on the input audio data, as described above.

[0066] Next, in block 450, the computer 300 instructs the output device 125, 135, 210, 225 of the second designated location 110 to output the second text, as described above. For example, the computer 300 may instruct the speaker 125, 225 to simultaneously play the output audio data and eliminate the audio data, as described above. For another example, the computer 300 may instruct the display 135 or the user interface 210 to display the second text, as described above. After block 450, the process 400 ends.

[0067] Generally speaking, the computing systems and / or devices described may employ any of a variety of computer operating systems, including but not limited to the following versions and / or varieties: Ford Application; AppLink / Smart Device Link middleware; Microsoft Operating system; Microsoft Operating system; Unix operating system (for example, distributed by Oracle Corporation of Redwood Shores, California operating systems); AIX UNIX operating system distributed by International Business Machines Corporation of Armonk, New York; Linux operating system; Mac OSX and iOS operating systems distributed by Apple Inc. of Cupertino, California; BlackBerry operating system distributed by BlackBerry Ltd. of Waterloo, Canada; and Android operating system developed by Google Inc. and the Open Handset Alliance; or provided by QNX Software Systems CAR Infotainment Platform. Examples of computing devices include, but are not limited to, an in-vehicle computer, a computer workstation, a server, a desktop, notebook, laptop or handheld computer, or some other computing system and / or device.

[0068] Computing devices typically include computer-executable instructions, wherein the instructions are executable by one or more computing devices such as those listed above. Computer-executable instructions may be compiled or interpreted from a computer program created using a variety of programming languages ​​and / or technologies, including, alone or in combination, but not limited to Java. TM , C, C++, Matlab, Simulink, Stateflow, Visual Basic, Java Script, Python, Perl, HTML, etc. Some of these applications can be compiled and executed on virtual machines such as Java virtual machines, Dalvik virtual machines, etc. Typically, a processor (e.g., a microprocessor) receives instructions from, for example, a memory, a computer-readable medium, etc., and executes these instructions to perform one or more processes, including one or more of the processes described herein. Such instructions and other data can be stored and transmitted using various computer-readable media. Files in a computing device are typically a collection of data stored on a computer-readable medium such as a storage medium, a random access memory, etc.

[0069] Computer-readable media (also referred to as processor-readable media) include any non-transitory (e.g., tangible) media that participate in providing data (e.g., instructions) that can be read by a computer (e.g., by a processor of a computer). Such media can take many forms, including, but not limited to, non-volatile media and volatile media. Instructions can be transmitted via one or more transmission media, including optical fibers, wires, wireless communications, including internals that make up a system bus coupled to a processor of a computer. Common forms of computer-readable media include, for example, RAM, PROM, EPROM, FLASH-EEPROM, any other memory chip or cassette, or any other medium from which a computer can read.

[0070] The database, data repository or other data storage described herein may include various mechanisms for storing, accessing / accessing and retrieving various data, including hierarchical databases, file sets in file systems, application databases in special formats, relational database management systems (RDBMS), non-relational databases (NoSQL), graphic databases (GDB), etc. Each such data storage is typically included in a computing device using a computer operating system such as one of the above-mentioned, and is accessed via a network in any one or more of a variety of ways. The file system is accessible from the computer operating system and may include files stored in various formats. In addition to the language (such as the above-mentioned PL / SQL language) for creating, storing, editing and executing stored programs, RDBMS typically also uses structured query language (SQL).

[0071] In some examples, system elements may be implemented as computer-readable instructions (e.g., software) on one or more computing devices (e.g., servers, personal computers, etc.) stored on computer-readable media associated therewith (e.g., disks, memories, etc.). A computer program product may include such instructions stored on a computer-readable medium for performing the functions described herein.

[0072] In the accompanying drawings, like reference numerals indicate like elements. In addition, some or all of these elements may be changed. With respect to the media, processes, systems, methods, heuristics, etc. described herein, it should be understood that although the steps of such processes, etc. have been described as occurring in a certain ordered sequence, such processes may be practiced by performing the steps in an order different from that described herein. It should also be understood that certain steps may be performed simultaneously, other steps may be added, or certain steps described herein may be omitted. The operations, systems, and methods described herein should always be implemented and / or performed in accordance with applicable owner / user manuals and / or safety guidelines.

[0073] The present disclosure has been described in an illustrative manner, and it should be understood that the terms that have been used are intended to be descriptive in nature, rather than restrictive. The adjectives "first" and "second" are used throughout this document as identifiers, and are not intended to indicate importance, order, or quantity. The use of "in response to" and "after determining..." indicates a causal relationship, not just a temporal relationship. In view of the above teachings, many modifications and variations of the present disclosure are possible, and the present disclosure can be practiced in other ways than those specifically described.

[0074] According to the present invention, a system for a vehicle is provided, comprising: a microphone, the microphone being focused on a first designated position of a first person relative to the vehicle; a camera, the camera having a field of view covering the first designated position; an output device, the output device being directed to a second designated position of a second person relative to the vehicle; and a computer, the computer being communicatively coupled to the microphone, the camera and the output device, the computer being programmed to: generate a first text in a first language based on input audio data from the microphone and video data from the camera; translate the first text into a second text in a second language; and instruct the output device to output the second text.

[0075] According to one embodiment, the first designated location is outside the vehicle and adjacent to the vehicle.

[0076] According to one embodiment, the microphone is mounted to the exterior of the vehicle and the field of view of the camera covers an area outside the vehicle.

[0077] According to one embodiment, the first designated location is in a passenger compartment of the vehicle.

[0078] According to one embodiment, the first designated location is in a seat of the vehicle, the microphone is positioned to receive audio from the first person sitting in the seat, and the field of view of the camera encompasses the seat.

[0079] According to one embodiment, the second designated location is in a passenger compartment of the vehicle.

[0080] According to one embodiment, the output device is a speaker, the second designated position is in a seat of the vehicle, and the speaker is mounted to the seat.

[0081] According to one embodiment, the second designated location is outside the vehicle and adjacent to the vehicle.

[0082] According to the present invention, a computer is provided, the computer having a processor and a memory, the memory storing instructions, the instructions being executable by the processor to: generate a first text in a first language based on input audio data from a microphone and video data from a camera, the microphone being focused on a first designated position of a first person relative to a vehicle, the camera having a field of view covering the first designated position; translate the first text into a second text in a second language; and instruct an output device to output the second text, the output device being pointed to a second designated position of a second person relative to the vehicle.

[0083] According to one embodiment, the instructions for generating the first text include instructions for performing the following operations: generating an audio-based text based on the input audio data, generating a video-based text based on the video data, and combining the audio-based text and the video-based text into the first text.

[0084] According to one embodiment, the instructions also include instructions for performing the following operations: generating an audio-based confidence level for the audio-based text and a video-based confidence level for the video-based text, and the instructions for combining the audio-based text with the video-based text into the first text include instructions for performing the following operations: combining the audio-based text with the video-based text into the first text based on the audio-based confidence level and the video-based confidence level.

[0085] According to one embodiment, the audio-based text includes an audio-based text sequence, the video-based text includes a video-based text sequence, the audio-based confidence level includes a sequence of audio-based confidence values ​​of the corresponding audio-based text, and the video-based confidence level includes a sequence of video-based confidence values ​​of the corresponding video-based text.

[0086] According to one embodiment, the instructions for combining the audio-based text with the video-based text into the first text include instructions for performing the following operations: selecting text of the first text from the audio-based text or the video-based text based on the larger of the corresponding audio-based confidence value or the video-based confidence value.

[0087] According to one embodiment, the instructions for generating the first text include instructions for performing the following operations: executing a speech-to-text algorithm on the input audio data.

[0088] According to one embodiment, the instructions for generating the first text include instructions for performing the following operations: performing a lip reading algorithm on the video data.

[0089] According to one embodiment, the lip reading algorithm outputs video-based text, and the instructions for generating the first text include instructions for performing the following operations: executing a speech-to-text algorithm on the input audio data to output audio-based text, and combining the audio-based text with the video-based text into the first text.

[0090] According to one embodiment, the output device is a speaker, and the instructions may further include instructions for performing the following operations: generating output audio data from the second text and instructing the speaker to play the output audio data.

[0091] According to one embodiment, the instructions further include instructions for generating cancellation audio data based on the input audio data, and instructing the speaker to play the cancellation audio data.

[0092] According to one embodiment, the instructions further include instructions for performing the following operation: instructing the speaker to play the output audio data and the elimination audio data simultaneously.

[0093] According to the present invention, a method includes: generating a first text in a first language based on input audio data from a microphone and video data from a camera, wherein the microphone is centered on a first specified position of a first person relative to a vehicle, and the camera has a field of view covering the first specified position; translating the first text into a second text in a second language; and instructing an output device to output the second text, wherein the output device is pointed to a second specified position of a second person relative to the vehicle.

Claims

1. A method comprising: generating a first text in a first language based on input audio data from a microphone centered on a first designated position of a first person relative to the vehicle and video data from a camera having a field of view encompassing the first designated position; translating the first text into a second text in a second language; as well as The output device is instructed to output the second text, the output device pointing to a second designated position of a second person relative to the vehicle.

2. The method of claim 1, wherein generating the first text comprises generating an audio-based text based on the input audio data, generating a video-based text based on the video data, and combining the audio-based text with the video-based text into the first text.

3. The method according to claim 2 also includes generating an audio-based confidence level for the audio-based text and a video-based confidence level for the video-based text, wherein combining the audio-based text with the video-based text into the first text includes combining the audio-based text with the video-based text into the first text based on the audio-based confidence level and the video-based confidence level.

4. The method of claim 3, wherein the audio-based text comprises an audio-based text sequence, the video-based text comprises a video-based text sequence, the audio-based confidence level comprises a sequence of audio-based confidence values ​​for the corresponding audio-based text, and the video-based confidence level comprises a sequence of video-based confidence values ​​for the corresponding video-based text.

5. The method of claim 4, wherein combining the audio-based text with the video-based text into the first text comprises selecting text of the first text from the audio-based text or the video-based text based on the larger of the corresponding audio-based confidence value or the video-based confidence value. The method of claim 1 , wherein generating the first text comprises performing a speech-to-text algorithm on the input audio data.

7. The method of claim 1, wherein generating the first text comprises performing a lip reading algorithm on the video data.

8. The method of claim 7, wherein the lip-reading algorithm outputs video-based text, and generating the first text comprises executing a speech-to-text algorithm on the input audio data to output audio-based text, and combining the audio-based text with the video-based text into the first text.

9. The method of claim 1, wherein the output device is a speaker, the method further comprising generating output audio data from the second text and instructing the speaker to play the output audio data.

10. The method of claim 9, further comprising generating cancellation audio data based on the input audio data, and instructing the speaker to play the cancellation audio data. The method of claim 10 , further comprising instructing the speaker to play the output audio data and the cancellation audio data simultaneously.

12. A computer comprising a processor and a memory, the memory storing instructions executable by the processor to perform the method according to claims 1 to 11.

13. An audio system comprising: The computer according to claim 12; the microphone; said camera; as well as the output device; Wherein the computer is communicatively coupled to the microphone, the camera, and the output device.

14. The audio system of claim 13, wherein one of the first designated location or the second designated location is in a passenger compartment of the vehicle.

15. The audio system of claim 13, wherein one of the first designated location or the second designated location is exterior to and adjacent to the vehicle.

Images