Vehicle, device and method for user authentication in vehicle, and computer readable medium

Abstract

The invention relates to a vehicle, a device and a method for user authentication in the vehicle, and a computer readable medium. The apparatus includes a memory that stores a reference embedding set that includes a feature embedding for an utterance of a registered user and a feature embedding for a result of synthesis between the utterance of the registered user and a plurality of ambient noises. The apparatus also includes a user interface to receive input audio including utterances of a vehicle occupant and noise; and a processor that converts the input audio into an input embedding and determines whether the occupant is a registered user based on a comparison between the input embedding and the reference embedding set.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit under 35 U.S.C. 119(a) of Korean Patent Application No. 10-2023-0112901 filed in the Korean Intellectual Property Office on August 28, 2023, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present disclosure relates to a method and apparatus for user authentication in a vehicle based on voice recognition, and more particularly, to a method and apparatus for authenticating a user taking into account background noise in a vehicle. Background Art

[0004] Intelligent Transportation Systems (ITS) include vehicles capable of wireless communications (e.g., WiFi, 3G, LTE, 5G, NR systems, etc.) configured to communicate from vehicle to vehicle and between vehicles and external infrastructure based on various service types. Vehicles may include one or more electronic control units (ECUs) for executing customer service requirements and various functions.

[0005] With the development of voice recognition technology, the use of voice recognition assistant services that recognize voice commands generated by the user's voice and execute corresponding commands has increased significantly. The application of voice recognition assistant services has expanded from homes to vehicles and other fields. In other words, the voice recognition assistant service and the telematics service are linked, and the voice commands generated by the user's voice are transmitted to the vehicle to control the vehicle. As a result, the user can lock / unlock the door of the vehicle or control the internal temperature of the vehicle by turning on the air conditioner in advance.

[0006] In order to use such a voice recognition assistant service, user authentication is required. This is because without user authentication, there is a possibility that an unauthorized person can use the vehicle without permission.

[0007] As a related art user authentication method, there is a method of inducing an occupant to utter a predetermined sentence and comparing the occupant's voice utterance with pre-registered voice utterances of the occupant to allow the occupant access.

[0008] However, if noise is generated due to the vehicle environment, such as noise from an air conditioner operating in the vehicle or noise from rain, such as rain, the performance of user authentication may be deteriorated.

[0009] Therefore, a method of performing more accurate user authentication within a vehicle is needed. Summary of the invention

[0010] According to at least one embodiment, the present disclosure provides a device for user authentication of an occupant in a vehicle. The device includes a memory configured to store a reference embedding set, the reference embedding set including feature embeddings for speech of a registered user and feature embeddings for synthesis results between speech of a registered user and multiple environmental noises. The device also includes: a user interface (e.g., a microphone) configured to receive input audio including speech and noise of an occupant of the vehicle; and a processor configured to convert the input audio into an input embedding, and determine whether the occupant is a registered user based on a comparison between the input embedding and the reference embedding set.

[0011] A vehicle may include means for user authentication of a vehicle occupant.

[0012] According to another embodiment of the present disclosure, a computer-implemented method for user authentication of a vehicle is provided. The computer-implemented method includes: receiving input audio including speech and noise of an occupant of the vehicle through a user interface (e.g., a microphone), and converting the input audio into an input embedding through a processor. The computer-implemented method further includes: determining by the processor whether the occupant is a registered user based on a comparison between the input embedding and a pre-stored reference embedding set, the reference embedding set including feature embeddings for speech of registered users and feature embeddings for synthesis results between speech of registered users and multiple environmental noises.

[0013] According to a further embodiment, a non-transitory computer-readable medium includes program instructions executed by a processor, including: program instructions for receiving input audio including speech and noise of a vehicle occupant; program instructions for converting the input audio into an input embedding, and program instructions for determining whether the occupant is a registered user based on a comparison between the input embedding and a pre-stored reference embedding set, the reference embedding set including feature embeddings for speech of registered users and feature embeddings for synthesis results between speech of registered users and multiple environmental noises. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 is a configuration diagram of a speech recognition system according to an embodiment.

[0015] Figure 2 is a configuration diagram of a user authentication device according to an embodiment of the present disclosure.

[0016] Figure 3 is a diagram illustrating user registration according to an embodiment of the present disclosure.

[0017] Figure 4 is a diagram illustrating user authentication according to an embodiment of the present disclosure.

[0018] Figure 5 is a flow chart of a user authentication method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] It should be understood that the term "vehicle" or "vehicle-based" or other similar terms as used herein include a broad range of motor vehicles, such as passenger cars, including sport utility vehicles (SUVs), buses, trucks, various commercial vehicles, watercraft (including various ships and boats), aircraft, etc., and include hybrid vehicles, electric vehicles, plug-in hybrid electric vehicles, hydrogen-powered vehicles, and other alternative fuel vehicles (e.g., fuels derived from resources other than petroleum). As referred to herein, a hybrid vehicle is a vehicle having two or more power sources, for example, gasoline-powered and electric vehicles.

[0020] The terms used herein are only used for the purpose of describing specific embodiments, and are not intended to limit the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular forms "one", "an" and "the" are intended to also include plural forms. It should also be understood that when the terms "including" and / or "comprising" are used in this specification, it specifies the presence of the features, wholes, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, parts and / or their combinations. As used in this article, the term "and / or" includes any and all combinations of one or more related listed items. Throughout the specification, unless explicitly described to the contrary, the word "including" and variants such as "including" or "containing" will be understood to imply the inclusion of the elements, but do not exclude any other elements. In addition, the terms "unit", "body", "device" and "module" described in the specification are meant to be used to process at least one function and operation unit, and can be implemented by hardware components or software components and combinations thereof.

[0021] In addition, the control logic of the present disclosure may be embodied as a non-transitory computer-readable medium on a computer-readable medium containing executable program instructions executed by a processor, controller, etc. Examples of computer-readable media include, but are not limited to, ROM, RAM, compact disk (CD)-ROM, magnetic tape, floppy disk, flash drive, smart card, and optical data storage device. The computer-readable medium may also be distributed in a network-coupled computer system so that the computer-readable medium is stored and executed in a distributed manner, such as by a telematics server or a controller area network (CAN).

[0022] In view of the above circumstances, the present disclosure provides a method and apparatus for improving the performance of user authentication based on voice recognition even in various noise environments of a vehicle.

[0023] Problems to be solved by the present disclosure are not limited to the above-mentioned problems, and other problems not mentioned can be clearly understood by those skilled in the art from the following description.

[0024] The embodiments of the present disclosure are described in detail below using various drawings. It should be noted that when the components are marked with reference numerals in the various drawings, the same reference numerals are used as much as possible even for components shown in different drawings. In addition, in the description of the present disclosure, when it has been determined that the detailed description of the relevant known configuration or function may obscure the main points of the present disclosure, its detailed description is omitted.

[0025] When describing components according to embodiments of the present disclosure, symbols such as first, second, i), ii), a), and b) may be used. These symbols are used only to distinguish components from other components. The marking or sequence or order of components is not limited by the symbols. In this specification, when a component "includes" or "is equipped with" an element, it means that the component may further include other elements, without excluding other elements unless explicitly stated to the contrary. In addition, when an element in the written description and claims is described as "for" performing or implementing a stated function, step, instruction set, etc., the element may also be regarded as "configured to" do so.

[0026] Each component of the device or method according to the present disclosure can be implemented in hardware or software, or in a combination of hardware and software. In addition, the function of each component can be implemented in software. A microprocessor or processor can execute the function of the software corresponding to each component.

[0027] Figure 1 is a configuration diagram of a speech recognition system according to an embodiment.

[0028] refer to Figure 1 , the speech recognition system 100 recognizes and understands the user's speech and provides a service corresponding to the user's speech.

[0029] To this end, the speech recognition system 100 includes a speech recognition module 110 that converts a user's voice utterance into text, a natural language understanding (NLU) module 120 that determines an intention included in the user's voice utterance, and a result processing module 130 that performs processing to provide a result corresponding to the user's intention.

[0030] The voice recognition module 110 may be implemented as an STT (Speech to Text) engine, and may convert a user's utterance, which is a voice signal, into text by applying a voice recognition algorithm.

[0031] For example, the speech recognition module 110 may extract a feature vector from the user's speech by applying a feature vector extraction technique such as cepstrum, linear prediction coefficient (LPC), Mel-frequency cepstral coefficient (MFCC), or filter bank energy.

[0032] The speech recognition module 110 can obtain a recognition result by comparing the extracted feature vector with a trained reference pattern. To this end, an acoustic model that models and compares the signal features of the speech or a language model that models the language order relationship of words or syllables corresponding to the recognition vocabulary can be used.

[0033] The speech recognition module 110 may also convert the user's speech into text based on a model using machine learning or deep learning.

[0034] The NLU module 120 uses natural language understanding technology to determine the user's intention included in the input sentence. Here, the input sentence refers to the text converted by the speech recognition module 110.

[0035] The NLU module 120 may extract information such as a domain, an entity name, and a voice action from an input sentence, and recognize an intent and an entity according to the intent based on the extraction result.

[0036] For example, if the input sentence is "Let's go home", the domain is [navigation], the intent is [route setting], and the entity required to perform the control corresponding to the intent is [origin, destination]. For another example, if the input sentence is "turn on the air conditioner", the domain is [vehicle control], the intent is [air conditioner power on], and the entity required to perform the control corresponding to the intent is [temperature, air volume].

[0037] The result processing module 130 may output a result processing signal to the vehicle, the user device, or an external server so as to perform processing to provide a service corresponding to the user's intention.

[0038] For example, when the service corresponding to the user's intention is vehicle-related control, the result processing module 130 can transmit a result processing signal for executing the vehicle-related control to the vehicle. As another embodiment, when the service corresponding to the user's intention is to provide specific information, the result processing module 130 can search for the specific information and provide the searched information to the user terminal. If necessary, the information search can also be performed on another external server. As another embodiment, when the service corresponding to the user's intention is to provide specific content, the result processing module 130 can request to send the target content from an external server that provides the content. In another embodiment, when the service corresponding to the user's intention is to continue a simple conversation, the result processing module 130 can generate a response to the user's speech and output the response visually or auditorily.

[0039] The speech recognition system 100 may be provided on an external server or a user terminal, and some of its components may be provided in the external server and other components may be provided in the user terminal. The user terminal may be a mobile device (such as a smart phone, a tablet PC, or a wearable device), a home appliance with a user interface, or a vehicle.

[0040] The speech recognition system 100 may further include a dialog manager that manages the overall dialog between the speech recognition system 100 and the user.

[0041] The components of the speech recognition system 100 are classified based on their operations or functions, and all or some of the components of the speech recognition system 100 may share a memory or a processor.

[0042] The speech recognition system 100 may be implemented in a vehicle or a server. Otherwise, some components of the speech recognition system 100 may be included in the vehicle, and other components may be included in the server. For example, the vehicle sends a voice signal of an occupant to the server, and the server processes the voice signal of the occupant, generates information or control commands required by the occupant, and sends the information or control commands to the vehicle.

[0043] Meanwhile, the voice recognition system 100 may provide a response to the occupant's request through a conversation with the occupant, but for safety purposes, it is necessary to confirm whether the occupant is a user authorized to use the voice recognition system 100 .

[0044] According to one embodiment of the present disclosure, the method for user authentication may be performed in the voice recognition module 110 or in front of the voice recognition module 110 .

[0045] Figure 2 is a configuration diagram of a user authentication device according to an embodiment of the present disclosure.

[0046] refer to Figure 2 , the user authentication device 200 registers the feature embedding of the low-noise speech of the user and the feature embedding of the result of synthesizing various noises with the speech of the user in the user registration operation, and in the user authentication process, confirms whether the occupant is a registered user by comparing the input embedding of the speech of the vehicle occupant with the registered feature embedding. Through this, the user authentication device 200 can accurately determine whether the occupant is a registered user by using the reference embedding for each noise, even if the occupant's sound speech includes noise in the user authentication operation.

[0047] To this end, the user authentication device 200 includes a memory 230 and a processor 240. The user authentication device 200 may further include at least one of a user interface 210 and a communication interface 220.

[0048] The user authentication device 200 may be implemented in a vehicle or an external server. When the user authentication device 200 is set in a vehicle, the user authentication device 200 may include a user interface 210 (such as a microphone), a memory 230 and a processor 240, and may further include a communication interface 220. When the user authentication device 200 is set in a server, the user authentication device 200 may include a communication interface 220, a memory 230 and a processor 240, and may further include a user interface 210. In the following, it is assumed that the user authentication device 200 is implemented in a server.

[0049] The user interface 210 may include an input device, such as a microphone that converts a user's voice speech into an electrical signal, a camera that images a user in the vehicle, or a touch panel that receives a user's touch input. In addition, the user interface 210 may include an output device, such as a display device for responding to the user's voice speech or providing vehicle status information, and a speaker for outputting sounds required for vehicle-related control or providing services desired by the user. The user can trigger the processor 240 to execute a program through the user interface 210.

[0050] The communication interface 220 provides access to external devices. For example, the user authentication device 200 can communicate with other devices through the communication interface 220.

[0051] For example, the communication interface 220 may receive data for voice recognition from the vehicle, and may communicate with a device inside the vehicle or a terminal device of a user to execute a voice command.

[0052] The communication interface 220 is a hardware device implemented with various electronic circuits to send and receive signals through wireless or wired connections. In the present disclosure, the communication interface 220 can use the vehicle network communication technology to perform communication, and can use wireless Internet access or short-range communication technology to perform V2I communication with a server, infrastructure, or other vehicles outside the vehicle. Here, using the vehicle network communication technology, the communication interface 220 can perform in-vehicle communication through controller area network (CAN) communication, local area Internet (LIN) communication, flexible ray communication, etc. In addition, wireless communication technology may include wireless LAN (WLAN), wireless broadband (Wibro), Wi-Fi, and global microwave access interoperability (Wimax). In addition, short-range communication technology may include Bluetooth, ZigBee, ultra-wideband (UWB), radio frequency identification (RFID), and infrared data association (IrDA).

[0053] The memory 230 may store a program that enables the processor 240 to perform the user authentication method according to an embodiment of the present disclosure. For example, the program may include a plurality of instructions that can be executed by the processor 240, and the user authentication method may be performed by executing the plurality of instructions by the processor 240.

[0054] The memory 230 may maintain the stored data even when the power supplied to the user authentication device 200 is cut off.

[0055] The memory 230 may be a single memory or a plurality of memories. In this case, the information required for data generation may be stored in a single memory or may be divided to be stored in a plurality of memories. When the memory 230 includes a plurality of memories, the plurality of memories may be physically separated.

[0056] The memory 230 may include at least one of a volatile memory and a nonvolatile memory. The volatile memory includes a static random access memory (SRAM) or a dynamic random access memory (DRAM), and the nonvolatile memory includes a flash memory.

[0057] The processor 240 may be electrically connected to the user interface 210 , the communication interface 220 , and the memory 230 , and may electrically control each component.

[0058] The processor 240 may execute instructions stored in the memory 230. The processor 240 may include at least one core capable of executing at least one instruction. The processor 240 may be a single processor or a plurality of processors.

[0059] According to an embodiment of the present disclosure, the processor 240 may perform a user registration process and a user authentication process to determine whether an occupant in the vehicle is a registered user. The user registration process refers to a process of storing voice utterances of a registered user having authority of a voice recognition system of the vehicle, and the user authentication process refers to a process of determining whether an occupant is a registered user based on the voice utterances of the occupant.

[0060] During the user registration process, the processor 240 requests the user to speak a predetermined sentence, converts the user's voice utterance into an embedding vector, and stores the embedding vector in the memory 250 .

[0061] Specifically, the processor 240 stores not only the feature embedding for the user's voice, but also the feature embedding for the result of synthesizing different noises with the user's voice in the memory 250. That is, the feature embedding for the user's clean speech and the speech containing noise is stored. The user's registration is completed.

[0062] Processor 240 may store feature embeddings for each of the plurality of users.

[0063] Thereafter, during the user authentication process, the processor 240 may improve the accuracy of user authentication of the occupant, even in a noisy environment, by using feature embeddings for the user's clean utterances and utterances containing noise.

[0064] Specifically, the processor 240 receives input audio including the occupant's speech. The processor 240 may receive the input audio from the vehicle. Thereafter, the processor 240 converts the input audio into an input embedding and compares the input embedding with a pre-stored feature embedding to determine whether the occupant is a registered user.

[0065] Here, the input audio may include noise depending on the vehicle environment.

[0066] If the occupant is a registered user, the processor 240 notifies the vehicle that the occupant is a registered user.

[0067] In addition, the processor 240 grants the occupant the authority to control the voice recognition system. The occupant can use the voice recognition system to receive the required services. In addition, the occupant can also control the vehicle through voice commands.

[0068] Meanwhile, the processor 240 may perform at least some functions of the speech recognition system 100. That is, the processor 240 may implement at least one function of the speech recognition module 110, the NLU module 120, and the result processing module 130.

[0069] Meanwhile, in another embodiment, the user authentication device 200 may be implemented in a vehicle. The user authentication device 200 may receive input audio through a microphone of the vehicle, perform authentication on an occupant, and control the vehicle based on a voice command of the authenticated occupant.

[0070] Figure 3 is a diagram illustrating user registration according to an embodiment of the present disclosure.

[0071] refer to Figure 3 , shows a process in which a user authentication device generates a reference embedding set for a user using an embedding model 300 and performs user registration by using the reference embedding set. The user registration process may be started by a vehicle operation of the user, or may be started by a terminal device of the user.

[0072] First, the user authentication device receives a user's speech as input. When the user authentication device is set in a vehicle, the user authentication device requests the user to speak a predetermined sentence and receives the user's speech of the predetermined sentence through a microphone. When the user authentication device is set in a server, the user's speech is received from the vehicle.

[0073] Here, the user utterance may be in the form of audio representing a one-dimensional vector with the amplitude of the signal as an element value in the time domain.

[0074] At the same time, the user authentication device may request the user to speak a sentence in an environment with minimal surrounding noise. Specifically, the user authentication device measures at least one of the noise level before the user speaks and the noise level after the user speaks. For example, the user authentication device may measure the root mean square (RMS) of the noise in a segment before or after the user's speech segment based on the sound starting point (speech start (BoS)) and the sound end point (speech end (EoS)). If at least one of the measured noise levels is higher than a preset noise threshold, the user authentication device may request the user to register in an environment where the noise level is lower than the noise threshold.

[0075] Thereafter, the user authentication device synthesizes the user speech with a plurality of environmental noises.

[0076] To this end, the user authentication device pre-stores a plurality of environmental noises having different acoustic characteristics. The plurality of environmental noises may include noises according to various environments of the vehicle, such as noise due to rain, noise due to operation of an air conditioner, or noise due to wind. These environmental noises may be collected by a microphone in the vehicle.

[0077] For example, the user authentication device may synthesize the user speech with the first environmental noise, and may synthesize the user speech with the second environmental noise. Here, the first environmental noise may be rain noise, and the second environmental noise may be air conditioning noise.

[0078] The user authentication device converts the user speech and the synthesis result into a spectrogram. The user authentication device converts the user speech into a first spectrogram, converts the synthesis result of the user speech and the first environmental noise into a second spectrogram, and converts the synthesis result of the user speech and the second environmental noise into a third spectrogram.

[0079] Here, the spectrogram is a two-dimensional image in which the horizontal axis represents time, the vertical axis represents frequency, and the element value is the amplitude of the frequency component.

[0080] The user authentication device converts the user speech and the synthesis result represented in the time domain into a spectrogram represented in the time domain. The user authentication device can perform spectrogram conversion using various methods, such as FFT (Fast Fourier Transform), STFT (Short Time Fourier Transform), Chroma-STFT, Chroma-CQT, and Chroma-CQT (Constant-Q Transform).

[0081] In another embodiment, the user authentication device may perform synthesis in a spectrogram operation. The user authentication device may convert the audio data of the user's speech into a first spectrogram, convert the first ambient noise into a noise spectrogram, and synthesize the first spectrogram and the noise spectrogram to generate a spectrogram of the speech mixed with the noise.

[0082] The user authentication device generates a reference embedding set from the spectrogram using the embedding model 300 .

[0083] Specifically, the user authentication device inputs the spectrogram into the embedding model 300, and the embedding model 300 processes the spectrogram and outputs probability information including probability values ​​for each category. Here, these categories relate to speakers. The spectrogram is processed in the form of multiple layers by the parameters of the embedding model 300. The embedding model 300 includes an input layer, a hidden layer, and an output layer, and each layer represents an intermediate result of processing the spectrogram. The user authentication device obtains the hidden layer closest to the output layer among the hidden layers in the embedding model 300 as a reference embedding.

[0084] Here, the embedding model 300 is a model of a neural network structure trained to convert a spectrogram into a vector.

[0085] The embedding model 300 can be supervised by a training device. First, as a training data set for the embedding model 300, a training spectrogram converted from training audio data and a label of the training spectrogram are prepared. The label can be in the form of a single-hot encoded vector. The embedding model 300 receives the training spectrogram as input and outputs category probability information. Here, the category probability information is a vector with the same dimension as the label. The training device updates the parameters of the embedding model 300 so that the difference between the output category probability information and the label of the training spectrogram is reduced. In other words, the embedding model 300 is trained to output the label of the training spectrogram from the training spectrogram.

[0086] Alternatively, or in addition, the embedding model 300 may be trained using other training methods such as unsupervised learning or reinforcement learning.

[0087] The embedding model 300 may be configured as a deep neural network and may have different neural network structures. For example, the embedding model 300 may have various neural network structures capable of implementing image processing techniques, such as a convolutional neural network (CNN), a recurrent neural network (RNN), or a combination of RNN and CNN.

[0088] At the same time, each reference embedding is the result of extracting the features of the corresponding spectrogram. That is, each reference embedding is a feature embedding for a user utterance or a synthesis result. In detail, the first reference embedding is a feature embedding of a user utterance collected in a quiet environment. The second reference embedding represents the features of the user utterance in a rainy environment, and the third reference embedding represents the features of the user utterance in an environment where the air conditioner is turned on.

[0089] The user authentication device completes user registration by storing the reference embedding set as the user's personal data.

[0090] The stored set of reference embeddings is later used for user authentication.

[0091] The user authentication device may pre-store reference embedding sets for multiple users.

[0092] Figure 4 is a diagram illustrating user authentication according to an embodiment of the present disclosure.

[0093] refer to Figure 4 , the user authentication device 200 determines whether the vehicle occupant is a registered user based on a comparison between the input embedding of the utterance of the vehicle occupant and a pre-stored reference embedding set for each user, thereby performing a user authentication process.

[0094] First, the user authentication device 200 receives input audio according to the speech of the vehicle occupant. Here, the input audio may include noise depending on the vehicle environment as well as the speech of the occupant. The environmental noise may refer to rain noise, air conditioning noise or wind noise.

[0095] When the user authentication device 200 is installed in the vehicle, the user authentication device 200 requests the occupant to speak a predetermined sentence and receives the occupant's speech of the predetermined sentence through the microphone. Here, the predetermined sentence may be the same as or partially different from the sentence used for user registration. At the same time, when the user authentication device 200 is set in the server, the input audio collected by the vehicle is received from the vehicle.

[0096] The user authentication device 200 converts the input audio into an input embedding. The user authentication device 200 may convert the input audio into an input spectrogram, and transform the input spectrogram into an input embedding using an embedding model.

[0097] Thereafter, the user authentication device 200 determines whether the occupant is a registered user by comparing the input embedding of the occupant with the reference embedding set.

[0098] The user authentication device 200 may calculate a similarity score between the input embedding and the reference embedding set, and determine whether the occupant is a registered user based on the similarity score. Here, the similarity score may refer to cosine similarity, Euclidean distance, or Jacquard similarity for determining the similarity between vectors. In the following, the similarity score represents cosine similarity.

[0099] For example, the user authentication device 200 calculates a similarity score between the input embedding of the occupant and a first reference embedding set of the first user.

[0100] At this time, the first reference embedding set includes a reference embedding for the noise-free speech of the first user and a reference embedding for the result of synthesizing ambient noise with the speech. The user authentication device 200 calculates a similarity score between the input embedding and each reference embedding, and calculates the sum or average of the similarity scores as the similarity score between the input embedding and the first reference embedding set.

[0101] The user authentication device 200 determines whether the occupant is a registered user based on the calculated similarity score. If the calculated similarity score is higher than a preset threshold score, the user authentication device 200 may determine that the occupant is a registered first user.

[0102] Meanwhile, if the calculated similarity score is lower than a preset threshold score, the user authentication device 200 determines that the occupant is not the first user, and compares the input embedding with the second reference embedding set to determine that the occupant is the same person as the second user.

[0103] If the occupant is a registered user, the user authentication device 200 may send a determination result of the occupant registration to the vehicle. When the user authentication device 200 is implemented in the vehicle, the user authentication device 200 grants the occupant authority to control the voice recognition system, and the voice recognition system controls the vehicle or provides the requested service according to the occupant's voice command.

[0104] The user authentication device 200 can accurately determine whether the occupant is a registered user by using the reference embedding for each noise even if the occupant's voice utterance includes noise in the user authentication operation.

[0105] Figure 5 is a flow chart of a user authentication method according to an embodiment of the present disclosure.

[0106] refer to Figure 5 , a user authentication device receives input audio including speech and noise of a vehicle occupant ( S510 ).

[0107] When the user authentication device is implemented in a server, the user authentication device may receive input audio from the vehicle. When the user authentication device is implemented in a vehicle, the user authentication device may obtain input audio through a microphone of the vehicle.

[0108] The noise included in the input audio may include various sounds generated in the vehicle.

[0109] The user authentication device converts the input audio into an input embedding (S520).

[0110] The user authentication device converts the input audio into an input spectrogram using Fourier transform, and converts the input spectrogram into an input embedding using an embedding model.

[0111] The user authentication device determines whether the occupant is a registered user based on a comparison between the input embedding and a pre-stored reference embedding set ( S530 ).

[0112] Here, the reference embedding set includes multiple reference embeddings, and the reference embeddings include feature embeddings for low-noise speech of a registered user and feature embeddings for synthesis results between the speech of the registered user and multiple environmental noises. The multiple environmental noises are acoustic noises with different acoustic characteristics.

[0113] Meanwhile, the registered user may be registered in an environment in which at least one of a noise level before the registered user's speech and a noise level after the registered user's speech is lower than a noise threshold.

[0114] For user authentication, the user authentication device calculates a similarity score between the input embedding and the reference embedding set. The user authentication device determines whether the occupant is a registered user based on the similarity score.

[0115] If the calculated similarity score is higher than the threshold score, the user authentication device may determine that the occupant is a registered user.

[0116] If the occupant is a registered user, the user authentication device may transmit a determination result of the occupant registration to the vehicle. When the user authentication device is implemented in the vehicle, the user authentication device may grant the occupant authority to control the voice recognition system.

[0117] As described above, according to the embodiments of the present disclosure, the performance of user authentication can be improved based on voice recognition even in various noise environments of a vehicle.

[0118] The effects of the present disclosure are not limited to the above-mentioned effects, and other effects that are not mentioned can be clearly understood by those skilled in the art from the accompanying description.

[0119] At least some of the components described in the exemplary embodiments of the present disclosure may be implemented as hardware elements, including at least one of a digital signal processor (DSP), a processor, a controller, an application specific IC (ASIC), a programmable logic device (FPGA, etc.), other electronic components, or a combination thereof. In addition, at least some of the functions or processes described in the exemplary embodiments may be implemented as software, and the software may be stored in a recording medium. At least some of the components, functions, and processes described in the exemplary embodiments of the present disclosure may be implemented as a combination of hardware and software.

[0120] The method according to the exemplary embodiment of the present disclosure may be written as a program executable on a computer, and may also be implemented as various recording media such as a magnetic storage medium, an optical reading medium, a digital storage medium, and the like.

[0121] The different techniques described herein may be implemented in digital electronic circuits, or in computer hardware, firmware, software, or a combination thereof. The implementation may be in the form of a computer program tangibly embodied in a computer program product (i.e., an information carrier, such as a machine-readable storage device (computer-readable medium) or a propagated signal) for processing or controlling the operation of a data processing device (e.g., a programmable processor, a computer, or multiple computers). Computer programs such as the computer programs mentioned above may be written in any form of programming language, including compiled or interpreted languages, and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment. The computer program may be deployed to run on a single computer or multiple computers at one site or distributed on multiple sites and interconnected by a communication network.

[0122] In addition, the components of the present disclosure may use integrated circuit structures such as memories, processors, logic circuits, lookup tables, etc. These integrated circuit structures perform each function described herein by controlling one or more microprocessors or other control devices. In addition, the components of the present disclosure may be specifically implemented by a part of a program or code including one or more executable instructions for performing a specific logical function, and executed by one or more microprocessors or other control devices. In addition, the components of the present disclosure may include or be implemented as a central processing unit (CPU), a microprocessor, etc. that performs various functions. In addition, the components of the present disclosure may store instructions executed by one or more processors in one or more memories.

[0123] For example, processors suitable for processing computer programs include general and special purpose microprocessors, and one or more processors of any kind of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The basic elements of a computer may include at least one processor that executes instructions and one or more memory devices that store instructions and data. Typically, a computer will also include one or more large-capacity storage devices (e.g., magnetic disks, magneto-optical disks, or optical disks) for storing data or operably coupled to receive data from or transmit data to or both. For example, information carriers suitable for embodying computer program instructions and data include semiconductor memory devices, such as magnetic media (such as hard disks, floppy disks, and tapes), optical media (such as compact disk read-only memory (CD-ROM) and digital video disks (DVD)), magneto-optical media (such as floppy disks), read-only memory (ROM), random access memory (RAM), flash memory, erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), etc. The processor and memory may be supplemented or incorporated into a dedicated logic circuit.

[0124] The processor may execute an operating system and software applications executed on the operating system. In addition, the processor device may access, store, manipulate, process, and generate data in response to software execution. For convenience, there are cases where a single processor device is used, but those skilled in the art will understand that the processor device may include multiple processing elements and / or multiple types of processing elements. For example, the processor device may include multiple processors or a single processor and a single controller. Other processing configurations (such as parallel processors) are also possible.

[0125] Additionally, non-transitory computer readable media may be any available media that can be accessed by a computer and may include both computer storage media and transmission media.

[0126] This specification includes details of various specific implementations, but they should not be construed as limiting the scope of any invention or the scope of what is claimed, and should be construed as descriptions of features that may be unique to a particular embodiment of a particular invention. In the context of each embodiment, the specific features described herein may also be implemented in conjunction with a single embodiment. Conversely, the various features described in the context of a single embodiment may also be implemented in multiple embodiments independently or in any suitable subcombination. Further, although features may operate in a particular combination and may initially be described as being claimed as such, in some cases, one or more features from the claimed combination may be excluded from the combination, and the claimed combination may be modified into a subcombination or a variant of the subcombination.

[0127] Likewise, although operations are described in a particular order in the accompanying drawings, it should not be understood that such operations must be performed in the particular order or sequential order shown to achieve the desired result or that all described operations should be performed. In some cases, multitasking and parallel processing can be advantageous. In addition, the separation of the various device components of the above-described embodiments should not be understood as requiring such separation in all embodiments, and it should be understood that the described program components and devices can generally be integrated in a single software product or packaged in multiple software products.

[0128] The above description is only an illustration of the technical concept of the present embodiment. A person of ordinary skill in the art can make various modifications and changes without departing from the basic features of each embodiment. Therefore, the present embodiment is not intended to limit but to describe the technical concept of the present embodiment. The scope of the technical concept of the embodiment is not limited by these embodiments. The protection scope of various embodiments should be interpreted by the attached claims. All technical concepts falling within the scope of their equivalents should be interpreted as being included within the scope of the present embodiment.

Claims

1. A device for user authentication of an occupant in a vehicle, the device comprising: A memory configured to store a reference embedding set, the reference embedding set comprising feature embeddings for an utterance of a registered user and feature embeddings for a synthesis result between the utterance of the registered user and a plurality of environmental noises; a user interface configured to receive input audio including speech and noise of an occupant of the vehicle; as well as A processor is configured to convert the input audio into an input embedding and determine whether the occupant is the registered user based on a comparison between the input embedding and the reference embedding set.

2. The device according to claim 1, wherein: The user interface includes a microphone for capturing the occupant's speech.

3. The device according to claim 1, wherein: The processor is configured to convert the input audio into an input spectrogram, and convert the input spectrogram into the input embedding using an embedding model.

4. The device according to claim 1, wherein: The processor is configured to calculate a similarity score between the input embedding and the set of reference embeddings, and determine whether the occupant is the registered user based on the similarity score.

5. The device according to claim 1, wherein: The plurality of environmental noises include noises having different acoustic characteristics.

6. The device according to claim 1, wherein: The registered user is registered in an environment where a noise level before or after an utterance of the registered user is lower than a noise threshold.

7. The device according to claim 1, wherein: The processor is configured to receive the input audio from the vehicle and transmit a determination of the registration of the occupant to the vehicle.

8. A vehicle comprising the device of claim 1.

9. A computer-implemented method for user authentication of a vehicle, the computer-implemented method comprising: receiving, by a user interface, input audio including speech and noise of an occupant of the vehicle; converting, by a processor, the input audio into an input embedding; as well as The processor determines whether the occupant is a registered user based on a comparison between the input embedding and a pre-stored reference embedding set, wherein the reference embedding set includes a feature embedding for the registered user's speech and a feature embedding for a synthesis result between the registered user's speech and multiple environmental noises.

10. The method according to claim 9, wherein: The user interface includes a microphone.

11. The method according to claim 9, wherein: The conversion includes: Converting the input audio into an input spectrogram; and The input spectrogram is converted into the input embedding using an embedding model.

12. The method according to claim 9, wherein: The determination includes: computing a similarity score between the input embedding and the set of reference embeddings; and Based on the similarity score, it is determined whether the occupant is the registered user.

13. The method according to claim 9, wherein: The multiple environmental noises have different acoustic characteristics.

14. The method according to claim 9, wherein: The registered user is registered in an environment where a noise level before or after an utterance of the registered user is lower than a noise threshold.

15. The method according to claim 9, further comprising: A determination result of the registration of the occupant is transmitted to the vehicle.

16. A non-transitory computer-readable medium containing program instructions executed by a processor, the non-transitory computer-readable medium comprising: program instructions to receive input audio including speech and noise of an occupant of a vehicle; Converting the input audio into input embedded program instructions; as well as Program instructions for determining whether the occupant is a registered user based on a comparison between the input embedding and a pre-stored reference embedding set, the reference embedding set including feature embeddings for the registered user's speech and feature embeddings for synthesis results between the registered user's speech and multiple environmental noises.

17. The non-transitory computer readable medium of claim 16, further comprising: Program instructions for converting the input audio into an input spectrogram; as well as Program instructions for converting the input spectrogram into the input embedding using an embedding model.

18. The non-transitory computer readable medium of claim 17, further comprising: program instructions for computing a similarity score between the input embedding and the set of reference embeddings; as well as Program instructions to determine whether the occupant is the registered user based on the similarity score.

19. The non-transitory computer readable medium of claim 16, wherein: The multiple environmental noises have different acoustic characteristics.

20. The non-transitory computer readable medium of claim 16, wherein: The registered user is registered in an environment where a noise level before or after an utterance of the registered user is lower than a noise threshold.

Images