Method, computer program and device for resolving ambiguities when processing a speech input

Abstract

Method for resolving ambiguities when processing a speech input (SE), comprising the steps of:- receiving (10) a speech input (SE) from a user;- applying (11) a speech recognition to the speech input (SE);- determining a first result list (EL1) of the speech recognition based on a confidence of the speech recognition, wherein the first result list (EL1) has an ambiguity;- determining (12) salience values (Si) for elements (Ei) of the first result list (EL1) by evaluating additional information (ZI) available for the elements (Ei);- processing (13) the first result list (EL1) based on the salience values (Si) to provide a second result list (EL2);and- processing (14) the second result list (EL2) to determine a response to the speech input (SE), wherein an action is carried out (15) in response to the speech input (SE) if the second result list (EL2) comprises only one element (Ei) or exactly one element (Ei) of the second result list (EL2) has a salience value (Si) that is greater than or equal to a threshold value (Sth), wherein the threshold value (Sth) depends on a context of the speech input (SE);

Description

[0001] The present invention relates to a method, a computer program with instructions, and a device for resolving ambiguities when processing a speech input. The invention further relates to a means of transportation in which a method or device according to the invention is used.

[0002] Modern motor vehicles are increasingly implementing voice-based approaches for user interaction. These involve both processing a user's voice input and providing information to the user in natural language.

[0003] With regard to speech output, US 2019 / 0096387 A1 describes a method for text-to-speech preprocessing for speech synthesis. First, an automated speech recognition output is received, comprising an n-best list and associated confidence scores. Text-to-speech preprocessing is performed on the n-best list and the associated confidence scores to generate a message to be read aloud. The message includes a read-aloud command. The method further comprises sending the message to be read aloud to a speech synthesizer for generating an audible signal based on the message, which is then played back by an audio device.

[0004] Traditional voice control and digital assistants always face the challenge of understanding human speech. Therefore, improved speech recognition solutions are continually being developed.

[0005] For example, US 2017 / 0 213 551 A1 describes a speech recognition system for a vehicle. The system includes a processor programmed to recognize speech using domain-specific language models and acoustic models. The processor is configured to recognize speech only using the acoustic model in response to the acoustic model having a confidence rating for recognized speech that falls within a predetermined range defined relative to a confidence rating for the domain-specific language model.

[0006] DE 10 2018 007 102 B3 describes a method for voice control of a motor vehicle. In the method, a computing unit determines a voice command from an input voice signal. To do this, at least one sound sequence is first determined from the voice signal. The sound sequence is assigned to the voice command using a voice database, wherein several voice commands are stored in the voice database for a plurality of sound sequences, along with situation-dependent usage frequencies and weights. Based on the respective situation in which the voice signal was emitted, usage frequencies and weights are determined from the voice database, from which a confidence level is formed. The voice command with the highest confidence level is then determined as the voice command.

[0007] US 2008 / 0 215 320 A1 describes a method for reducing recognition errors through contextual relationships between multiple dialogue rounds. The method analyzes a dialogue history using an evolutionary approach based on massive parallelism to obtain a rule set that describes the contextual relationship between the dialogue rounds. Based on the rule set and the recognition result of a speech recognition system, the recognition result is re-evaluated, and the confidence measure of the re-evaluated recognition result is measured. After each successful dialogue round, the rule set is dynamically adjusted.

[0008] DE 10 2014 010 315 A1 describes a method for processing speech inputs at a human-machine interface, wherein the speech inputs indicate destination inputs. The method comprises the steps of providing at least one hypothesis for the speech input determined by speech recognition, extracting address components from the at least one provided hypothesis, and generating an address set for each hypothesis based on the extracted address components such that the generated address set can be assigned to at least one valid address.

[0009] Regardless of the quality of speech recognition, it can happen that the user of a digital assistant omits relevant information for performing an action during voice input, resulting in ambiguities. For example, the voice input may consist of a request to "Call Sarah." If there are multiple Sarahs in the phone book, the voice input is not clear. Furthermore, even high-quality systems can produce misidentifications during speech recognition. Both cases lead to a list of results that must be displayed to the system user so that they can make a selection. The user is therefore repeatedly confronted with lists to which they must pay attention in order to make a selection. This inhibits the flow of interaction, the user's willingness to engage with voice control, and the enjoyment of using the system.Furthermore, if such a digital assistant is used in a motor vehicle, the user is usually forced to take their eyes off the road when a list is displayed in order to make a selection. However, in motor vehicles, distraction-free operation must be the focus.

[0010] Ambiguities will not be resolved in the future solely through more robust speech recognition. Spoken language is inherently ambiguous, and users will continue to omit relevant information for resolving ambiguities.

[0011] DE 199 37 490 A1 describes a device for inputting control commands for comfort devices using a control unit connected to at least one database, an input unit, and an output unit. The user enters a non-standardized command via a first interaction, which the control unit breaks down into command attributes. The database is then searched for the recorded command attributes, and the result is checked for uniqueness. The control unit repeatedly queries the user for further command attributes until the search result is unique.

[0012] It is an object of the invention to provide improved solutions for resolving ambiguities when processing speech inputs.

[0013] This object is achieved by a method having the features of claim 1, by a computer program with instructions according to claim 8, and by a device having the features of claim 9. Preferred embodiments of the invention are the subject of the dependent claims.

[0014] According to a first aspect of the invention, a method for resolving ambiguities when processing a speech input comprises the steps of: - Receiving a voice input from a user; - Applying speech recognition to voice input; - determining a first speech recognition result list based on a speech recognition confidence, wherein the first result list has an ambiguity; - Determining salience values for elements of the first result list by evaluating additional information available for the elements; - processing the first result list based on the salience values to provide a second result list; and - Processing the second result list to determine a response to the speech input, wherein an action is performed in response to the speech input if the second result list comprises only one element or exactly one element of the second result list has a salience value that is greater than or equal to a threshold value, wherein the threshold value depends on a context of the speech input.

[0015] According to a further aspect of the invention, a computer program includes instructions which, when executed by a computer, cause the computer to perform the following steps for resolving ambiguities when processing a speech input: - Receiving a voice input from a user; - Applying speech recognition to voice input; - determining a first speech recognition result list based on a speech recognition confidence, wherein the first result list has an ambiguity; - Determining salience values for elements of the first result list by evaluating additional information available for the elements; - Processing the first result list based on the salience values to provide a second result list; - Processing the second result list to determine a response to the speech input, wherein an action is performed in response to the speech input if the second result list comprises only one element or exactly one element of the second result list has a salience value that is greater than or equal to a threshold value, wherein the threshold value depends on a context of the speech input.

[0016] The term "computer" should be understood broadly. In particular, it also includes control units, integrated systems, and other processor-based data processing devices.

[0017] The computer program may, for example, be made available for electronic retrieval or stored on a computer-readable storage medium.

[0018] According to a further aspect of the invention, an apparatus for resolving ambiguities when processing a speech input comprises: - a receiving module for receiving a user's voice input; - a speech recognition module for applying speech recognition to the speech input and for determining a first speech recognition result list based on a confidence of the speech recognition, wherein the first result list has an ambiguity; - an analysis module for determining saliency values for elements of the first result list by evaluating additional information available for the elements and for processing the first result list based on the saliency values to provide a second result list; and - a processing module for processing the second result list to determine a response to the speech input, wherein an action is performed in response to the speech input if the second result list comprises only one element or exactly one element of the second result list has a salience value that is greater than or equal to a threshold value, wherein the threshold value depends on a context of the speech input.

[0019] In previous solutions, the result lists only show the results based on the user's interpreted voice input. The basis for the selection and sorting of list entries is the confidence of the speech recognition. Targeted queries can also be used to resolve ambiguities. The inventive solution extends this approach so that, in the event of ambiguity, the digital assistant uses the information available to determine salience values for the various elements of the result list. The first result list can, for example, be presented as a selection list or as an n-best list. The salience values help to evaluate the results delivered by speech recognition and to mark individual results as salient. In the best case, a best estimate is then sufficient to intelligently resolve the ambiguity without actively prompting the user to select from list entries.This is particularly relevant in light of driver distraction and established heuristics for evaluating the holistic user experience. Salience values can be a specific weighting of individual values. Preferably, however, they can also be based on previously defined social norms within a market. For example, a mobile phone number may be preferred over a landline number.

[0020] Through an intelligent best estimate, which is derived from the consideration of various information, the digital assistant becomes more user-centric, intelligent, personalized, and relevant to the user's everyday life. Motor vehicle drivers, in particular, are less frequently confronted with ambiguities that they can only resolve by shifting their gaze from the road to the results list display. This is particularly relevant for markets where legal regulations regarding driver distraction exist. The use of the inventive solution increases trust in the assistant. Barriers to operating a technical system using voice commands are further reduced.

[0021] To determine the salience values, additional information available for the elements is evaluated, e.g. a history of previously processed speech inputs or information from a source assigned to the originator of the speech input. The information from the source assigned to the originator of the speech input can, for example, be the originator's preferences or favorites defined by the originator in a contact directory. For example, if a user wants to call someone like "Mara" from their contacts, they can do so by saying "Call Mara!". How certain can the speech dialog system be that the user said "Call Mara!" and not "Call Lara!"? Mara and Lara can have a similar speech recognition confidence, so that both are returned as a result. However, the speech dialog system knows that Mara has been called, for example, 35 times so far, while Lara has not been called once.It is therefore very likely that the voice input was actually “Call Mara!”

[0022] According to the invention, an action is executed in response to the voice input if the second result list comprises only one element or if exactly one element of the second result list has a salience value that is greater than or equal to a threshold value. If speech recognition delivers only a single result or, alternatively, if multiple results are present, only a single result is sufficiently salient, it is unnecessary to bother the user with a dialog to resolve ambiguities. Instead, the action assigned to the single element or the single sufficiently salient element can be executed directly. The context of the voice input is taken into account, for example by using context-dependent threshold values or by specifying contexts in which a dialog must take place if multiple results are present. This can be particularly the case in security-relevant contexts.

[0023] According to one aspect of the invention, processing the first result list based on the saliency values comprises ordering the elements of the first result list. A result list ordered in this way facilitates further use of the result list, since the most salient and thus most probable elements are at the top of the list and can thus be processed or captured by the user with priority.

[0024] According to one aspect of the invention, in all other cases, the user is provided with a choice between elements of the second result list in response to the voice input. In cases where an ambiguity remains despite the use of the salience values, the user is given a choice. The selection of an element from the result list can be made, for example, voice-based, haptic, or using a multifunction steering wheel. This allows for the resolution of remaining ambiguities.

[0025] According to one aspect of the invention, the elements of the second result list are filtered to provide the selection option. The filtering allows, for example, the elements of the second result list to be divided into groups, e.g., into highly salient elements, moderately salient elements, and less salient elements. This generally allows the user to select the desired element very quickly.

[0026] According to one aspect of the invention, when filtering the elements of the second result list, elements whose salience value is lower than the threshold are discarded. In this way, less likely elements are removed from the result list and thus not displayed to the user. This increases the clarity of the result list and thus reduces the time the user needs to spend selecting an element from the result list.

[0027] Preferably, a method or device according to the invention is used in a (semi-)autonomously or manually controlled means of transport. The means of transport can be, in particular, a motor vehicle, but also a ship, a manned or unmanned aircraft, e.g., a drone or a Volocopter, etc. Of course, the solution according to the invention can also be used in other application scenarios, e.g., in a user terminal. Examples of such user terminals include smartphones, tablets, or portable and stationary computers.

[0028] Further features of the present invention will become apparent from the following description and the appended claims taken in conjunction with the figures. Fig. 1 schematically shows a method for resolving ambiguities when processing speech input; Fig. 2 shows a first embodiment of an apparatus for resolving ambiguities when processing a speech input; Fig. 3 shows a second embodiment of an apparatus for resolving ambiguities when processing speech input; Fig. Figure 4 schematically represents a means of transport in which a solution according to the invention is implemented; Fig. 5 shows the language processing process until the second result list is provided; Fig. Figure 6 shows the further course of language processing when there is no ambiguity; Fig. 7 shows a display when an action is executed; Fig. Figure 8 shows the further course of language processing when there is an ambiguity between two elements; Fig. 9 shows a display with a selection option; Fig. 10 shows the further course of language processing when there is an ambiguity between more than two elements; and Fig. 11 shows a display with a selection list.

[0029] To better understand the principles of the present invention, embodiments of the invention are explained in more detail below with reference to the figures. It is understood that the invention is not limited to these embodiments and that the described features may also be combined or modified without departing from the scope of the invention as defined in the appended claims.

[0030] Fig. 1 schematically shows a method for resolving ambiguities when processing speech input. In a first step, a user's speech input is received 10. Speech recognition is then applied to the speech input 11. The result of the speech recognition is a first results list, which may contain an ambiguity. The basis for selecting and sorting the list entries is the confidence of the speech recognition. Salience values are then determined 12 for the elements of the first speech recognition results list. For this purpose, additional information available for the elements can be evaluated, e.g. a history of previously processed speech inputs or information from a source that is assigned to an originator of the speech input. Based on the salience values, the first results list is processed 13 to provide a second results list.The elements of the first result list can, for example, be ordered according to their salience values. The second result list is finally processed 14 to determine a response to the voice input. In response to the voice input, for example, an action can be carried out 15 or a choice between elements of the second result list can be provided 16. In particular, an action is carried out in response to the voice input if the second result list contains only one element or exactly one element of the second result list has a salience value that is greater than or equal to a threshold value. The threshold value depends on the context of the voice input. In order to provide the choice, the elements of the second result list can be filtered beforehand, e.g. by discarding elements whose salience value is lower than a threshold value.

[0031] Fig. 2 shows a simplified schematic representation of a first embodiment of a device 20 for resolving ambiguities when processing a speech input SE. The device 20 has an input 21 via which a receiving module 22 can receive a user's speech input SE and additional information ZI. A speech recognition module 23 applies speech recognition to the speech input SE. The result of the speech recognition is a first result list, which may contain an ambiguity. The basis for the selection and sorting of the list entries is the confidence of the speech recognition. An analysis module 24 determines salience values for the elements of the first result list of the speech recognition. For this purpose, the analysis module 24 can evaluate additional information ZI available for the elements, e.g. a history of previously processed speech inputs or information from a source associated with an originator of the speech input.In addition, the analysis module 24 processes the first result list based on the saliency values to provide a second result list. For example, the elements of the first result list can be ordered according to the saliency values. A processing module 25 finally processes the second result list to determine a response to the voice input. In response to the voice input, for example, an action can be performed or a selection option between elements of the second result list can be provided. In particular, an action is performed in response to the voice input if the second result list contains only one element or exactly one element of the second result list has a saliency value that is greater than or equal to a threshold value. The threshold value depends on the context of the voice input. When providing the selection option, the elements of the second result list can be filtered beforehand, e.g.by discarding elements whose salience value is less than a threshold. Information about the specific response or data resulting from the response can be output via an output 27 of the device 20.

[0032] The reception module 22, the speech recognition module 23, the analysis module 24, and the processing module 25 can be controlled by a control module 26. Settings of the reception module 22, the speech recognition module 23, the analysis module 24, the processing module 25, or the control module 26 can be changed via a user interface 29. The data generated in the device 20 can be stored in a memory 28 of the device 20 if necessary, for example, for later evaluation or for use by the components of the device 20. The reception module 22, the speech recognition module 23, the analysis module 24, the processing module 25, and the control module 26 can be implemented as dedicated hardware, for example, as integrated circuits. Of course, they can also be partially or completely combined or implemented as software running on a suitable processor, for example, a GPU.The input 21 and the output 27 can be implemented as separate interfaces or as a combined bidirectional interface.

[0033] Fig. 3 shows a simplified schematic representation of a second embodiment of a device 30 for resolving ambiguities when processing a speech input. The device 30 has a processor 32 and a memory 31. For example, the device 30 is a computer, a workstation, or a control unit. Instructions are stored in the memory 31 which, when executed by the processor 32, cause the device 30 to perform the steps according to one of the described methods. The instructions stored in the memory 31 thus embody a program executable by the processor 32 which implements the method according to the invention. The device has an input 33 for receiving speech inputs and additional information. Data generated by the processor 32 are provided via an output 34. Furthermore, they can be stored in the memory 31.The input 33 and the output 34 can be combined to form a bidirectional interface.

[0034] The processor 32 may include one or more processor units, such as microprocessors, digital signal processors, or combinations thereof.

[0035] The memories 28, 31 of the described embodiments can have volatile and / or non-volatile memory areas and can comprise a wide variety of storage devices and storage media, for example hard disks, optical storage media or semiconductor memories.

[0036] Fig. 4 schematically illustrates a means of transport 40 in which a solution according to the invention is implemented. In this example, the means of transport 40 is a motor vehicle. The motor vehicle has a series of assistance systems 41, one of which is shown as an example. Furthermore, a sensor system 42 is present, which is used by the assistance systems 41 and can be used to acquire information about the surroundings of the motor vehicle. Operation of the motor vehicle can be partially voice-based. The motor vehicle therefore has a device 20 according to the invention for resolving ambiguities when processing a voice input from a driver or another user. Further components of the motor vehicle are, for example, a navigation system 43 and a data transmission unit 44. A connection to a service provider can be established by means of the data transmission unit 44.A memory 45 is provided for storing data. Data exchange between the various components of the motor vehicle takes place via a network 46.

[0037] The following will be based on Fig. 5 to Fig. 11 explains the processing of a speech input. As an example, a user request to perform an action is used. The desired action here is a call to a person named "Sarah." The process described is based on the assumptions that the user does not provide all the relevant information to perform the action, or that speech recognition does not necessarily produce a clear result. Furthermore, it is assumed that there are no homophonic elements.

[0038] Fig. Figure 5 shows the speech processing sequence up to the provision of the second result list EL2. In a step S1, the user starts a speech dialogue system by pressing a key or using a defined voice command. Subsequently, in a step S2, a voice input SE is made. Examples of possible voice inputs SE are "Call Sarah!", "I want to call Sarah!", or "Connect me with Sarah!". In a step S3, speech recognition now takes place, which generates a first result list EL1 with one or more elements E. i Following the speech recognition, in a step S4, salience values for the elements E i and processing of the first result list EL1. In step S5, it is now checked whether the resulting second result list EL2 contains an element E i or more than one element E iDepending on the result of the check, the process branches. If the second result list EL2 contains more than one element E i , a check is carried out in step S6 to determine how many elements E i a salience value S i above a threshold value S th None of the elements E i a salience value S i above the threshold S th , a distinction is also made as to whether the second result list EL2 contains two elements E i or more than two elements E i Depending on the result of the review, the process branches off.

[0039] Fig. Figure 6 shows the further flow of speech processing when there is no ambiguity. This is the case when the second result list contains only one element or when exactly one element has a saliency value above the threshold. Elements with a saliency value below the threshold are discarded in this case. Since there is no ambiguity in the speech input, the desired action is executed in step S7, i.e., a call to "Sarah" is initiated. In step S8, a corresponding speech output SA is output, e.g., in the form "Sarah is being called." In addition, the user can be informed about the execution of the action via a display. A corresponding display is shown in Fig. 7 shown.

[0040] Fig. Figure 8 shows the further sequence of language processing when an ambiguity exists between two elements. This is particularly the case when exactly two elements have a salience value above the threshold. Any further elements with a salience value below the threshold are discarded in this case. An ambiguity between two elements also exists when the second result list contains exactly two elements and both have a salience value below the threshold. In step S9, it is checked whether the elements have a suitable distinguishing feature. If this is the case, the user is presented with a choice in the form of an either / or question in step S10. In step S11, a corresponding voice output SA occurs, e.g., in the form “Do you want to call Sarah Meier or Sarah Schmidt?”. In addition, the user can be informed of the choice via a display.A corresponding advertisement is in . Fig. 9. If no suitable distinguishing feature was found, a selection option is provided in a step S12 in the form of a selection list. In a step S13, a corresponding voice output SA is provided, e.g., in the form "I have found two Sarahs. Which one would you like to call?". In addition, the user can be informed of the selection option via a display. For this purpose, the Fig. The display shown in Figure 9 can be used. In both cases, an interaction with the user takes place in a step S14, which may also include a dialog. During this interaction, one of the elements can be selected, but the result list can also be changed or the results re-evaluated. If the ambiguity is resolved after the interaction has been completed, an action is finally executed in a step S15.

[0041] Fig. Figure 10 shows the further sequence of language processing when an ambiguity exists between more than two elements. This is the case when more than two elements have a saliency value above the threshold or when the second result list comprises more than two elements and all elements have a saliency value below the threshold. In a step S16, a best estimate is made and the user is provided with a selection option in the form of a selection list. In a step S17, a corresponding voice output SA is provided, e.g., in the form "Do you want to call Sarah Müller?". In addition, the user can be informed of the selection option via a display. A corresponding display is shown in Fig.11. Subsequently, in step S18, which may also include a dialog, an interaction with the user takes place. During this interaction, one of the items from the selection list can be selected, but the results list can also be changed or the results re-evaluated. If the ambiguity is resolved after the interaction has been completed, an action is finally executed in step S19. List of reference symbols 10 Receiving a voice input 11 Applying speech recognition to voice input 12 Determining salience values for elements of a first result list 13 Editing the first result list to provide a second result list 14 Processing the second result list 15 Performing an action 16 Providing a choice 20 Device 21 Entrance 22 Receiver module 23 Speech recognition module 24 Analysis module 25 Processing module 26 Control module 27 Exit 28 storage 29 User interface 30 Device 31 storage 32 processor 33 Entrance 34 Exit 40 means of transport 41 Assistance system 42 Sensor technology 43 Navigation system 44 Data transmission unit 45 storage 46 Network E i element EL1 First results list EL2 Second Results List S1-S19 Process steps SA voice output SE Voice input S i Salience value S th Threshold ZI Additional Information

Claims

[1] A method for resolving ambiguities when processing a speech input (SE), comprising the steps: - receiving (10) a voice input (SE) from a user; - applying (11) speech recognition to the speech input (SE); - determining a first result list (EL1) of the speech recognition based on a confidence of the speech recognition, wherein the first result list (EL1) has an ambiguity; - Determination (12) of salience values (S i ) for elements (E i ) of the first result list (EL1) by evaluating for the elements (E i ) available additional information (ZI); - Editing (13) the first result list (EL1) based on the salience values (S i ) to provide a second result list (EL2); and - processing (14) the second result list (EL2) to determine a reaction to the speech input (SE), wherein an action is carried out (15) in response to the speech input (SE) if the second result list (EL2) contains only one element (E i ) or exactly one element (E i ) of the second result list (EL2) a salience value (S i ) that is greater than or equal to a threshold value (S th ), where the threshold value (S th ) depends on a context of speech input (SE). [2] Method according to claim 1, wherein the additional information (ZI) comprises a history of previously processed speech inputs (SE) or information from a source associated with an originator of the speech input (SE). [3] Method according to claim 1 or 2, wherein the processing (13) of the first result list (EL1) is carried out on the basis of the salience values (S i ) an ordering of the elements (E i) of the first result list (EL1). [4] Method according to one of the preceding claims, wherein in all other cases the user is given the option of selecting between elements (E i ) of the second result list (EL2) (16). [5] Method according to claim 4, wherein for providing (16) the selection option the elements (E i ) of the second result list (EL2). [6] Method according to claim 5, wherein in filtering the elements (E i ) of the second result list (EL2) elements (E i ) whose salience value (S i ) is smaller than the threshold value (S th ). [7] A computer program comprising instructions which, when executed by a computer, cause the computer to carry out the steps of a method according to any one of claims 1 to 6 for resolving ambiguities when processing a speech input (SE). [8] Device (20) for resolving ambiguities when processing a speech input (SE), comprising: - a receiving module (22) for receiving (10) a voice input (SE) of a user; - a speech recognition module (23) for applying (11) a speech recognition to the speech input (SE) and for determining a first result list (EL1) of the speech recognition on the basis of a confidence of the speech recognition, wherein the first result list (EL1) has an ambiguity; - an analysis module (24) for determining (12) salience values (S i ) for elements (E i ) of the first result list (EL1) by evaluating for the elements (E i) available additional information (ZI) and for processing (13) the first result list (EL1) based on the salience values (S i ) to provide a second result list (EL2); and - a processing module (25) for processing (14) the second result list (EL2) to determine a reaction to the speech input (SE), wherein an action is carried out (15) in response to the speech input (SE) if the second result list (EL2) contains only one element (E i ) or exactly one element (E i ) of the second result list (EL2) a salience value (S i ) that is greater than or equal to a threshold value (S th ), where the threshold value (S th ) depends on a context of speech input (SE). [9] Means of transport (40), wherein the means of transport (40) comprises a device (20) according to claim 8 or is arranged to carry out a method according to one of claims 1 to 6 for resolving ambiguities when processing a speech input (SE).

Images