Chaos testing for voice-enabled devices

Abstract

A computer-implemented method, computer program product, and computer system for testing a voice assistant device may include one or more processors configured to receive test data from a database, wherein the test data may include a first set of encoding parameters and a first user utterance having an expected device response (202). Additionally, the one or more processors may be configured to generate a first modified user utterance by applying the first set of encoding parameters to the first user utterance, wherein the first modified user utterance is acoustically different from the first user utterance (204). The one or more processors may be configured to audibly present the first modified user utterance to a voice assistant assistant device, receive a first device response from the voice assistant assistant device, and determine whether the first voice assistant assistant response is substantially similar to the expected device response (206).

Description

Technical Field

[0001] The present invention relates generally to the field of device testing, and more particularly to testing voice-enabled devices using machine learning techniques. Background Art

[0002] Interactive voice assistants are able to detect voice commands, process them, and determine the user's intent to perform a specific task. Voice assistants interact with users and evolve to engage in a continuous conversation. Voice assistants have become the primary interface for virtual assistant systems.

[0003] However, voice assistants can be hampered by noisy environments, making it more challenging to determine the precise meaning of a voice command or user utterance. Furthermore, voice assistants are not tested with varying degrees of voice types and modulation, making it difficult for voice assistants to discern naturally modulated words or phrases from the standard words or phrases used during testing. Summary of the Invention

[0004] Embodiments of the present invention disclose a computer-implemented method, computer program product, and system for testing a voice-assistant device. The computer-implemented method for testing a voice-assistant device may include one or more processors configured to receive test data from a database, the test data including a first set of encoding parameters and a first user utterance with an expected device response. The one or more processors may also be configured to generate a first modified user utterance by applying the first set of encoding parameters to the first user utterance, wherein the first modified user utterance is acoustically different from the first user utterance.

[0005] The one or more processors may also be configured to audibly present the first modified user utterance to a voice assistant device and receive a first voice assistant response from the voice assistant device, wherein the first voice assistant response is a response to the first modified user utterance. The first modified user utterance may be audibly presented to the voice assistant device by outputting a first audio signal via a speaker communicatively connected to the one or more processors, wherein the first voice assistant response received from the voice assistant device may be detected as a second audio signal by a microphone communicatively connected to the one or more processors. In addition, the one or more processors may be further configured to determine whether the first voice assistant response is substantially similar to the expected device response.

[0006] In response to determining that the first voice assistant response is substantially similar to the first expected device response, the one or more processors may be configured to stop receiving additional test data. In addition, in response to determining that the first voice assistant response is not substantially similar to the expected device response, the one or more processors may be configured to identify a first edge case based on at least the first set of encoding parameters and the first modified user utterance.

[0007] Furthermore, a computer-implemented method for testing a voice assistant device may include one or more processors configured to determine a second set of encoding parameters based on at least one of the first set of encoding parameters, the first modified user utterance, and the first edge source condition. The one or more processors may also be configured to generate a second modified user utterance by applying the second set of encoding parameters to the first user utterance, wherein the second modified user utterance is acoustically different from the first user utterance and the first modified user utterance. The one or more processors may further be configured to audibly present the second modified user utterance to the voice assistant device, receive a second voice assistant response, and determine whether the second voice assistant response is substantially similar to the expected device response. In addition, in response to determining that the second voice assistant response is substantially similar to the expected device response, the one or more processors may be configured to stop receiving additional test data including the first user utterance. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a functional block diagram illustrating a distributed data processing environment for testing a speech assistance device according to an embodiment of the present invention;

[0009] Figure 2 This is a description of an embodiment of the present invention. Figure 1 A flowchart of the operational steps of a computer-implemented method for testing a speech assistant device on a server computer within a distributed data processing environment; and

[0010] Figure 3 Describes the implementation of the present invention for testing Figure 1 A block diagram of components of a server computer of a computer-implemented method of a speech assistant device within a distributed data processing environment. Specific embodiments

[0011] Consumer experiences are reaching new heights every day. At the forefront is interactive voice assistance. Currently, a variety of artificial intelligence ("AI") and voice assistant ("VA") systems are available on the market. Like any other product, VA technology continues to improve with breakthroughs in voice recognition and supporting software. VA devices are capable of interacting with users and are being refined to simulate human responses with continuous conversation. VA devices have become the primary interface for systems utilizing VAs developed by various technology companies.

[0012] One of the biggest challenges VA systems face is operating in noisy environments and competing with other ambient sounds. While much research has been conducted to identify correct speech, VA systems still cannot authenticate users if noise or some modulation is present in the speech. As VA applications are implemented in enterprise systems, enabling VA device response accuracy is becoming increasingly complex, which further complicates testing those systems, making testing VA systems a daunting task. Inadequate testing leads to VA systems being prone to many errors, making users suspicious when VA applications fail. As the user base of VA devices and systems expands, there is a clear need to enhance their user experience.

[0013] Embodiments of the present invention recognize that VA devices should be tested in a variety of environments, including different types and levels of noise, to improve VA device response accuracy. In addition, VA devices are tested to identify edge cases that indicate a specific modulation of the user's speech that is sufficient to prevent the VA device from understanding the user's speech. Once the edge case is identified, the test data used to modulate the user's speech that caused the edge case can be saved and associated with the conditions that made the user's speech unrecognizable. In addition, the test data associated with the edge case can be fine-tuned to modulate the user's speech, wherein the fine-tuning will be just enough to avoid the edge case in subsequent test iterations. By fine-tuning the test data, and in particular fine-tuning the encoding parameters (e.g., vocoder parameters), the VA device becomes better adapted to recognize background noise and sound modulation so as to ignore the background noise and sound modulation and thereby correctly recognize the command or question in the user's speech. In addition, the VA device should also be tested with different types of user speech that audibly output the same command or phrase to increase the VA device response accuracy. Implementation of embodiments of the present invention can take various forms, and exemplary implementation details are discussed subsequently with reference to the accompanying drawings.

[0014] Figure 1 is a functional block diagram illustrating a distributed data processing environment for testing a speech assistant device, generally designated 100, according to one embodiment of the present invention. The term "distributed" as used herein describes a computer system that includes multiple physically distinct devices that operate together as a single computer system. Figure 1This merely provides an illustration of one implementation and does not imply any limitation with respect to the environments in which different embodiments may be implemented. Many modifications to the described environments may be made by those skilled in the art without departing from the scope of the invention as set forth in the claims.

[0015] In the depicted embodiment, the distributed data processing environment 100 includes a speech assistant 120, a server 125, a database 122, a speech-to-text module 140, and a machine learning (ML) model 150 interconnected via a network 110. The distributed data processing environment 100 may include a database 122 configured to store data received from components within the distributed data processing environment 100 (e.g., the speech assistant 120, the vocoder 130, the speech-to-text (VTT) module 140, and / or the ML model 150) and to send data to components within the distributed data processing environment 100 for testing the speech assistant. The distributed data processing environment 100 may also include additional servers, computers, sensors, or other devices not shown. Each component (e.g., the speech assistant 120, the vocoder 130, the speech-to-text (VTT) module 140, and / or the ML model 150) may be configured to transmit data between each other independently of the network 110.

[0016] The network 110 operates as a computing network that can be, for example, a local area network (LAN), a wide area network (WAN), or a combination of both, and can include wired, wireless, or fiber optic connections. In general, the network 110 can be any combination of connections and protocols that will support communication between the voice assistant device 120, the server 125, the database 122, the VTT module 140, and the ML model 150.

[0017] The voice assistant device 120 operates as a voice assistant device for responding to user prompts. In embodiments, the voice assistant device 120 may be configured to receive voice data from the vocoder 130 or detect voice data from the vocoder 130, process the voice data, and output an acoustic signal including a voice assistant response corresponding to the voice data. The voice assistant device 120 may be configured to send data to and / or receive data from the network 110, or to send and / or receive data from other system components within the distributed data processing environment 100. In some embodiments, the voice assistant device 120 may be a laptop computer, a tablet computer, a netbook computer, a personal computer (PC), a desktop computer, a smartphone, a smart speaker, a virtual assistant, a voice command device, or any programmable electronic device capable of receiving or detecting audible input, processing audible input, and audibly outputting an associated response. The voice assistant device 120 may include, for example, a computer system ... Figure 3 Components described in further detail in .

[0018] The database 122 can be configured to operate as a repository for data flowing to and from the network 110 and other connected components. Examples of data include test data, device data, network data, data corresponding to user utterances processed by the vocoder, encoding parameters, vocoder parameters, data corresponding to modified user utterances, and data corresponding to voice assistance responses. A database is an organized collection of data. The database 122 can be implemented using any type of storage device capable of storing data and configuration files that can be accessed and used by the server 125 and / or voice assistance device 120, such as a database server, a hard drive, or flash memory. In an embodiment, the database 122 can be accessed by the vocoder 130, the voice assistance device 120, the server 125, the speech-to-text module 140, and / or the ML model 150 via the network 110 or independently of the network 110 to store and receive data corresponding to test scenarios executed on the voice assistance device 120. For example, test data can be received and stored in database 122 and sent to vocoder 130 or other components connected to network 110 as required by the test plan. In another embodiment, database 122 can be accessed by server 125 and / or ML model 150 to access user data, device data, network data, or other data associated with the test plan. Database 122 can also be accessed by VTT module 140 to store data corresponding to test result data processed and generated by speech assistant device 120 and VTT module 140. In another embodiment, database 122 can reside elsewhere in distributed network environment 100, as long as database 122 can access network 110.

[0019] The test data may include data compatible with the JavaScript Object Notation ("JSON") data interchange format and voice command data. The test data may also include data corresponding to a text-to-speech conversation between a user and another party. For example, the test data may include audible user utterances that are voice sounds spoken by the user. The user utterances may include commands to perform specific tasks or functions. The user utterances may also include questions posed by the user to solicit responses from a virtual assistant executed as part of the operation of the VA device 120. The test data may also include encoding parameters (e.g., vocoder parameters) and values ​​used to perform tests on the VA device 120. For example, the test data may be provided to the database 122 via an external source or received from one or more components in communication with the database 122. In particular, the voice commands may include a trigger word / phrase followed by "What's the weather outside?", "It's going to rain," and the expected device responses may be "Hello AB," "25 degrees and sunny," and "Yes, there's a chance of rain," respectively.

[0020] The server 125 may be a standalone computing device, a management server, a web server, or any other electronic device or computing system capable of receiving, sending, and processing data and communicating with the voice assistant device 120 via the network 110. In other embodiments, the server 125 represents a server computing system that utilizes multiple computers as a server system, such as a cloud computing environment. In other embodiments, the server 125 represents a computing system that utilizes clustered computers and components (e.g., database server computers, application server computers, etc.) that act as a single seamless resource pool when accessed within the distributed data processing environment 100. The server 125 may include, for example, Figure 3 Describe the components in more detail.

[0021] The vocoder 130 may be a speech codec or speech processor configured to analyze and synthesize an input human speech signal for use in audio data compression, multiplexing, speech encryption, or speech conversion. The vocoder 130 may include a plurality of filter bands tuned across the audio spectrum. The vocoder 130 may be configured to generate speech texture using vocoder parameters. The vocoder parameters are encoding parameters used to program the vocoder 130 to manipulate the received audio signal according to a number of units measured for the vocoder parameters. The vocoder 130 may be configured to receive test data, process the test data by applying the vocoder parameters to user utterances in the test data, and output a modified user utterance that may be acoustically different from the received user utterance in the test data.

[0022] The vocoder 130 may also be configured to generate multiple variations of speech or user utterances based on vocoder parameters and values ​​provided in the test data. The received user utterances may be modulated or modified by applying vocoder parameters, background noise, or accents to alter the characteristics of the user utterances. For example, Fliger distortion techniques or other known audio signal processing techniques may be used to modulate the user utterances in the test data so that the modulated or modified user utterances sound acoustically different from the received user utterances. Furthermore, the vocoder 130 may receive test data including user utterances corresponding to voice commands or voice questions to a virtual assistant executed via the voice assistance device 120, wherein the virtual assistant is configured to provide the expected device response.

[0023] In another embodiment, the vocoder 130 may be configured to add noise data to the vocoder parameters to be applied to the user utterance. The vocoder 130 may also be configured to add noise data directly to the user utterance so that when the modified user utterance is presented to the voice assistant device 120, the modified user utterance will include the noise data as part of the output audio signal. Thus, the encoding parameters may include noise data. For example, one or more noise features may be added to the user utterance to simulate a real-world environment that includes ambient noise in the background. The noise data may include sounds generated by background TV audio, background conversations between people, natural (e.g., animals, weather) sounds, traffic (e.g., public or private vehicles / traffic patterns), or any other sound that can be detected by a microphone present in the surrounding environment.

[0024] User utterances may include voice characteristics based on user characteristics that uniquely distinguish one user's voice from another user's voice. For example, voice characteristics may include pitch, speech rate, pitch, texture, intonation, loudness, etc., wherein a combination of one or more voice characteristics may produce a unique voice corresponding to an accent or dialect.

[0025] A voice command may be any command in a user's utterance that is one of a plurality of commands that the voice assistant device 120 is configured to process, wherein upon processing the voice command, the voice assistant device 120 may be configured to perform a function or speak an expected device response. The expected device response may include the voice assistant device 120 performing a function corresponding to the command or question in the user's utterance. For example, a voice command may include a command to "set an alarm for 6 AM," which, when processed and executed by the voice assistant device 120, generates an expected device response that audibly presents "set an alarm for 6 AM." In addition to generating the expected device response, the voice assistant device 120 may be configured to execute the voice command and perform the function included in the voice command.

[0026] However, due to various factors including unfamiliar accents / dialects or background noise, the voice assistant device 120 may not be adequately configured to understand the voice command, resulting in edge cases. According to the present invention, an edge case occurs when the voice assistant device 120 receives and processes test data including modified user utterances and generates a voice assistant response that does not match the corresponding expected device response. In addition, an edge case may occur when the voice assistant device 120 receives and processes test data including modified user utterances and fails to generate a voice assistant response or generates an error message indicating that the voice assistant device 120 failed to process the modified user utterance.

[0027] Thus, in an embodiment, the vocoder 130 may be configured to receive test data including vocoder parameters that, when applied to a user utterance in the test data, generates a modified user utterance and audibly presents it to the voice assistant device 120. After processing and executing the modified user utterance, the voice assistant device 120 may be configured to audibly present a device response that may substantially match or substantially mismatch an expected device response corresponding to the user utterance. If the voice assistant device 120 presents a device response that substantially mismatches an expected device response, an edge case may be identified based on the test data used to generate the modified user utterance.

[0028] The speech-to-text module 140 may be a component configured to perform a speech recognition process, wherein speech data is received and processed to output text data corresponding to the received speech data. For example, the speech-to-text module 140 may be configured to receive an audio signal including speech data from the speech assistant device 120, process the audio signal, and output text data corresponding to the received audio signal. The output text data may be included in the test result data sent to the database 122. In addition, the speech-to-text module 140 may be configured to capture the speech audio signal and convert the speech audio signal into text data. The speech-to-text module 140 may be further configured to compare the text data corresponding to the actual speech assistant response with the test data corresponding to the expected device response to determine whether there is a match. The test result data may be captured and stored in the database 122. The test result data may include data corresponding to the speech assistant response captured from the speech assistant device 120. The test result data may also include vocoder parameters and the number of units of each vocoder parameter used to modulate the user utterance in the test data.

[0029] The number of units of a vocoder parameter corresponds to a quantifiable unit of measurement for each parameter. For example, vocoder parameters may include the number of frequency bands, frequency range, envelope, unvoiced element, formant shift, bandwidth, and the like. Each parameter can be quantified by the number of units applied to the corresponding parameter. For example, the number of frequency bands determines the total number of filter bands applied to the user's speech as a voice signal. A smaller number (e.g., in the range of 4 to 6) delivers a vintage sound, while any number above 16 bands provides more detail, typically at the expense of processor utilization. The frequency range vocoder parameter can be set to have an upper and lower limit, which can be divided by the number of available frequency bands. The frequency range vocoder parameter can be used to tune the filter bands in the vocoder 130 to accommodate the higher or lower vocal ranges of female and male voices, respectively. The envelope vocoder parameter determines how quickly the effect responds to dynamic volume changes. When applied to voices or user speech, the envelope vocoder parameter provides a relatively fast response time, but a longer response time is implemented for more impressive effects.

[0030] Additional vocoder parameters may be associated with frequency filters (e.g., high-pass, low-pass) that may help improve the intelligibility of certain speech elements. For example, when plosives (e.g., the letters t, d, b) and sibilants (e.g., s, z, x) are processed by the vocoder 130, a high-pass filter vocoder parameter may improve intelligibility by allowing these unvoiced elements to be better detected and identified.

[0031] The machine learning ("ML") model 150 may include one or more processors configured to receive test result data, process the received test result data, and generate output data corresponding to the received test result data. For example, the ML model 150 may be configured to receive test result data from the database 122, process the test result data to generate modified vocoder parameters, and update the test data, i.e., the received test data, based on edge cases. In other words, the ML model 150 may be configured to fine-tune the vocoder parameters based on the test result data to generate new test data to further test the voice assistant device 120 to identify more edge cases. If more edge cases are identified, the ML model 150 may be better configured to generate fine-tuned vocoder parameters to modulate the user utterance, thereby generating a modified user utterance that will result in the desired device response from the voice assistant device 120.

[0032] In addition, in another embodiment, the ML model 150 can be configured to apply chaos engineering principles to test resilience, wherein the ML model 150 can compare test results to identify edge cases and fine-tune the vocoder parameters to identify more edge cases within preset control limits. For example, if the actual voice assistant response does not match the expected device response, the ML model 150 can determine that the failure to match constitutes an edge case and identify the modified test data as an edge case. Once an edge case is identified, the ML model 150 can be configured to modify the first vocoder parameters used in the edge case to generate second vocoder parameters, which will be applied by the vocoder 130 to the user utterance to generate another modified user utterance for additional testing on the voice assistant device 120. The ML model 150 can be configured to continue the vocoder parameter fine-tuning process until enough edge cases have been identified that meet the control limits of the testing process. Once an edge case has been identified, the ML model 150 can be configured to correct or fine-tune the vocoder parameters corresponding to the identified edge case. The corrected or fine-tuned vocoder parameters can then be applied to the user utterance to produce or generate a modified user utterance for presentation to the VA device 120 for further testing. The VA device 120 testing can continue for a number of iterations sufficient to satisfy the conditions associated with a successful test scenario. For example, the VA device 120 testing can continue until the device response substantially matches the expected device response corresponding to the user utterance processed by the vocoder 120 and is provided to the VA device 120 as the modified user utterance.

[0033] Once enough test iterations are completed, the condition may be satisfied. For example, the condition may correspond to a determination that the number of edge cases exceeds or does not reach a predetermined threshold. For example, after 3 iterations of fine-tuning the vocoder parameters for edge case identification, the one or more processors may be configured to stop or cease sending test data to the vocoder 130, thereby signaling the end of a phase of the test scenario. The number of test iterations may be based on the VA device processing time, which is the amount of time it takes the VA device 120 to process the user utterance and the modified user utterance. The number of test iterations may also be based on the number of edge cases identified, wherein a predetermined number of edge cases identified may satisfy the condition to stop sending test data to the vocoder 130. Furthermore, the tester may manually stop the process of providing test data to the VA device 120.

[0034] In at least some embodiments, ML model 150 may implement a trained component or trained model configured to perform the process described above with respect to ML model 150. The trained component may include one or more machine learning models, including but not limited to one or more classifiers, one or more neural networks, one or more probability graphs, one or more decision trees, and the like. In other embodiments, the trained component may include a rule-based engine, one or more statistical-based algorithms, one or more mapping functions, or other types of functions / algorithms to determine whether natural language input is complex or non-complex. In some embodiments, the trained component may be configured to perform binary classification, where natural language input can be classified into one of two categories. In some embodiments, the trained component may be configured to perform multi-class or multinomial classification, where natural language input can be classified into one of three or more categories. In some embodiments, the trained component may be configured to perform multi-label classification, where natural language input can be associated with more than one class / category. As described herein, one or more processors may include ML model 150.

[0035] Various machine learning techniques can be used to train and operate trained components to perform the various processes described herein. Models can be trained and operated according to various machine learning techniques. Such techniques can include, for example, neural networks (such as deep neural networks and / or recurrent neural networks), inference engines, trained classifiers, etc. Examples of trained classifiers include support vector machines (SVMs), neural networks, decision trees, AdaBoost (abbreviated as "adaptive boosting") combined with decision trees, and random forests. Taking SVM as an example, SVM is a supervised learning model that has a related learning algorithm that analyzes data and identifies patterns in the data, and is commonly used for classification and regression analysis. Given a training example set, each training example is marked as belonging to one of two categories, and the SVM training algorithm establishes a model that assigns new examples to one category or another, making it a non-probabilistic binary linear classifier. More complex SVM models can be constructed using training sets that identify more than two categories, where the SVM determines which category is most similar to the input data. The SVM model can be mapped so that the examples of the individual categories are clearly divided by gaps. New examples are then mapped into this same space and predicted to belong to a class based on which side of the gap they fall in. The classifier can emit a "score" that indicates which class the data most closely matches. This score can provide an indication of how closely the data matches the class.

[0036] In order to apply machine learning techniques, the machine learning process itself needs to be trained. Training machine learning components requires establishing "ground truth" for the training examples. In machine learning, the term "ground truth" refers to the accuracy of the classification of the training set for supervised learning techniques. Models can be trained using a variety of techniques, including backpropagation, statistical learning, supervised learning, semi-supervised learning, stochastic learning, or other known techniques.

[0037] refer to Figure 2 A computer-implemented method for testing the speech assistant device 120 is depicted and described in further detail.

[0038] Figure 2 is a flowchart illustrating an embodiment of the present invention for Figure 1 Flowchart of the operational steps of a computer-implemented method 200 for testing a speech assistant device in a distributed data processing environment.

[0039] In an embodiment, a computer-implemented method 200 for testing a speech assistant device may be performed by one or more processors configured to receive 202 test data from a database, such as database 122, wherein the test data may include at least a first set of encoding parameters and a first user utterance with an expected device response. For example, the one or more processors may include a vocoder 130 configured to receive 202 test data from the database 122 via a communication link between the database 122 and the vocoder 130. The vocoder 130 may include a data port configured to receive data transmissions and relay the data to a processor resident in the vocoder 130 for further processing. The first set of encoding parameters may be used to configure the vocoder 130 settings, which may be used to modulate or modify the first user utterance.

[0040] The computer-implemented method 200 for testing a speech assistant device may include one or more processors further configured to generate 204 a first modified user utterance by applying a first set of encoding parameters to a first user utterance, wherein the first modified user utterance is acoustically different from the first user utterance. For example, the one or more processors may include a vocoder 130 configured to generate 204 the first modified user utterance by applying a first set of encoding parameters to the first user utterance, as explained above. At least due to the vocoder 130 applying the first set of encoding parameters to the first user utterance, the first modified user utterance is acoustically different from the first user utterance. Therefore, due to the vocoder 130 modifying one or more acoustic features, the first modified user utterance may sound different.

[0041] The computer-implemented method 200 for testing a speech assistant device may include one or more processors further configured to audibly present 206 the first modified user utterance to the speech assistant device 120. For example, the vocoder 130 may include a speaker configured to output an audio signal including the first modified user utterance for detection by the speech assistant device 120. The speech assistant device 120 may include a microphone configured to detect the audio signal near the speaker of the vocoder 130 and receive the audio signal as sound waves.

[0042] The computer-implemented method 200 for testing a voice assistant device may include one or more processors further configured to receive 208 a first voice assistant response from the voice assistant device 120. The one or more processors may be configured to receive the first voice assistant response via the voice-to-text module 140, wherein the VTT module 140 may include a microphone to detect and receive the first voice assistant response as an audio input signal, convert the audio input signal into text data, and send the text data to the one or more processors for further processing. For example, the microphone may be located within range of a speaker of the voice assistant device 120 so that the microphone can detect an output audio signal from the speaker of the voice assistant device 120. The microphone may be configured to transmit the detected audio signal as data corresponding to the voice assistant response to the one or more processors.

[0043] The computer-implemented method 200 for testing a voice assistant device may include one or more processors further configured to determine 210 whether a first voice assistant response is substantially similar to an expected device response. In an embodiment, the one or more processors may include a ML model 150 for determining 210 whether the first voice assistant response is substantially similar to an expected device response. Additionally, the one or more processors may be configured to determine a first score based on a comparison between the first voice assistant response and the expected device response. For example, if the first voice assistant response substantially matches the expected device response, the first score may be 1. In the alternative, if the first voice assistant response does not substantially match the expected device response, the first score may be 0. If the first score is 0, the one or more processors may determine that an edge case including the first test data and the voice assistant response has been identified.

[0044] In another embodiment, in response to determining that the first voice assistant response is substantially similar to the expected device response, the computer-implemented method 200 for testing a voice assistant device may include one or more processors further configured to stop receiving additional test data. Additionally, in response to determining that the first voice assistant response is not substantially similar to the expected device response, the one or more processors may be configured to identify a first edge case based on at least the first encoding parameter set and the first modified user utterance. The one or more processors may be further configured to determine a second encoding parameter set based on at least one of the first encoding parameter set, the first modified user utterance, and the first edge case.

[0045] Furthermore, the computer-implemented method 200 for testing a voice assistant device may include one or more processors further configured to generate a second modified user utterance by applying a second set of encoding parameters to a first user utterance, wherein the second modified user utterance may be acoustically distinct from the first user utterance and the first modified user utterance. The one or more processors may further be configured to audibly present the second modified user utterance to the voice assistant device 120, receive a second voice assistant response from the voice assistant device 120, and determine whether the second voice assistant response is substantially similar to the expected device response. Furthermore, in response to determining that the second voice assistant response is substantially similar to the expected device response, the one or more processors may be configured to stop receiving additional test data from the database 122. For example, once the one or more processors determine that the voice assistant response is substantially similar to the expected device response corresponding to the first user utterance, the one or more processors may then be configured to stop receiving test data including the first user utterance. However, the one or more processors may be configured to continue receiving test data including additional user utterances that are distinct from the first user utterance to continue the testing scenario.

[0046] In an embodiment, the first modified user utterance may be audibly presented to the speech assistance device 120 by a speaker communicatively coupled to the one or more processors (e.g., the vocoder 130), wherein the speaker is configured to output the modified user utterance as an audio signal.

[0047] In an embodiment, the first voice assistance response may be received as an audio signal from the voice assistance device 120 by a microphone communicatively coupled to one or more processors (eg, the speech-to-text module 140 ).

[0048] Figure 3 The embodiment of the present invention is shown Figure 1 It should be understood that Figure 3This merely provides an illustration of one embodiment and does not imply any limitation with respect to the environments in which different embodiments may be implemented. Many modifications to the depicted environments are possible.

[0049] Figure 3 A block diagram of a computer 300 suitable for use with the speech assistant device 120 is depicted in accordance with an illustrative embodiment of the present invention. It should be understood that Figure 3 This merely provides an illustration of one embodiment and does not imply any limitation with respect to the environments in which different embodiments may be implemented. Many modifications to the depicted environments are possible.

[0050] The computer 300 includes a communications fabric 302 that provides communications between a cache 316, memory 306, persistent storage 308, a communications unit 310, and one or more input / output (I / O) interfaces 312. The communications fabric 302 can be implemented using any architecture designed to transfer data and / or control information between processors (such as microprocessors, communications and network processors, etc.), system memory, peripheral devices, and any other hardware components within the system. For example, the communications fabric 302 can be implemented using one or more buses or crossbar switches.

[0051] Memory 306 and persistent storage 308 are computer-readable storage media. In this embodiment, memory 306 includes random access memory (RAM). In general, memory 306 may include any suitable volatile or non-volatile computer-readable storage media. Cache 316 is a fast memory that enhances the performance of one or more computer processors 304 by storing recently accessed data and data near the accessed data from memory 306.

[0052] Software and data 314 may be stored in persistent storage 308 and memory 306 for execution and / or access by one or more corresponding computer processors 304 via cache 316. In one embodiment, persistent storage 308 comprises a magnetic hard drive. As an alternative to or in addition to a magnetic hard drive, persistent storage 308 may comprise a solid-state hard drive, a semiconductor memory device, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), flash memory, or any other computer-readable storage medium capable of storing program instructions or digital information.

[0053] The media used by persistent storage 308 may also be removable. For example, a removable hard drive may be used for persistent storage 308. Other examples include optical and magnetic disks, thumb drives, and smart cards that are inserted into a drive for transfer to another computer-readable storage medium that is also part of persistent storage 308.

[0054] In these examples, communications unit 310 provides for communications with other data processing systems or devices. In these examples, communications unit 310 includes one or more network interface cards. Communications unit 310 can provide for communications using one or both of physical and wireless communication links. Software and data 314 can be downloaded to persistent storage 308 via communications unit 310.

[0055] The I / O interface(s) 312 allow for input and output of data with other devices that may be connected to the voice assistant device 120. For example, the I / O interface(s) 312 may provide a connection to an external device 318 such as a keyboard, a keypad, a touch screen, and / or some other suitable input device. The external device 318 may also include a portable computer-readable storage medium such as a thumb drive, a portable optical or magnetic disk, and a memory card. Software and data 314 for implementing embodiments of the present invention may be stored on such a portable computer-readable storage medium and may be loaded onto the persistent storage 308 via the I / O interface(s) 312. The I / O interface(s) 312 are also connected to a display 320.

[0056] Display 320 provides a mechanism for displaying data to a user and may be, for example, a computer monitor.

[0057] The present invention may include various accessible data sources, such as database 122, which may contain personal data, content, or information that users wish not to be processed. Personal data includes personally identifiable information or sensitive personal information, as well as user information such as tracking or geolocation information. Processing refers to any automated or non-automated operation or set of operations performed on personal data, such as collecting; recording; organizing; structuring; storing; adapting; altering; retrieving; consulting; using; making it available by transmission, distribution, or other means; combining; restricting; erasing; or destroying. Software and data 314 can implement the authorized and secure processing of personal data. Software and data 314 can be configured to provide informed consent, notify the collection of personal data, and allow users to opt in or opt out of processing personal data. This can take several forms. Opt-in consent can force users to take affirmative action before personal data is processed. Alternatively, opt-out consent can force users to take affirmative action to prevent personal data from being processed before it is processed. Software and data 314 can provide information about personal data and the nature of the processing (e.g., type, scope, purpose, duration, etc.). Software and data 314 provides users with a copy of the stored personal data. Software and data 314 allow for the correction or completion of incorrect or incomplete personal data. Software and data 314 allow for the immediate deletion of personal data.

[0058] The computer-implemented methods described herein are identified based on the applications in which they are implemented in specific embodiments of the invention. However, it should be understood that any specific program nomenclature herein is used for convenience only, and thus the present invention should not be limited to use solely in any specific application identified and / or implied by such nomenclature.

[0059] The present invention may be a system, a computer-implemented method, and / or a computer program product. The computer program product may include a computer-readable storage medium (or medium) having computer-readable program instructions thereon, the computer-readable program instructions being used to cause a processor to perform various aspects of the present invention.

[0060] Computer readable storage medium can be any tangible device that can hold and store the instructions used by the instruction execution device.Computer readable storage medium can be, for example, but not limited to, electronic storage device, magnetic storage device, optical storage device, electromagnetic storage device, semiconductor storage device or any suitable combination of the foregoing.A non-exhaustive list of more specific examples of computer readable storage medium includes the following: portable computer disk, hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanical encoding device such as the punch card or the raised structure in the groove with instructions recorded above, and any suitable combination of the foregoing.Computer readable storage medium as used herein should not be interpreted as a temporary signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagated by waveguide or other transmission media (for example, light pulses by optical fiber cable), or electrical signals transmitted by wire.

[0061] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to a corresponding computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions to be stored in a computer-readable storage medium within the corresponding computing / processing device.

[0062] The computer-readable program instructions for performing the operation of the present invention can be assembly instructions, instruction set architecture (ISA) instructions, machine-related instructions, microcode, firmware instructions, state setting data or source code or object code written in any combination of one or more programming languages, programming languages ​​include object-oriented programming languages ​​(such as Smalltalk, C++ etc.) and conventional procedural programming languages ​​(such as " C " programming language or similar programming languages). The computer-readable program instructions can be performed completely on the user's computer, partly on the user's computer, performed as an independent software package, partly on the user's computer and partly on a remote computer, or performed completely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer through any type of network, and the network includes a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (such as, using an Internet service provider through the Internet). In certain embodiments, in order to perform various aspects of the present invention, the electronic circuit comprising for example programmable logic circuit, field programmable gate array (FPGA) or programmable logic array (PLA) can perform computer-readable program instructions to personalize the electronic circuit by utilizing the state information of the computer-readable program instructions.

[0063] Aspects of the present invention are described herein with reference to the flowcharts and / or block diagrams of the methods, apparatus (systems) and computer program products according to embodiments of the present invention. It will be understood that each block of the flowcharts and / or block diagrams and the combination of blocks in the flowcharts and / or block diagrams can be implemented by computer-readable program instructions.

[0064] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device create a means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium, which can direct the computer, programmable data processing device, and / or other device to operate in a specific manner, so that the computer-readable storage medium having the instructions stored therein includes an article of manufacture, which includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0065] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable apparatus, or other device to produce a computer-implemented process, so that the instructions executed on the computer, other programmable apparatus, or other device implement the functions / actions specified in one or more boxes of the flowchart and / or block diagram.

[0066] The flow charts and block diagrams in the accompanying drawings illustrate the architecture, functions and operations of possible implementations of the system, method and computer program product according to various embodiments of the present invention. In this regard, each frame in the flow chart or block diagram can represent a module, segment or portion of an instruction, which includes one or more executable instructions for realizing one or more specified logical functions. In some alternative embodiments, the functions noted in the frame may not occur in the order noted in the figure. For example, the two frames shown in succession can actually be performed substantially simultaneously, or these frames can sometimes be performed in reverse order, depending on the functions involved. It will also be noted that each frame of the block diagram and / or flow chart and the combination of the frames in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs a specified function or action or performs a combination of dedicated hardware and computer instructions.

[0067] The description of various embodiments of the present invention has been provided for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope of the invention. The terminology used herein is selected to best explain the principles of the embodiments, practical applications, or technical improvements over existing technologies on the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A computer-implemented method for testing a speech-assisted device, the computer-implemented method comprising: receiving, by one or more processors, test data from a database, the test data comprising a first set of encoding parameters and a first user utterance having an expected device response; generating, by the one or more processors, a first modified user utterance by applying the first set of encoding parameters to the first user utterance, wherein the first modified user utterance is acoustically distinct from the first user utterance; audibly presenting, by the one or more processors, the first modified user utterance to a speech assistance device; receiving, by the one or more processors, a first voice assistance response from the voice assistance device; determining, by the one or more processors, whether the first voice assistance response is substantially similar to the expected device response; in response to determining that the first voice assistance response is not substantially similar to the expected device response, identifying, by the one or more processors, a first edge case based on at least the first set of encoding parameters and the first modified user utterance; determining, by the one or more processors, a second set of encoding parameters based on at least one of the first set of encoding parameters, the first modified user utterance, and the first edge case; and The received test data is updated based on the determined second encoding parameter set to further test the voice assistant device.

2. The computer-implemented method of claim 1 , further comprising: Responsive to determining that the first voice assistance response is substantially similar to the expected device response, receiving additional test data by the one or more processors is ceased.

3. The computer-implemented method of claim 1 , further comprising: generating, by the one or more processors, a second modified user utterance by applying the second set of encoding parameters to the first user utterance, wherein the second modified user utterance is acoustically distinct from the first user utterance and the first modified user utterance; audibly presenting, by the one or more processors, the second modified user utterance to the speech assistance device; receiving, by the one or more processors, a second voice assistance response; determining, by the one or more processors, whether the second voice assistance response is substantially similar to the expected device response; and In response to determining that the second voice assistance response is substantially similar to the expected device response, receiving additional test data including the first user utterance is stopped.

4. The computer-implemented method of claim 1 , wherein: The first modified user utterance is audibly presented to the speech assistance device by a speaker communicatively connected to the one or more processors, the speaker configured to output the first modified user utterance as a first audio signal.

5. The computer-implemented method of claim 1 , wherein: The first voice assistance response received from the voice assistance device is detected as a second audio signal by a microphone communicatively coupled to the one or more processors.

6. A computer program product for testing a speech-assistance device, the computer program product comprising: One or more computer-readable storage media and program instructions stored on the one or more computer-readable storage media, the stored program instructions comprising: program instructions for receiving test data from a database, the test data comprising a first set of encoding parameters and a first user utterance having an expected device response; program instructions for generating a first modified user utterance by applying the first set of encoding parameters to the first user utterance, wherein the first modified user utterance is acoustically distinct from the first user utterance; program instructions for audibly presenting the first modified user utterance to a speech assistance device; program instructions for receiving a first voice assistance response from the voice assistance device; program instructions for determining whether the first voice assistant response is substantially similar to the expected device response; program instructions for identifying a first edge case based at least on the first set of encoding parameters and the first modified user utterance in response to determining that the first voice assistance response is not substantially similar to the expected device response; program instructions for determining a second set of encoding parameters based on at least one of the first set of encoding parameters, the first modified user utterance, and the first edge case; and Program instructions for updating the received test data based on the determined second set of encoding parameters to further test the speech assistant device.

7. The computer program product of claim 6, further comprising: In response to determining that the first voice assistance response is substantially similar to the expected device response, program instructions are provided for ceasing to receive additional test data.

8. The computer program product of claim 6, further comprising: program instructions for generating a second modified user utterance by applying the second set of encoding parameters to the first user utterance, wherein the second modified user utterance is acoustically distinct from the first user utterance and the first modified user utterance; program instructions for audibly presenting the second modified user utterance to the speech assistance device; program instructions for receiving a second voice assistance response; program instructions for determining whether the second voice assistant response is substantially similar to the expected device response; and In response to determining that the second voice assistance response is substantially similar to the expected device response, program instructions for ceasing to receive additional test data comprising the first user utterance.

9. The computer program product of claim 6, wherein: The first modified user utterance is audibly presented to the speech assistance device by a speaker communicatively connected to the one or more processors, the speaker configured to output the first modified user utterance as a first audio signal.

10. The computer program product of claim 6, wherein: The first voice assistance response received from the voice assistance device is detected as a second audio signal by a microphone communicatively coupled to the one or more processors.

11. A computer system for testing a speech-assisted device, the computer system comprising: one or more computer processors; one or more computer-readable storage media; Program instructions collectively stored on the one or more computer-readable storage media for execution by at least one of the one or more computer processors, the stored program instructions comprising: program instructions for receiving test data from a database, the test data comprising a first set of encoding parameters and a first user utterance having an expected device response; program instructions for generating a first modified user utterance by applying the first set of encoding parameters to the first user utterance, wherein the first modified user utterance is acoustically distinct from the first user utterance; program instructions for audibly presenting the first modified user utterance to a speech assistance device; program instructions for receiving a first voice assistance response from the voice assistance device; program instructions for determining whether the first voice assistant response is substantially similar to the expected device response; program instructions for identifying a first edge case based at least on the first set of encoding parameters and the first modified user utterance in response to determining that the first voice assistance response is not substantially similar to the expected device response; program instructions for determining a second set of encoding parameters based on at least one of the first set of encoding parameters, the first modified user utterance, and the first edge case; Program instructions for updating the received test data based on the determined second set of encoding parameters to further test the speech assistant device.

12. The computer system of claim 11, further comprising: In response to determining that the first voice assistance response is substantially similar to the expected device response, program instructions are provided for ceasing to receive additional test data.

13. The computer system of claim 11, further comprising: program instructions for generating a second modified user utterance by applying the second set of encoding parameters to the first user utterance, wherein the second modified user utterance is acoustically distinct from the first user utterance and the first modified user utterance; program instructions for audibly presenting the second modified user utterance to the speech assistance device; program instructions for receiving a second voice assistance response; program instructions for determining whether the second voice assistant response is substantially similar to the expected device response; and In response to determining that the second voice assistance response is substantially similar to the expected device response, program instructions for ceasing to receive additional test data comprising the first user utterance.

14. The computer system according to claim 11, wherein: The first modified user utterance is audibly presented to the voice assistance device by a speaker communicatively connected to the one or more processors, the speaker being configured to output the first modified user utterance as a first audio signal, and wherein the first voice assistance response received from the voice assistance device is detected as a second audio signal by a microphone communicatively connected to the one or more processors.

Images