Chaos Testing Voice-Enabled Devices

Abstract

A computer-implemented method, computer program product, and computer system for testing a voice assistant device may include one or more processors configured to receive test data from a database, the test data including a first set of encoding parameters and a first user utterance having an expected device response. The one or more processors may further be configured to generate a first modified user utterance by applying the first set of encoding parameters to the first user utterance, the first modified user utterance being acoustically different from the first user utterance. The one or more processors may be configured to audibly present the first modified user utterance to the voice assistant device, receive a first device response from the voice assistant device, and determine whether the first voice assistant response is substantially similar to the expected device response.

Description

[Technical Field]

[0001] The present invention relates generally to the field of device testing, and more particularly to testing voice-enabled devices using machine learning techniques. [Background technology]

[0002] Conversational voice assistants can detect voice commands, process them, and determine the user's intent to perform a specific task. Voice assistant devices are evolving to interact with users and engage in ongoing conversations with them. Voice assistants have become the primary interface for systems for virtual assistants.

[0003] However, voice assistant devices can be hindered in noisy environments, making it more difficult to determine the exact meaning of a voice command or user utterance. Furthermore, because voice assistants are not tested with varying voice types and modulation levels, it is difficult for the voice assistant device to distinguish naturally modulated words or phrases from standard words or phrases used during testing. Summary of the Invention

[0004] Embodiments of the present invention disclose a computer-implemented method, computer program product, and system for testing a voice assistant device. The computer-implemented method for testing a voice assistant device may include one or more processors configured to receive test data from a database, the test data including a first set of encoding parameters and a first user utterance having an expected device response. The one or more processors may also be configured to generate a first modified user utterance by applying the first set of encoding parameters to the first user utterance, the first modified user utterance being acoustically different from the first user utterance.

[0005] The one or more processors may be configured to audibly present the first modified user utterance to the voice assistant device and receive a first voice assistant response from the voice assistant device, the first voice assistant response being responsive to the first modified user utterance. The first modified user utterance may be audibly presented to the voice assistant device by outputting it as a first audio signal via a speaker communicatively connected to the one or more processors, and the first voice assistant response received from the voice assistant device may be detected as a second audio signal by a microphone communicatively connected to the one or more processors. Further, the one or more processors may be configured to determine whether the first voice assistant response is substantially similar to an expected device response.

[0006] In response to determining that the first voice assistant response is substantially similar to the expected device response, the one or more processors may be configured to stop receiving additional test data. Further, in response to determining that the first voice assistant response is not substantially similar to the expected device response, the one or more processors may be configured to identify a first edge case based on at least the first set of encoding parameters and the first modified user utterance.

[0007] Further, a computer-implemented method for testing a voice assistant device may include one or more processors configured to determine a second set of encoding parameters based on at least one of the first set of encoding parameters, the first modified user utterance, and the first edge case. The one or more processors may be further configured to generate a second modified user utterance by applying the second set of encoding parameters to the first user utterance, the second modified user utterance being acoustically different from the first user utterance and the first modified user utterance. The one or more processors may be further configured to audibly present the second modified user utterance to the voice assistant device, receive a second voice assistant response, and determine whether the second voice assistant response is substantially similar to an expected device response. Furthermore, in response to determining that the second voice assistant response is substantially similar to the expected device response, the one or more processors may be configured to stop receiving additional test data including the first user utterance. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a functional block diagram illustrating a distributed data processing environment for testing voice assistant devices, according to one embodiment of the present invention. [Figure 2] 2 is a flowchart illustrating the operational steps of a computer-implemented method for testing a voice assistant device on a server computer in the distributed data processing environment of FIG. 1 according to one embodiment of the present invention. [Figure 3] FIG. 2 is a block diagram of components of a server computer that executes a computer-implemented method for testing voice assistant devices within the distributed data processing environment of FIG. 1 according to one embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0009] Consumer experiences are evolving to new levels every day. At the forefront of this are conversational voice assistants. There are currently a variety of artificial intelligence (AI) and voice assistant (VA) systems on the market. Like other products, VA technology is constantly improving as breakthroughs in voice recognition and software support are achieved. VA devices are able to interact with users and are evolving to simulate human responses during ongoing conversations. VA devices have become the primary interface for VA-enabled systems developed by various technology companies.

[0010] One of the biggest challenges facing speech recognition systems is operating in noisy environments and competing with other environmental sounds. Although much research has been done to identify the correct voice, VA systems cannot authenticate users if the voice is noisy or modulated. Furthermore, as VA applications are increasingly implemented in enterprise systems, the response accuracy of VA devices becomes increasingly complex, which in turn complicates VA system testing, making VA system testing a major undertaking. Insufficient testing can lead to error-prone VA systems, leading to poor user experience when VA applications fail. As the user base expands, there is a clear need to improve the user experience of VA devices and systems.

[0011] Embodiments of the present invention recognize that to improve the response accuracy of a VA device, the VA device must be tested in multiple environments containing various types and levels of noise. Furthermore, the VA device is tested to identify edge cases that exhibit specific modulations of user utterances sufficient to prevent the VA device from understanding the user utterance. Once an edge case is identified, the test data used to modulate the user utterance that resulted in the edge case can be saved and associated with the conditions that made the user utterance unidentifiable. Additionally, the test data associated with the edge case can be fine-tuned to modulate the user utterance, such that the fine-tuning will be sufficient to avoid the edge case in subsequent test iterations. By fine-tuning the test data, particularly the encoding parameters (e.g., vocoder parameters), the VA device is tuned to distinguish between background noise and voice modulations in order to properly identify the user's spoken command or question while ignoring the background noise and voice modulations. Furthermore, the VA device must also be tested with various types of user utterances that audibly output the same command or phrase to improve the response accuracy of the VA device. Implementations of embodiments of the present invention can take various forms, and details of exemplary implementations are described below with reference to the figures.

[0012] FIG. 1 is a functional block diagram illustrating a distributed data processing environment for testing a voice assistant device, generally designated 100, according to one embodiment of the present invention. As used herein, the term "distributed" describes a computer system that includes multiple physically distinct devices that operate together as a single computer system. FIG. 1 provides only an illustration of one implementation and does not imply any limitations with respect to the environments in which different embodiments may be implemented. Many modifications to the depicted environment may be made by one skilled in the art without departing from the scope of the invention, as set forth in the claims.

[0013] In the depicted embodiment, distributed data processing environment 100 includes voice assistant device 120, server 125, database 122, speech-to-text module 140, and machine learning (ML) model 150 interconnected via network 110. Distributed data processing environment 100 may include database 122 configured to store and transmit data received from components within distributed data processing environment 100 (e.g., voice assistant device 120, vocoder 130, speech-to-text (VTT) module 140, or ML model 150, or a combination thereof) for testing the voice assistant device. Distributed data processing environment 100 may also include additional servers, computers, sensors, or other devices not shown. Each component (e.g., voice assistant device 120, vocoder 130, speech-to-text (VTT) module 140, or ML model 150, or a combination thereof) may be configured to communicate data between each other independently of network 110.

[0014] Network 110 operates as a computing network that can be, for example, a local area network (LAN), a wide area network (WAN), or a combination of the two, and can include wired, wireless, or fiber optic connections. In general, network 110 can be any combination of connections and protocols that support communication between voice assistant device 120, server 125, database 122, VTT module 140, and ML model 150.

[0015] Voice assistant device 120 operates to operate as a voice assistant device for responding to user prompts. In one embodiment, voice assistant device 120 may be configured to receive or detect voice data from vocoder 130, process the voice data, and output an acoustic signal including a voice assistant response corresponding to the voice data. Voice assistant device 120 may be configured to transmit and / or receive data from network 110 or through other system components in distributed data processing environment 100. In some embodiments, voice assistant device 120 may be a laptop computer, tablet computer, netbook computer, personal computer (PC), desktop computer, smartphone, smart speaker, virtual assistant, voice command device, or any programmable electronic device capable of receiving or detecting audible input, processing the audible input, and audibly outputting an associated response. Voice assistant device 120 may include components as described in further detail in FIG. 3.

[0016] Database 122 may be configured to act as a repository for data flowing between network 110 and other connected components. Examples of data include test data, device data, network data, data corresponding to user utterances processed by a vocoder, encoding parameters, vocoder parameters, data corresponding to modified user utterances, and data corresponding to voice assistant responses. A database is an organized collection of data. Database 122 may be implemented on any type of storage device capable of storing data and configuration files that can be accessed and utilized by server 125, voice assistant device 120, or both, such as a database server, hard disk drive, or flash memory. In one embodiment, database 122 may be accessed by vocoder 130, voice assistant device 120, server 125, speech-to-text module 140, or ML model 150, or a combination thereof, over network 110 or independently of network 110, to store and receive data corresponding to test plans executed on voice assistant device 120. For example, test data may be received and stored in database 122 and transmitted to vocoder 130 or other components connected to network 110 as desired by the test plan. In another embodiment, database 122 may be accessed by server 125 or ML model 150, or both, to access user data, device data, network data, or other data related to the test plan. Database 122 may also be accessed by VTT module 140 to store data corresponding to test result data processed and generated by voice assistant device 120 and VTT module 140. In another embodiment, database 122 may reside elsewhere in distributed network environment 100, provided that database 122 has access to network 110.

[0017] The test data may include data corresponding to the JavaScript Object Notation (JSON) data interchange format and voice command data. The test data may also include data corresponding to a text-to-speech conversation between a user and another person. For example, the test data may include audible user utterances as spoken by a user. The user utterances may include commands to perform specific tasks or functions. The user utterances may also include questions posed by a user to solicit responses from a virtual assistant running as part of the operation of VA device 120. The test data may also include encoding parameters (e.g., vocoder parameters) and values ​​for running tests on VA device 120. For example, the test data may be provided to database 122 via an external source or received from one or more components in communication with database 122. In particular, the voice commands may include trigger words / phrases such as "What's the weather outside?" and "Is it going to rain?", and the expected device responses may be "Hello AB," "It's 25 degrees and sunny," and "Yes, it's going to rain," respectively.

[0018] Server 125 may be a standalone computing device, an administration server, a web server, or any other electronic device or computing system capable of receiving, transmitting, and processing data and communicating with voice assistant device 120 via network 110. In other embodiments, server 125 represents a server computing system that utilizes multiple computers as a server system, such as a cloud computing environment. In yet other embodiments, server 125 represents a computing system that utilizes clustered computers and components (e.g., database server computers, application server computers, etc.) that function as a single pool of seamless resources when accessed within distributed data processing environment 100. Server 125 may include components such as those described in further detail in FIG. 3.

[0019] The vocoder 130 may be a speech codec or speech processor configured to analyze and synthesize an input human speech signal for speech data compression, multiplexing, speech encryption, or speech conversion. The vocoder 130 may include several filter bands tuned across the speech spectrum. The vocoder 130 may be configured to generate speech textures using vocoder parameters. Vocoder parameters are a type of encoding parameter used to program the vocoder 130 to manipulate the received speech signal according to a number of measurable units of the vocoder parameter. The vocoder 130 may be configured to receive test data, process the test data by applying the vocoder parameters to the user utterances in the test data, and output modified user utterances that may be acoustically different from the received user utterances in the test data.

[0020] Vocoder 130 may also be configured to generate multiple variations of voice or user utterances based on values ​​provided in the vocoder parameters and test data. The received user utterances may be modulated or modified by applying vocoder parameters, background noise, or accents to change the characteristics of the user utterances. For example, Fliger distortion techniques or other known audio signal processing techniques may be used to modulate the user utterances in the test data so that the modulated or modified user utterances sound acoustically different from the received user utterances. Furthermore, vocoder 130 may receive test data including user utterances corresponding to voice commands or questions to a virtual assistant executed via voice assistant device 120, and the virtual assistant may be configured to provide expected device responses.

[0021] In another embodiment, vocoder 130 may be configured to add noise data to the vocoder parameters applied to the user utterance. Vocoder 130 may also be configured to add noise data directly to the user utterance so that when the modified user utterance is presented to voice assistant device 120, the modified user utterance includes the noise data as part of the output audio signal. Thus, the encoding parameters may include noise data. For example, one or more noise signatures may be added to the user utterance to simulate a real-world environment that includes ambient noise in the background. The noise data may include sounds generated by background television sounds, background conversations between people, natural (e.g., animal, weather) sounds, traffic (e.g., public or private vehicles / transport), or other sounds that can be detected by a microphone present in the surrounding environment.

[0022] User speech may include voice characteristics based on user characteristics that uniquely distinguish one user's voice from another user's voice. For example, voice characteristics may include pitch, speaking rate, tone, texture, intonation, loudness, etc., and one or more combinations of voice characteristics may result in a unique voice corresponding to an accent or dialect.

[0023] The voice command may be any command in a user's utterance that is one of multiple commands that voice assistant device 120 is configured to process, and upon processing the voice command, voice assistant device 120 may be configured to perform a function or issue an expected device response. The expected device response may include voice assistant device 120 performing a function corresponding to the command or question in the user's utterance. For example, the voice command may include the command "Set alarm for 6 AM," which, when processed and executed by voice assistant device 120, generates an expected device response audibly presenting "Set alarm for 6 AM." In addition to generating the expected device response, voice assistant device 120 may be configured to execute the voice command and perform the function included in the voice command.

[0024] However, voice assistant device 120 may not be fully configured to understand voice commands due to various factors, including unfamiliar accents / dialects or background noise, resulting in edge cases. According to the present invention, an edge case occurs when voice assistant device 120 receives and processes test data including modified user utterances and generates a voice assistant response that does not match the corresponding expected device response. Furthermore, an edge case may occur when voice assistant device 120 receives and processes test data including modified user utterances and fails to generate a voice assistant response or generates an error message indicating that voice assistant device 120 failed to process the modified user utterance.

[0025] Thus, in one embodiment, vocoder 130 may be configured to receive test data including vocoder parameters that, when applied to the user utterance in the test data, generate and audibly present a modified user utterance to voice assistant device 120. Upon processing and executing the modified user utterance, voice assistant device 120 may be configured to audibly present a device response that may or may not substantially match an expected device response corresponding to the user utterance. If voice assistant device 120 presents a device response that does not substantially match the expected device response, an edge case may be identified based on the test data used to generate the modified user utterance.

[0026] Speech-to-text module 140 may be a component configured to perform speech recognition processing, whereby speech data is received and processed to output text data corresponding to the received speech data. For example, speech-to-text module 140 may be configured to receive an audio signal containing speech data from voice assistant device 120, process the audio signal, and output text data corresponding to the received speech signal. The output text data may be included in the test result data transmitted to database 122. Furthermore, speech-to-text module 140 may be configured to capture a speech audio signal and convert the speech audio signal into text data. Speech-to-text module 140 may further be configured to compare text data corresponding to actual voice assistant responses with test data corresponding to expected device responses to determine whether there is a match. The test result data may be captured and stored in database 122. The test result data may include data corresponding to the voice assistant responses captured from voice assistant device 120. The test result data may also include vocoder parameters and the number of units of each vocoder parameter used to modulate user utterances in the test data.

[0027] The number of units for vocoder parameters corresponds to the quantifiable units of each parameter. For example, vocoder parameters may include the number of bands, frequency range, envelope, non-voice elements, formant shift, bandwidth, etc. Each parameter may be quantified by the number of units applied to the corresponding parameter. For example, the number of bands determines the total number of filter bands applied to the user's speech as a voice signal. A smaller number of bands (e.g., in the range of 4-6) produces a more vintage sound, while more than 16 bands provide more detail, often at the expense of processor usage. The frequency range vocoder parameter can have upper and lower limits and can be divided by the number of available bands. The frequency range vocoder parameter can be used to adjust the filter bands of the vocoder 130 to accommodate the higher or lower vocal ranges of female and male voices, respectively. The envelope vocoder parameter determines how quickly the effect reacts to dynamic volume changes, when applying this to vocals or user speech the envelope vocoder parameter provides a fairly fast reaction time, but longer reaction times can be achieved for a more impressive effect.

[0028] Additional vocoder parameters may be associated with frequency filters (e.g., high-pass, low-pass) that may help improve intelligibility for certain speech elements. For example, a high-pass filter vocoder parameter may improve the intelligibility of plosives (e.g., letters t, d, b) and sibilants (e.g., s, z, x) when processed by vocoder 130 by allowing these unvoiced elements to be better detected and identified.

[0029] Machine learning (“ML”) model 150 may include one or more processors configured to receive test result data, process the received test result data, and generate output data corresponding to the received test result data. For example, ML model 150 may be configured to receive test result data from database 122, process the test result data to generate modified vocoder parameters, and update the test data based on edge cases (i.e., the received test data). In other words, ML model 150 may be configured to fine-tune the vocoder parameters based on the test result data to generate new test data and further test voice assistant device 120 to identify more edge cases. If more edge cases are identified, ML model 150 may be better configured to generate fine-tuned vocoder parameters for modulating user utterances to generate modified user utterances that result in expected device responses from voice assistant device 120.

[0030] Additionally, in another embodiment, ML model 150 may be configured to apply chaos engineering principles to test resilience, where ML model 150 can compare test results to identify edge cases and fine-tune vocoder parameters to identify more edge cases within preset control limits. For example, if an actual voice assistant response does not match an expected device response, ML model 150 can determine that the mismatch constitutes an edge case and identify modified test data as the edge case. Once an edge case is identified, ML model 150 may be configured to modify a first vocoder parameter used in the edge case to generate a second vocoder parameter to be applied to the user utterance by vocoder 130, generating another modified user utterance for further testing on voice assistant device 120. ML model 150 may be configured to continue the vocoder parameter fine-tuning process until enough edge cases are identified to satisfy the control limits of the testing process. Once an edge case is identified, ML model 150 may be configured to modify or fine-tune the vocoder parameter corresponding to the identified edge case. The modified or fine-tuned vocoder parameters may then be applied to the user utterance to produce or generate a modified user utterance for presentation to VA device 120 for further testing. Testing of VA device 120 may continue through a sufficient number of iterations to meet the conditions associated with a successful testing plan. For example, testing of VA device 120 may continue until the device response substantially matches the expected device response corresponding to the user utterance processed by vocoder 130 and presented to VA device 120 as the modified user utterance.

[0031] The condition can be satisfied upon completion of sufficient test iterations. For example, the condition may correspond to a determination that the number of edge cases exceeds or falls short of a predetermined threshold. For example, after three iterations of fine-tuning vocoder parameters identified by edge cases, one or more processors may be configured to stop or cease sending test data to vocoder 130, signaling the end of a stage in the test plan. The number of test iterations may be based on a VA device processing time, which is the amount of time it takes for VA device 120 to process user utterances and modified user utterances. The number of test iterations may also be based on the number of identified edge cases, and a predetermined number of identified edge cases may satisfy the condition for stopping sending test data to vocoder 130. A tester may also manually stop the process of providing test data to VA device 120.

[0032] In at least some embodiments, ML model 150 may implement a trained component or trained model configured to perform the operations described above with respect to ML model 150. The trained component may include one or more machine learning models, including, but not limited to, one or more classifiers, one or more neural networks, one or more probabilistic graphs, one or more decision trees, etc. In other embodiments, the trained component may include a rule-based engine, one or more statistical-based algorithms, one or more mapping functions, or other types of functions / algorithms for determining whether a natural language input is a complex natural language input or a non-complex natural language input. In some embodiments, the trained component may be configured to perform binary classification, where the natural language input may be classified into one of two classes / categories. In some embodiments, the trained component may be configured to perform multi-class or multinomial classification, where the natural language input may be classified into one of three or more classes / categories. In some embodiments, the trained component may be configured to perform multi-label classification, where the natural language input may be associated with multiple classes / categories. The one or more processors may include an ML model 150 as described herein.

[0033] Various machine learning techniques may be used to train and operate the trained components to perform the various processes described herein. Models may be trained and operated according to various machine learning techniques. Such techniques may include, for example, neural networks (deep neural networks, recurrent neural networks, or both), inference engines, trained classifiers, etc. Examples of trained classifiers include support vector machines (SVMs), neural networks, decision trees, AdaBoost (short for "Adaptive Boosting") combined with decision trees, and random forests. Taking SVMs as an example, SVMs are supervised learning models that involve learning algorithms that analyze data and recognize patterns in the data, and are often used in classification and regression analysis. Given a set of training examples marked as belonging to one of two categories, SVM training algorithms build a model that assigns new training examples to one of these categories, making them non-probabilistic binary linear classifiers. More complex SVM models are built with training sets that distinguish between two or more categories, allowing the SVM to determine which category is most similar to the input data. An SVM model may be mapped so that examples from different categories are separated by a clear gap. New examples are then mapped into the same space and predicted to belong to a category based on which side of the gap they fall on. The classifier can emit a "score" that indicates which category the data most closely matches. The score is an indication of how closely the data matches the category.

[0034] To apply machine learning techniques, the machine learning process itself must be trained. To train the machine learning component, a "ground truth" for the training examples must be established. In machine learning, "ground truth" refers to the classification accuracy of the training set in supervised learning techniques. Various techniques can be used to train the model, including backpropagation, statistical learning, supervised learning, semi-supervised learning, stochastic learning, or other known techniques.

[0035] A computer-implemented method for testing the voice assistant device 120 is depicted and described in further detail with respect to FIG. 2.

[0036] FIG. 2 is a flowchart illustrating the operational steps of a computer-implemented method 200 for testing voice assistant devices within the distributed data processing environment of FIG. 1, according to an embodiment of the present invention.

[0037] In one embodiment, the computer-implemented method 200 for testing a voice assistant device may be executed by one or more processors configured to receive 202 test data from a database (e.g., database 122), where the test data may include at least a first set of encoding parameters and a first user utterance having an expected device response. For example, the one or more processors may include a vocoder 130 configured to receive 202 the test data from database 122 via a communication link between database 122 and vocoder 130. The vocoder 130 may include a data port configured to receive the data transmission and relay the data to a processor resident in vocoder 130 for further processing. The first set of encoding parameters may be used to configure settings for vocoder 130 and may be used to modulate or modify the first user utterance.

[0038] The computer-implemented method 200 for testing a voice assistant device may further include one or more processors configured to generate 204 a first modified user utterance by applying a first set of encoding parameters to the first user utterance, where the first modified user utterance is acoustically different from the first user utterance. For example, the one or more processors may include a vocoder 130 configured to generate 204 a first modified user utterance by applying a first set of encoding parameters to the first user utterance, as described above. The first modified user utterance may be acoustically different from the first user utterance due to at least the vocoder 130 applying the first set of encoding parameters to the first user utterance. Thus, the first modified user utterance may sound different due to one or more of the sound characteristics modified by the vocoder 130.

[0039] The computer-implemented method 200 for testing a voice assistant device may further include one or more processors configured to audibly present 206 the first modified user utterance to the voice assistant device 120. For example, the vocoder 130 may include a speaker configured to output an audio signal including the first modified user utterance for detection by the voice assistant device 120. The voice assistant device 120 may include a microphone configured to detect and receive audio signals as sound waves within proximity of the speaker of the vocoder 130.

[0040] The computer-implemented method 200 for testing a voice assistant device may further include one or more processors configured to receive 208 a first voice assistant response from the voice assistant device 120. The one or more processors may be configured to receive the first voice assistant response via the speech-to-text module 140, which may include a microphone that detects and receives the first voice assistant response as a voice input signal, converts the voice input signal to text data, and transmits the text data to one or more processors for further processing. For example, the microphone may be positioned within range of the speaker of the voice assistant device 120 such that the microphone can detect an output audio signal from the speaker of the voice assistant device 120. The microphone may be configured to transmit the detected audio signal as data corresponding to the voice assistant response to one or more processors.

[0041] The computer-implemented method 200 for testing a voice assistant device may further include one or more processors configured to determine 210 whether the first voice assistant response is substantially similar to an expected device response. In one embodiment, the one or more processors may include an ML model 150 for determining 210 whether the first voice assistant is substantially similar to an expected device response. Further, the one or more processors may be configured to determine a first score based on a comparison between the first voice assistant response and the expected device response. For example, if the first voice assistant response substantially matches the expected device response, the first score may be 1. As another embodiment, if the first voice assistant response does not substantially match the expected device response, the first score may be 0. If the first score is 0, the one or more processors may determine that an edge case involving the first test data and the voice assistant response is identified.

[0042] In another embodiment, the computer-implemented method 200 for testing a voice assistant device may further include one or more processors configured to stop receiving additional test data in response to determining that the first voice assistant response is substantially similar to the expected device response. Furthermore, in response to determining that the first voice assistant response is not substantially similar to the expected device response, the one or more processors may be configured to identify a first edge case based on at least the first set of encoding parameters and the first modified user utterance. The one or more processors may further be configured to determine a second set of encoding parameters based on at least one of the first set of encoding parameters, the first modified user utterance, and the first edge case.

[0043] Furthermore, the computer-implemented method 200 for testing a voice assistant device may further include one or more processors configured to generate a second modified user utterance by applying a second set of encoding parameters to the first user utterance, where the second modified user utterance may be acoustically different from the first user utterance and the first modified user utterance. The one or more processors may further be configured to audibly present the second modified user utterance to the voice assistant device 120, receive a second voice assistant response from the voice assistant device 120, and determine whether the second voice assistant response is substantially similar to an expected device response. Furthermore, in response to determining that the second voice assistant response is substantially similar to the expected device response, the one or more processors may be configured to stop receiving additional test data from the database 122. For example, when the one or more processors determine that the voice assistant response is substantially similar to the expected device response corresponding to the first user utterance, the one or more processors may then be configured to stop receiving test data including the first user utterance. However, the one or more processors may be configured to continue receiving test data that includes additional user utterances that are different from the first user utterance to continue the test regime.

[0044] In one embodiment, the first modified user utterance may be audibly presented to the voice assistant device 120 by a speaker communicatively connected to one or more processors (e.g., vocoder 130), the speaker configured to output the modified user utterance as an audio signal.

[0045] In one embodiment, the first voice assistant response may be received as an audio signal from the voice assistant device 120 by a microphone communicatively connected to one or more processors (e.g., speech-to-text module 140).

[0046] 3 is a block diagram of components of a server computer in the distributed data processing environment 100 of FIG. 1 in accordance with embodiments of the present invention. It should be understood that FIG. 3 is provided only as an illustration of one implementation and is not intended to suggest any limitations with regard to the environments in which different embodiments may be implemented. Many modifications to the depicted environments may be made.

[0047] 3 illustrates a block diagram of a computer 300 suitable for a voice assistant device 120 according to an exemplary embodiment of the present invention. It should be understood that FIG. 3 provides only an example of one implementation and does not imply any limitations with regard to the environments in which different embodiments may be implemented. Many modifications to the depicted environment are possible.

[0048] Computer 300 includes communications fabric 302, which provides communications between cache 316, memory 306, persistent storage 308, communications unit 310, and input / output (I / O) interface 312. Communications fabric 302 may be implemented with any architecture designed to pass data and / or control information between processors (such as microprocessors, communications and network processors), system memory, peripheral devices, and any other hardware components in the system. For example, communications fabric 302 may be implemented using one or more buses or crossbar switches.

[0049] The memory 306 and the persistent storage 308 are computer-readable storage media. In this embodiment, the memory 306 includes RAM. Generally, the memory 306 may include any suitable volatile or non-volatile computer-readable storage medium. The cache 316 is a fast memory that improves the performance of the computer processor 304 by retaining recently accessed data and data near recently accessed data from the memory 306.

[0050] Software and data 314 can be stored in persistent storage 308 and memory 306 for execution and / or access by each of the one or more computer processors 304 via cache 316. In one embodiment, persistent storage 308 includes a magnetic hard disk drive. Alternatively, or in addition to a magnetic hard disk, persistent storage 308 can include a solid-state hard drive, a semiconductor memory device, a ROM, an erasable programmable ROM (EPROM), a flash memory, or any other computer-readable storage medium capable of storing program instructions or digital information.

[0051] The media used by persistent storage 308 may be removable. For example, a removable hard drive may be used for persistent storage 308. Other examples include optical disks, magnetic disks, thumb drives, and smart cards, which are inserted into a drive for transfer to another computer-readable storage medium that is also part of persistent storage 308.

[0052] In these examples, communications unit 310 enables communication with other data processing systems or devices. In these examples, communications unit 310 includes one or more network interface cards. Communications unit 310 may enable communication using either or both physical and wireless communications links. Software and data 314 may be downloaded to persistent storage 308 via communications unit 310.

[0053] The I / O interface 312 allows for the input and output of data to and from other devices connectable to the voice assistant device 120. For example, the I / O interface 312 allows for connection to an external device 318, such as a keyboard, keypad, touchscreen, or other suitable input device, or a combination thereof. The external device 318 may also include a portable computer-readable storage medium, such as a thumb drive, a portable optical disk, a portable magnetic disk, or a memory card. Software and data 314 used to implement embodiments of the present invention may be stored on such a portable computer-readable storage medium and loaded into the persistent storage 308 via the I / O interface 312. The I / O interface 312 also connects to a display 320.

[0054] Display 320 provides a mechanism for displaying data to a user and may be, for example, a computer monitor.

[0055] The present invention can include various accessible data sources, such as databases 122, that may contain personal data, content, or information that a user does not want processed. Personal data includes user information, such as personally identifiable information or sensitive personal information, as well as tracking or geolocation information. Processing refers to any operation or set of operations, whether automated or non-automated, such as collecting, recording, organizing, structuring, storing, adapting, altering, retrieving, consulting, using, disclosing by transmission, disseminating, or otherwise making available, combining, restricting, erasing, or destroying personal data. The software and data 314 may enable the authorized and secure processing of personal data. The software and data 314 may be configured to provide informed consent with notice of the collection of personal data and allow users to opt in or out of the processing of their personal data. Consent can take several forms. Opt-in consent can require the user to take an affirmative action before their personal data is processed. Alternatively, opt-out consent can require the user to take an affirmative action to prevent the processing of their personal data before their personal data is processed. The software and data 314 may provide information about the nature of the personal data and processing (e.g., type, scope, purpose, duration, etc.). The software and data 314 provides the user with a copy of the stored personal data. The software and data 314 allows for the correction or completion of inaccurate or incomplete personal data. The software and data 314 allows for the immediate deletion of personal data.

[0056] The computer-implemented methods described herein are identified based on the application for which they are implemented in particular embodiments of the invention. However, it should be understood that any particular program nomenclature herein is used merely for convenience, and thus the present invention should not be limited to use in any particular application identified and / or implied by such nomenclature.

[0057] The present invention may be a system, a computer-implemented method, or a computer program product, or a combination thereof. The computer program product may include a computer-readable storage medium having computer-readable program instructions stored thereon for causing a processor to perform aspects of the present invention.

[0058] A computer-readable storage medium may be a tangible device capable of retaining and storing instructions for use by an instruction execution device. The computer-readable storage medium may be, by way of example, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or a suitable combination thereof. More specific examples of computer-readable storage media include portable computer diskettes, hard disks, RAM, ROM, EPROM (or flash memory), SRAM, CD-ROMs, DVDs, memory sticks, floppy disks, mechanically encoded devices having instructions recorded on punch cards or ridge-in-groove structures, or the like, and suitable combinations thereof. Computer-readable storage devices, as used herein, should not be construed as ephemeral signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission medium (e.g., light pulses passing through a fiber optic cable), or electrical signals transmitted over wires.

[0059] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device or to an external computer or storage device via a network (e.g., the Internet, a local area network, a wide area network, or a wireless network, or a combination thereof). The network may be comprised of copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, or edge servers, or a combination thereof. A network adapter card or network interface of each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage on a computer-readable storage medium within the respective computing / processing device.

[0060] The computer-readable program instructions for carrying out the operations of the present invention may be either source code or object code written in any combination of one or more programming languages, including assembler instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and procedural programming languages ​​such as the "C" programming language and similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, as a standalone software package, or partially on the user's computer. Alternatively, the computer may be executed partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA) can execute computer-readable program instructions by utilizing state information of the computer-readable program instructions to personalize the computer-readable program instructions in order to carry out aspects of the present invention.

[0061] Aspects of the present invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0062] These computer-readable program instructions can be provided to a general-purpose computer, a processor of a special-purpose computer, or other programmable data processing apparatus to create a machine, such that the instructions, executed by a processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams. These computer-readable program instructions can also be stored in a computer-readable storage medium connectable to a computer, programmable data processing apparatus, or other device, or combination thereof, that functions in a particular way, such that the computer-readable program instructions stored therein configure one of the products including instructions that implement aspects of the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams.

[0063] Computer-readable program instructions, such as instructions to perform the functions / acts specified in one or more blocks of the flowcharts and / or block diagrams on a computer, other programmable apparatus, or other device, can also be loaded into a computer, other programmable data processing apparatus, or other device to perform a series of operational steps on the computer, other programmable apparatus, or other device to produce a computer-implemented process.

[0064] The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of executable aspects of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowcharts or block diagrams may represent a module, segment, or portion of instructions, which constitute one or more executable instructions for implementing the specified logical function(s). In some alternative embodiments, the functions shown in the blocks may occur out of the order shown in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may be executed in the reverse order, depending on the functionality involved. It should also be noted that each block of the block diagrams and / or flowchart diagrams, and combinations of blocks in the block diagrams and / or flowchart diagrams, may be implemented by a special-purpose hardware-based system that performs the specified functions or operations or executes a combination of special-purpose hardware and computer instructions.

[0065] The description of various embodiments of the present invention is presented for illustrative purposes, but is not intended to be exhaustive or limited to the disclosed embodiments. It will be apparent to those skilled in the art that many modifications and variations are possible without departing from the scope of the present invention. The terms used herein have been selected to best explain the principles of the embodiments, practical applications or technical improvements to technology found in the market, or to enable those skilled in the art to understand the embodiments disclosed herein.

Claims

1. 1. A computer-implemented method for testing a voice assistant device, the computer-implemented method comprising: receiving, by one or more processors, test data from a database, the test data including a first set of encoding parameters and a first user utterance having an expected device response; generating, by the one or more processors, a first modified user utterance by applying the first set of encoding parameters to the first user utterance, the first modified user utterance being acoustically distinct from the first user utterance; audibly presenting the first modified user utterance to a voice assistant device by the one or more processors; receiving, by the one or more processors, a first voice assistant response from the voice assistant device; determining, by the one or more processors, whether the first voice assistant response is substantially similar to the expected device response; In response to determining that the first voice assistant response is substantially dissimilar to the expected device response, identifying, by the one or more processors, a first edge case based on at least the first set of encoding parameters and the first modified user utterance; determining, by the one or more processors, a second set of encoding parameters based on at least one of the first set of encoding parameters, the first modified user utterance, and the first edge case; generating, by the one or more processors, a second modified user utterance by applying the second set of encoding parameters to the first user utterance, wherein the second modified user utterance is acoustically distinct from the first user utterance and the first modified user utterance; audibly presenting the second modified user utterance to the voice assistant device by the one or more processors; receiving, by the one or more processors, a second voice assistant response; determining, by the one or more processors, whether the second voice assistant response is substantially similar to the expected device response; In response to determining that the second voice assistant response is substantially similar to the expected device response, stopping receiving additional test data including the first user utterance; and 11. A computer-implemented method comprising:

2. and stopping, by the one or more processors, receiving additional test data in response to determining that the first voice assistant response is substantially similar to the expected device response. The computer-implemented method of claim 1 , further comprising:

3. The first modified user utterance is audibly presented to the voice assistant device by a speaker communicatively connected to the one or more processors, the speaker being configured to output the first modified user utterance as a first audio signal; The computer-implemented method of claim 1 .

4. The first voice assistant response received from the voice assistant device is detected as a second audio signal by a microphone communicatively connected to the one or more processors; The computer-implemented method of claim 1 .

5. A computer program for testing a voice assistant device, the computer program comprising: and program instructions, the program instructions comprising: program instructions for receiving test data from a database, the test data including a first set of encoding parameters and a first user utterance having an expected device response; program instructions for generating a first modified user utterance by applying the first set of encoding parameters to the first user utterance, the first modified user utterance being acoustically distinct from the first user utterance; Program instructions for audibly presenting the first modified user utterance to a voice assistant device; Program instructions for receiving a first voice assistant response from the voice assistant device; program instructions that determine whether the first voice assistant response is substantially similar to the expected device response; program instructions that, in response to determining that the first voice assistant response is substantially dissimilar to the expected device response, identify a first edge case based on at least the first set of encoding parameters and the first modified user utterance; program instructions for determining a second set of encoding parameters based on at least one of the first set of encoding parameters, the first modified user utterance, and the first edge case; program instructions for generating a second modified user utterance by applying the second set of encoding parameters to the first user utterance, the second modified user utterance being acoustically distinct from the first user utterance and the first modified user utterance; Program instructions for audibly presenting the second modified user utterance to the voice assistant device; program instructions for receiving a second voice assistant response; program instructions that determine whether the second voice assistant response is substantially similar to the expected device response; In response to determining that the second voice assistant response is substantially similar to the expected device response, program instructions for stopping receiving additional test data including the first user utterance; a computer program comprising:

6. and program instructions for stopping receiving additional test data in response to determining that the first voice assistant response is substantially similar to the expected device response. The computer program of claim 5 , further comprising:

7. The first modified user utterance is audibly presented to the voice assistant device by a speaker communicatively connected to the one or more processors, the speaker being configured to output the first modified user utterance as a first audio signal; 6. A computer program according to claim 5.

8. The first voice assistant response received from the voice assistant device is detected as a second audio signal by a microphone communicatively connected to the one or more processors; 6. A computer program according to claim 5.

9. A computer system for testing a voice assistant device, the computer system comprising: one or more computer processors; one or more computer-readable storage media; and program instructions collectively stored on said one or more computer-readable storage media for execution by at least one of said one or more computer processors, said stored program instructions comprising: program instructions for receiving test data from a database, the test data including a first set of encoding parameters and a first user utterance having an expected device response; program instructions for generating a first modified user utterance by applying the first set of encoding parameters to the first user utterance, the first modified user utterance being acoustically distinct from the first user utterance; Program instructions for audibly presenting the first modified user utterance to a voice assistant device; Program instructions for receiving a first voice assistant response from the voice assistant device; program instructions that determine whether the first voice assistant response is substantially similar to the expected device response; program instructions that, in response to determining that the first voice assistant response is substantially dissimilar to the expected device response, identify a first edge case based on at least the first set of encoding parameters and the first modified user utterance; program instructions for determining a second set of encoding parameters based on at least one of the first set of encoding parameters, the first modified user utterance, and the first edge case; program instructions for generating a second modified user utterance by applying the second set of encoding parameters to the first user utterance, the second modified user utterance being acoustically distinct from the first user utterance and the first modified user utterance; Program instructions for audibly presenting the second modified user utterance to the voice assistant device; program instructions for receiving a second voice assistant response; program instructions that determine whether the second voice assistant response is substantially similar to the expected device response; In response to determining that the second voice assistant response is substantially similar to the expected device response, program instructions for stopping receiving additional test data including the first user utterance; 2. A computer system comprising:

10. and program instructions for stopping receiving additional test data in response to determining that the first voice assistant response is substantially similar to the expected device response.

10. The computer system of claim 9, further comprising:

11. The first modified user utterance is audibly presented to the voice assistant device by a speaker communicatively connected to the one or more processors, the speaker configured to output the first modified user utterance as a first audio signal; The first voice assistant response received from the voice assistant device is detected as a second audio signal by a microphone communicatively connected to the one or more processors; 10. The computer system of claim 9.

Images