System and method for automatically verifying the performance of a virtual robot

Abstract

In an automated interface program designed to interact and communicate with users, said program executing actions when a category among a predefined set of categories is activated, a method is disclosed for automatically verifying the performance of said program, the steps of said method comprising: specifying inputs under which the program should be tested; associating said inputs with conditions within categories in the program, each said condition comprising at least one response which could be given if said condition is satisfied; executing said program under at least one said input; determining whether the associated condition is satisfied upon said input; and determining whether the response associated with said condition is given upon said input. In another aspect of the present invention, the test inputs are embedded within the script itself, and specifically, within categories that can be automatically listed upon compilation of the script. Such list of test inputs can then be automatically executed to test the program. The execution of a test input can be used to check whether the test input activated the category in which the test input is embedded. The response given upon execution of a test input can then determine whether other categories are erroneously activated; or whether inputs, other than the test input, erroneously activate the category associated with the test input.

Description

REFERENCE TO A "MICROFICHE APPENDIX"Submitted herewith this present application (and on deposit in the United States Patent and Trademark Office) is a microfiche appendix comprising source code of a present embodiment of the present invention. There are 178 frames contained in 2 pages of microfiche.BACKGROUND OF THE ARTFor the purposes of the present invention, "virtual robots" (or "BOTs") are software programs that interact and / or communicate with users (human, machine or otherwise) that take actions or make responses according to input from these users. BOTs are the subject of the co-pending and co-assigned parent application entitled "Methods for Automatically Focusing the Attention of a Virtual Robot Interacting with Users", filed Jun. 4, 1997, Ser. No. 08 / 868,713 (pending), and incorporated by reference in its entirety herein. A common use of such a BOT is as an interface to a web site wherein the administrator of that site has programmed the BOT to answer simple inquiries that...

AI Technical Summary

Problems solved by technology

If the BOT can be tested in advance, the person or organization that is deploying the BOT can be more certain of its likely performance, and errors can be detected in advance that might otherwise result in mistakes that could mislead users interacting with the BOT and / or reflect poorly on the authors or deployers of the BOT.

Testing can also be said to occur after deployment as users interact with the BOT; errors found through this form of testing indicate that the BOT has already made a mistake when publicly deployed.

First, it is time-consuming.

Second, it is usually incomplete.

Finally, it is difficult to maintain assurance as changes are made to the BOT.

In most BOTs, each change can potentially affect the responses given to many other inputs, so the entire testing effort must be repeated for each set of changes that are made to the BOT.

However, there are still a number of drawbacks to the use of scripts for testing BOT performance.

First, it is a significant effort to create the initial script.

Second, modification of such a script is difficult.

The size of the script increases the complexity of this task.

Third, a straightforward script still does not allow for the easy testing of cases in which the response may vary depending on the sequence of previous inputs--although a more complicated "test scripting" language can help with this problem.

Finally, in cases where the correct response was not given, such a script does not ordinarily provide enough information to produce an error message that contains more information than the fact that the correct answer was not given.

However, the problems faced in automatic verification of natural language systems are significantly different than the problems faced in verification of other computer programs.

However, in a typical natural language system, there may be thousands of possible responses to inputs, all of which must be tested to insure that they will be given in response to appropriate inputs, and not given in response to inappropriate inputs.

Well-known techniques of black-box testing can be applied to such a system, but as described in the previous paragraph, there are significant problems with such an approach.

It is an error to end a block with SwitchBack if the block is not inside a Sequence topic.

Finally, a positive-only matching list does not allow the use of and not, in order to prevent double negatives such as "DoesNotContain X and not Y".

Currently, these Example statements are limited to textual input.

Ordinary Examples are tested in arbitrary order, which can create a problem if a BOT has some kind of intialization routine such as a routine that asks for the user's name when they first log in.

Sequences are generally more difficult to test, because all of the possible branches of the interaction must be tested.

Images