Universal automated test of embedded system

Abstract

To provide a system capable of testing the functions of a lowly embedded system.SOLUTION: In an automated test platform, a computing device including a control application is communicatively coupled to a camera, a sensor, and a robotic handler, consumes power of less than 25 watt, receives the sensor signals generated during a test, provides sensor data corresponding to the sensor signals, receives commands for the test, and provides instructions for operating the robotic handler corresponding to at least one of the commands for the test. The computing device is also communicatively coupled to an image processing interface and a database interface, receives sensor data, receives image data corresponding to images of the embedded system captured during the test, receives executable tests, and provides commands for the test.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates generally to a system and method for verifying and validating stakeholder expectations of equipment through generic automated testing of embedded systems. [Background technology]

[0002] Modern consumer, commercial, or industrial products can have logic, functionality, or workflow implemented using software, hardware, or a combination of the two. One type of product, known as an embedded system, usually contains a combination of software and hardware and generally has an inherent purpose that functions within the context of a larger whole. Embedded systems can be fixed in function or programmable, and there are a variety of embedded systems. Embedded systems include parts that operate independently, packages that require a specific environment to operate, and modules that can interoperate with software or hardware that meet certain communication standards or design tolerances.

[0003] Verification of correct operation and validation of stakeholder value (hereafter collectively referred to as "testing" or "testing") is a critical step in the design and manufacture of the embedded systems described above. There are a variety of different categories of testing, including stress testing, performance testing, usability testing, compatibility testing, and functional testing. Within these categories, many of the tests evaluate the internals of the product being tested, such as the specific way the product handles a particular task. Thus, developing such tests requires knowledge of the product internals, and running such tests requires access to the product internals.

[0004] In contrast, one type of testing known as "black box testing" or "behavioral testing" is a testing methodology that typically does not require knowledge of or access to the internal workings of the product. Instead, the product is treated as a "black box" and testing evaluates how the product works from an outside perspective. Black box testing may consider phenomena such as the behavior of the product in response to various inputs (or lack thereof) and the outputs produced by the product. Because black box testing is externally focused, it tends to be more closely aligned with the experience of actually using the product.

[0005] Generally, product testing can be manual or automated. Automated testing reduces errors and provides more thorough and meaningful documentation. Automated testing is advantageous for consistently handling repetitive or large volumes of testing that need to be performed over a long period of time or across many devices. As an example, automated testing is widely used in the software industry to test consumer software in various situations and across different platforms. In these automated tests, software companies generally run certain tools on a computer to simulate user actions with respect to a particular program and check whether the program behaves correctly in response to the simulated actions.

[0006] It is even more complicated to simulate user behavior for other types of products, such as embedded systems in consumer electronics, home appliances, and even automobiles. Some of these systems require specialized connectivity for testing and may expose internal processes through certain interfaces to allow for the simulation of user behavior, but testing such systems must be specifically designed and customized tests for each interface, making it cumbersome and expensive. A further challenge is that other embedded systems do not provide this type of testing interface, but instead only provide an interface for use by the end user.

[0007] Therefore, there is a long-standing need for an automated test platform that is versatile, flexible, and powerful enough to perform testing of embedded systems from different vendors (various types of testing, including black-box testing, to verify correct operation and stakeholder value) by interacting with the interfaces provided by each system. Additionally, there is a long-standing need for an automated test platform that does not require expensive industrial robotic equipment, other types of expensive hardware, or space to accommodate large equipment. There is a long-standing need for an automated test platform that is optimized for office use and affordable to use for testing purposes, including simultaneous testing of multiple devices.

[0008] At the same time, the automated test platform must provide detailed measurements of the tests, including, for example, timing data related to the occurrence of a particular action or response. The automated test platform must be able to provide highly relevant test results, for example, by end-to-end testing of all components that would normally be involved in a non-fake user operation. Finally, the automated test platform must be flexible enough to allow the creation of tests using a simple structured language suitable for humans, not complex computer code, and that does not require special training to read or understand. Summary of the Invention

[0009] In one embodiment of the present invention, a system for testing functionality of an embedded system includes a low-power computing device communicatively connected to a control application interface, a sensor interface, and a robot interface. The low-power computing device is configured to receive a plurality of sensor signals generated during functional testing of the embedded system from the sensor interface, provide sensor data corresponding to the plurality of sensor signals via the control application interface, receive a plurality of commands for the functional testing via the control application interface, and provide a plurality of instructions for operation of a robot handler corresponding to at least one of the plurality of commands for the functional testing via the robot interface. The system for testing functionality of an embedded system also includes a computing device communicatively connected to the control application interface, an image processing interface, and a database interface. The computing device is configured to receive the sensor data from the control application interface, receive image data corresponding to a plurality of images of the embedded system acquired during the functional testing from the image processing interface, receive a plurality of tests executable by functionality of the embedded system from the database interface, and provide a plurality of commands for the functional testing via the control application interface.

[0010] In one embodiment, the computing device is connected to a network and configured to transmit test and robot handler status data to a user over the network. In one embodiment, the computing device is configured to communicate with a user over the network to allow the user to use or modify applications running on the computing device.

[0011] In one embodiment, the low-power computing component is configured to process the sensor signal before providing the sensor data via the control application interface. In one embodiment, the low-power computing component is configured to process the commands before providing the instructions via the robot interface. In one embodiment, the computing device is communicatively connected to the sensor data processing interface, and the computing device receives processed sensor data via the sensor data processing interface corresponding to the sensor data acquired during the functional test. In one embodiment, at least one of the low-power computing device and the computing device is configured to synchronize to the same reference time as another computing device at an application level. In one embodiment, at least one of the tests includes a plurality of keywords, and the computing device is configured to convert the keywords into commands executable by the robot handler. In one embodiment, at least one of the tests includes a plurality of keywords, and the computing device is configured to convert the keywords into commands for transmission via the interface.

[0012] In one embodiment, the sensor or image data corresponds to a three-dimensional (3D) aspect of the embedded system or a test product of which the embedded system is a part, hi one embodiment, the feature being tested is a two-dimensional (2D) feature.

[0013] In one embodiment of the present invention, a method for testing a low power computing device and functionality of an embedded system executed by the computing device includes the steps of receiving, by the low power computing device, sensor signals generated during testing of the functionality of the embedded system from a sensor interface; providing, by the low power computing device, sensor data corresponding to the plurality of sensor signals via a control application interface; receiving, by the low power computing device, a plurality of commands for functional testing from the control application interface; providing, by the low power computing device, a plurality of instructions for operation of a robot handler corresponding to at least one of the plurality of commands for functional testing via the robot interface; receiving, by the computing device, the sensor data from the control application interface; receiving, by the computing device, image data corresponding to a plurality of images of the embedded system acquired during the functional testing from the image processing interface; receiving, by the computing device, a plurality of tests executable by the functionality of the embedded system from a database interface; and providing, by the computing device, a plurality of commands for functional testing via the control application interface.

[0014] In one embodiment, the computing device is connected to a network and the computing device is configured to transmit test and robot handler status data to a user over the network. In one embodiment, the computing device communicates with a user over the network to receive and process requests to use or modify an application executing on the computing device.

[0015] In one embodiment, the low power computing component processes the sensor signals before providing the sensor data via the control application interface. In one embodiment, the low power computing component processes the commands before providing the instructions via the robot interface. In one embodiment, the computing device is communicatively coupled to the sensor data processing interface, and the computing device receives processed sensor data via the sensor data processing interface corresponding to sensor data acquired during functional testing. In one embodiment, at least one of the low power computing device and the computing device are synchronized at an application level to the same reference time as another computing device.

[0016] In one embodiment, at least one of the plurality of tests includes a plurality of keywords, and the computing device converts the keywords into commands executable by the robot handler. In one embodiment, at least one of the plurality of tests includes a plurality of keywords, and the computing device is configured to convert the keywords into commands for transmission over the interface.

[0017] In one embodiment, the sensor or image data corresponds to a three-dimensional (3D) aspect of the embedded system or a test product of which the embedded system is a part, hi one embodiment, the feature being tested is a two-dimensional (2D) feature. [Brief description of the drawings]

[0018] [Figure 1] 1 is a block diagram of an automatic test platform according to one embodiment of the present invention; [Diagram 2] FIG. 2 is a diagram of components of a robot handler for use in an automated testing platform in accordance with one embodiment of the present invention. [Diagram 3] FIG. 2 is a block diagram of components of a control application for use in an automatic test platform according to one embodiment of the present invention. [Figure 4]FIG. 2 is a diagram illustrating an overview of a database for use in an automatic testing platform according to an embodiment of the present invention. [Figure 5A] 1 is a screen image illustrating an exemplary user interface for setting up a control application according to one embodiment of the present invention; [Figure 5B] 5B is a screen image showing an example of a sub-screen of a user interface that can be reached from the user interface of FIG. 5A according to one embodiment of the present invention; [Figure 6] 2 is a flow chart for testing in an automated testing platform according to an embodiment of the present invention showing the steps of creating a new definition of a product. [Figure 7] 4 is a flow chart illustrating the steps performed to calibrate a robot in an automatic testing platform according to one embodiment of the present invention. [Figure 8] 5B is a screen image illustrating a calibration user interface that can be reached from the user interface of FIG. 5A. [Figure 9] FIG. 8 is a flow chart illustrating steps performed in a semi-automatic calibration method for performing an automatic recalibration of a robotic handler after a first pass through the flow of FIG. 7 according to one embodiment of the present invention. [Figure 10] 2 is a flow chart illustrating steps for performing image acquisition and image processing in an automated test platform according to one embodiment of the present invention. [Figure 11] 1 is a sample test according to an embodiment of the present invention that may be performed on an automated test platform. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0019] FIG. 1 is a block diagram of an automated testing platform according to an embodiment of the present invention, showing how each part of the platform is connected to the other parts. As shown in FIG. 1, a control application 104 runs on a computing device 100 and communicates with a database 116. The computing device 100 may be any device capable of running a control application, such as a personal computer (PC), a server, a laptop, a smart phone, a tablet, or the like. The computing device 100 typically includes a processor in communication with volatile and / or non-volatile memory, a peripheral interface, and a network interface. The processor interfaces with the database 116 and other components of the automated testing platform, such as a robotic handler 118, through the network interface. The peripheral interface connects the computing device 100 to various input devices, which may include a keyboard, a mouse, a touch screen, a VR input mechanism capable of corresponding to the position of the robotic handler in 3D space, or other devices suitable for inputting local user commands to control the computing device 100. It should be understood that the computing device 100 and / or the control application 104 may include dedicated hardware (or may be an embedded system) tailored to operate as part of an automated test platform. Such specialization may improve the speed and responsiveness of the test process. For example, the dedicated hardware in the computing device 100 may include multiple computing components connected to each other via several communication paths, such as a set of interfaces. The low-power computing components may, for example, consume less than 25 watts and may be connected to the robot handler 118 via a robot interface and / or to the sensors 126 via a sensor interface, while being connected to other computing components that support, depend on, or execute modules in the computing device 100 via a control application interface.Moreover, such low-power computing components may be multiple-linked to the same shared computing component that supports, depends on, or executes modules in the computing device 100 through a shared or individual control application interface. Low-power computing components may offer resource savings and scalability advantages over other types of computing components. It should be noted that low-power computing components do not necessarily mean "low-function" computing components. For example, specialized hardware, such as dedicated circuitry for a specific purpose (e.g., graphics processing), may be used to increase the computing power of a low-power computing component for a given purpose. It should also be understood that there may be multiple computing devices and multiple control applications, or multiple computing devices that share the processing of one or more control applications. Furthermore, there may be libraries or services that allow for consistent packaging and deployment of software updates across multiple computing devices and control applications. Such libraries or services would allow users to maintain different versions of software and update those versions centrally.

[0020] The control application 104 interfaces with a shared test library module 106, an image acquisition module 108, and a sensor data processing module 110 to communicate with a robot handler 118. The shared test library module 106 interfaces with a robot framework test environment 112, which interfaces with a database 116.

[0021] It should be understood that, as in the embodiment shown in FIG. 1, the database 116 and the robot framework test environment 112 can be separate from the computing device 100 and communicate via the database 116, or in other embodiments, the database 116 and the robot framework test environment 112 can be part of the computing device 100. The robot handler 118 interacts with an embedded system 122 of the product 120 being tested. The robot handler 118 can achieve interaction through a user interface, such as a two-dimensional (2D) function of the embedded system. The 2D function of the embedded system includes at least a touchscreen interface, but it should be understood that the 2D function can also refer to physical analogs to a touchscreen (e.g., buttons, dials, and sliders). It should be understood that operations on a touchscreen interface need not be limited to pressing buttons. The robot handler 118 can also perform operations required to act on the following types of user interfaces: (1) a pressure-sensitive touch screen capable of detecting different levels of pressure, (2) a multi-point-aware touch screen capable of recognizing simultaneously activated points, or (3) a touch screen that recognizes an executed movement such as a swipe direction or symbol tracing. It should also be understood that there may be multiple robotic handlers controlled by one or more control applications. Additionally, there may be multiple embedded systems or products being tested by one or more robotic handlers. In situations involving multiple robotic handlers and / or multiple embedded systems or products being tested, a robotic management system may be used that coordinates the testing of all devices. Such a robotic management system would allow the use of multiple users' robotic handlers to be scheduled to avoid overlaps. The robotic management system also allows for prioritization of tasks to enable collaboration of all testing robotic handlers and / or devices with matching priorities such as speed, coverage, minimization of resource costs, or other efficiency measures.

[0022] The shared test library module 106 may include or make accessible definitions of various procedures and methods for performing different aspects of each test. One example may be a method of authenticating using a PIN number, which may include operating the robot handler 118 or other robot handler to interact with the embedded system 122, for example by tapping a button based on various flows defined in the database 116, which navigates to an appropriate user interface and enters a PIN number into the embedded system 122. The method of authenticating using a PIN number may also include requesting feedback from other components of the test platform, such as the image acquisition module 108 and the sensor data processing module 110, so that the actual feedback can be compared with the expected feedback and appropriate return values ​​or conditions can be updated for evaluation by the robot framework test environment 112.

[0023] The robot framework test environment 112 is an application that can manage the robot handler 118 or other robot handlers. As such, the robot framework test environment 112 is aware of the status of any such robot handlers in the system, as well as the status of other modules of the test platform, including, but not limited to, the image processing module 114 and the camera 124. The robot framework test environment can perform actions such as processing test requests, reserving a robot handler to perform a test, and running the test. The robot framework test environment 112 can execute a test that includes a series of steps. Each step corresponds to a keyword or key phrase. The keyword or key phrase can correspond to a particular procedure or method that is stored in, made accessible by, or defined in the shared test library module 106, for example. For example, there can be a keyword or key phrase "PIN authentication" that refers to a particular procedure or method that is stored in, made accessible by, or defined in the shared test library module 106 referenced in the previous paragraph. The robot framework test environment 112 can use the keywords or key phrases to execute each procedure or method of the shared test library module 106 and receive a return value or evaluate a state updated by the procedure or method. Upon receiving such a value or evaluation of such a state, the robot framework test environment 112 can decide to stop execution or to proceed to execute another test step.

[0024] The computing device 100 can be connected to a network 128. The network 128 may be any known communication network, including the Internet, a local area network (LAN), a wide area network (WAN), and the like. A user 130 is connected to the computing device 100 via the network 128 and runs a control application 104 via the server 102. Through the connection, the user can use the control application to monitor the activities of the bot handlers and modify the software that controls them. The server 102 may be, for example, a Representational State Transfer (REST) ​​server for communicating with user clients. The server 102 may implement other mechanisms for server system architecture, including Common Object Request Broker Architecture (CORBA), Distributed Component Object Model (DCOM), Remote Procedure Calls (RPC), or Simple Object Access Protocol (SOAP), for communicating with user clients.

[0025] In an embodiment where multiple users share one automated testing platform, users can "reserve" the use of a robot handler to avoid overlaps caused by potentially conflicting simultaneous requests from different users. Users can also "unreserve" a robot handler after completing a test or determining that they no longer need the robot handler. However, it should be understood that a user may accidentally leave a robot handler in a "reserved" state, preventing other users from using the robot handler. In such a case, the robot handler can be "unreserved" by itself after a certain period of inactivity, making it available to other users.

[0026] The camera 124, as a particular version of a visual sensor (see general sensor description below), observes the embedded system 122 or other parts of the product under test 120, including 2D features or 3D aspects of the embedded system 122 or product under test 120, and communicates with the image acquisition module 108. As an example, during testing of 2D features in the embedded system 122, the observation of the 3D aspects by the camera 124 can result in image data reflecting mechanical aspects of the response behavior of the embedded system 122 or product under test 120. The image acquisition module 108 then interfaces with the image processing module 114, which further communicates with the database 116. It should be understood that, as in the embodiment shown in FIG. 1, the image processing module 114 can be separate from the computing device 100 and communicate via an image processing interface. Alternatively, it should be understood that in other embodiments, the image processing module 114 can be part of the computing device 100. It should also be understood that there can be multiple cameras 124 observing the embedded system 122 or other parts of the product under test 120.

[0027] Similarly, the sensor 126 observes other parts of the embedded system 122 or the product under test 120, including 2D features or 3D aspects of the embedded system 122 or the product under test 120, and communicates with the sensor data processing module 110. As an example, during testing of the 2D features of the embedded system 122, the observation of the 3D aspects by the sensor 126 can result in a sensor signal reflecting a mechanical aspect of the response manner of the embedded system 122 or the product under test 120. The sensor data processing module 110 receives the sensor signal and performs any processing if necessary to generate sensor data. For example, some sensor signals provided by sensors that detect binary states or provide numerical values ​​can constitute sensor data without any additional processing depending on the test for which the sensor data is interpreted. The sensor data processing module 110 communicates with the database 116 and provides data to the control application 104 so that the sensor data can be interpreted in a manner corresponding to the product under test. The sensor 126 could be an optical sensor, a visual sensor, a temperature sensor, a humidity sensor, a motion sensor, a mechanical sensor, a pressure sensor, an audio sensor, or any other sensor adapted to the particular characteristic being evaluated of the embedded system or product under test. For example, in one embodiment, the product under test is a multi-function printing device and the test involves checking a particular characteristic of the print output. The sensor 126 is an optical / mechanical sensor that detects the presence or absence of paper, whether it is double-sided or single-sided, the size of the paper, the weight of the paper, the black level of the print output, or the color of the print output, among other printed outputs to be checked. In another embodiment, the product under test is a 3D printer and the sensor 126 could be a temperature or humidity sensor to assess the condition of the print chamber, or an optical sensor to assess the progress of the print. In another embodiment, the product under test includes a physical switch or other pressure sensitive input. The sensor 126 could be a tactile sensor to assess the force required to actuate such a switch or input. It should be noted that there are different embodiments that operate with multiple sensors 126. For example, in one embodiment, the test product is a washing machine and the first sensor can be a temperature sensor to assess the water temperature.The second sensor may be a microphone to assess noise levels during operation. Additionally, in some embodiments, the sensor 126 may be connectable to a local computing device. The local computing device may be a low-power computing device, such as a microcomputer Raspberry Pi™, that processes the input locally before sending it to the computing device 100. The local computing device may be a low-power computing device that performs processing of the signals received from the sensor 126 to generate sensor data. In such embodiments, the functionality of the sensor data processing module 110 may be simplified or eliminated.

[0028] The robot handler 118 of FIG. 1 may include various components. FIG. 2 is a diagram illustrating the robot handler 218 in a physical testing environment according to an embodiment of the present invention, depicting its components. In some embodiments, the robot handler 218 may be used as the robot handler 118 shown in FIG. 1 and described in conjunction therewith. The robot handler 218 includes at least one manipulation device 214 driven by one or more servo motors 202, 204, 206, and 208. The number of servo motors is not limited to four, and may be more or less. The characteristics of the manipulation device 214 depend on the desired manipulation of the embedded system 222 or the test product 220. For example, if the embedded system 222 is a touch screen, the manipulation device 214 may be a stylus suitable for pressing a button on the screen. For example, if the embedded system 222 is a display with physical buttons, the manipulation device 214 may be a stylus suitable for pressing a physical button. If the embedded system 222 includes a touch screen in addition to physical buttons, the manipulation device 214 may be a stylus or multiple styluses capable of pressing both the screen and the physical buttons. If the embedded system 222 includes other methods of manipulation, such as a dial, the manipulation device 214 may be a gripper suitable for rotating the dial. In other embodiments, the manipulation device 214 may include multiple styli, tools, or attachments from which an appropriate one can be selected for a desired manipulation of the embedded system 222. The camera 234 may be mounted on an adjustable tripod 230 at a suitable distance, height, and angle to view the embedded system 222 or other parts of the test system 220. Although not depicted in FIG. 2, it should be understood that there may be test environments in which multiple cameras are used.

[0029] The robot handler 218 also includes a positioning device 200, such as an adjustable stand, so that it is positioned in the correct position to perform operations on the embedded system 222 and the test product 220. In addition, there may be one or more auxiliary servo motors, such as servo motor 210, which have specific roles that can be performed independently of the operation device. For example, the servo motor 210 could configure the hardware 212 to allow an access card or other authentication device 216 to be presented for verification by the reader 224 of the test product 220. The access card or authentication device may be a single card or device or emulator that can represent the account information of other cards or devices, so that the card or device does not need to be swapped for testing of different user accounts. The auxiliary servo motors are optional and could be replaced by other auxiliary devices that do not need to perform physical operations. For example, the authentication device 216 could interface with the embedded system 222 or the reader 224 through other means, including through a direct electronic link, eliminating the need for the servo motor 210 to perform physical operations.

[0030] In one embodiment, the robot handler 218 can move the manipulation device to a specific resting position and turn off the servo motors 202, 204, 206, and 208 after a predefined but configurable period of inactivity, e.g., 5 minutes, reducing the load on the servo motors during such resting periods and extending their lifespan. Upon receiving a new command, the robot handler 218 wakes up from the resting state and turns the servo motors back on to execute the command.

[0031] FIG. 3 is a block diagram of components of a control application 304 according to an embodiment of the present invention. In some embodiments, the control application 304 can be used as the control application 104 shown in FIG. 1 and described herein with reference to the accompanying drawings. The robot control module 306 regulates connectivity to and general control of a robot handler, such as 118 or 218 shown in FIG. 1 and FIG. 2, respectively, and described herein with reference to the accompanying drawings. The calibration module 308 can be used to calibrate the robot handler, a camera, such as 124 shown in FIG. 1 and described herein with reference to the accompanying drawings, or a sensor, such as 126 shown in FIG. 1 and described herein with reference to the accompanying drawings. The database module 310 manages the information and structure of a database, such as the database 116 shown in FIG. 1 and described herein with reference to the accompanying drawings. The database module 310 can include a user interface or other tools to modify the information and / or structure of the database. For example, the user interface can provide the ability to update different types of input fields or other UI components in the database, or to input new instances of existing forms. Finally, network module 312 manages connections to servers, such as server 102 shown in and described with reference to Figure 1, or shared test library modules, such as shared test library module 106 shown in and described with reference to Figure 1. These components are all exposed to the end user via user interface 314, a possible example of which is shown in Figure 5.

[0032] 4 is a block diagram illustrating a database 416 according to one embodiment of the present invention, showing an overview in an automated testing platform. In some embodiments, database 416 can be used as database 116 shown in and described with respect to FIG. 1. A "Manufacturer" table 420 contains information about each manufacturer. A "Printer Model" table 422 contains definitions of manufacturers and models. A "Hw Version" table 424 contains entries corresponding to versions of hardware used in the printer models. A "Firmware Version" table 426 contains entries corresponding to versions of firmware used in the printer models.

[0033] The "Sq Version" table 428 includes entries corresponding to the version of the Print Manager server used with the printer model. An exemplary implementation of the Print Manager server may be a YSoft SafeQ™ server. The "SafeQ Server" table 430 includes information about each Print Manager server, which in one embodiment may be a YSoft SafeQ™ server. The "Path" table 432 defines constants for moving the robot. Such constants may include values ​​specific to certain types of devices with touch displays, such as, but not limited to, capacitive and resistive touch displays. The "Path Parts" table 432 defines constants for speed and pauses of robot movement. The "MFPs" (for multifunction printers or multifunction devices) table 436 includes definitions for the printer model, Print Manager version, firmware version, hardware version, and screen size. The "Flow Name" table 438 includes the names of user-defined flows. Each user-defined flow may be implemented differently across different vendors and different devices. Thus, the "Flow" table 440 defines a user-defined flow for a desired operation, including a name identification from the "Flow Name", the particular vendor and / or device to which the flow is applicable, and a corresponding entry in the "Flow Step" table. The "Flow Step" table 442 contains step definitions that describe the flow for the desired operation, as referenced by the "Flow" table 440. Each step definition may include the particular interface part to be operated on, the order in the flow, and other options for the step, such as the ability to skip confirmation of the target screen for faster execution. The "Screen" table 444 contains definitions for each screen in the embedded system's interface, including, for example, the login screen and the main screen. The "Button" table 446 defines the text, tag, target screen, source screen, and whether the button is a hardware button.The "Button Layout" table 448 includes definitions of the MFP, location, size, and buttons. The "Functional Areas" table 450 includes definitions of areas on the screen that may be used, for example, by image processing services. These definitions include areas that have a specific purpose, such as text areas, image areas, and pop-up window areas, or areas where a specific type of processing can be performed, such as a reading area. The "Test Results" table 452 includes definitions of the results of various tests that have been performed, including a definition of the test name, Internet Protocol or other network address, timing information such as start, end, duration, output, format, and the final result or conclusion of the test. The "Print Times" table 454 includes definitions of print jobs, including job size, job format, device, and version control information at the print manager server, which in one embodiment may be a YSoft SafeQ™ server.

[0034] FIG. 5A is a screen image according to one embodiment of the present invention illustrating a user interface 500 for the components of the control application of FIG. 3. As shown in this image, the robot control user interface 510 includes controls for connecting to the robot handler, such as selecting a network port for connection, for disconnecting from the robot handler, and for controlling the auxiliary servo motors to swipe a card at the test product. In this exemplary user interface, there is a button labeled "Swipe Card," but as previously mentioned, it should be understood that the authentication device may connect to the embedded system 222 or reader 224 through other means that do not require a physical card swipe. In such a scenario, the "Swipe Card" button initiates the authentication process necessary to gain access to the test product. The calibration user interface 520 includes controls for calibrating the robot, calibrating the camera, and loading a saved calibration. Briefly, calibrating the camera ensures that the camera is in the best position to take a clear photograph and also provides the user with a way to configure the camera's characteristics (e.g., exposure settings such as shutter speed, ISO speed, aperture, etc.). Calibrating the robot is described in further detail in relation to Figures 7-9. Network user interface 530 includes controls for indicating the port on which the REST server responds, controls for indicating the port on which the shared test library module responds, and controls for switching the connection on and off. Database user interface 540 includes controls for managing each of the tables described in the overview of Figure 5. Control 550 allows updating the version of the control application stored on the server, opening a dialog where the user can select and install a different version of the control application.

[0035] It should be noted that in this embodiment, there are no visible controls over the "Path" table 432 shown in and described with reference to FIG. 4, which defines constants for moving the robot in response to the touch display, and the "Path Parts" table 434 shown in and described with reference to FIG. 4, which defines constants for the speed and pauses of the robot's movements. Typically, these tables do not need to be changed unless a new type or configuration of robot handler is added to the test platform. Other embodiments could include controls over these tables, with special warnings to warn when modifying these tables, or access controls to only allow modifications to certain users with the proper authority or permission level.

[0036] FIG. 5B is a screen image according to an embodiment of the present invention illustrating user interface sub-screens 560 and 570 from which the "Button" table 446 and the "Button Layout" table 448 shown in and described with respect to FIG. 4 can be reached from the user interface of FIG. 5A. According to this embodiment, the "Button" table contains definitions of buttons in various test product embedded systems. An example of an entry in the "Button" table could include text: "PIN", tag: "openKeyboard", destination screen: "PIN", and source screen: "Login screen". This would represent a button in the test product embedded system that is displayed on the login screen so that the user can enter a PIN, and that when pressed opens the keyboard in the PIN screen. The "Button Layout" table could tie the buttons defined in the "Button" table to a particular printer. An example of an entry in the "Button Layout" table for a button with a tag of "openKeyboard" could include printer: "Konica Minolta 421" (Konica Minolta is a trademark), position "x: 57 y: 66 z: 0", and size "W: 33, H: 6". This indicates that on the "Konica Minolta 421" test product, the openKeyboard button is at coordinates x:57, y:66, z:0, and is 33mm wide and 6mm high. In this embodiment, the coordinates are defined relative to x:0, y:0, z:0, which corresponds to the bottom left corner of the screen.

[0037] FIG. 6 is a flow chart according to one embodiment of the present invention showing steps for creating a new definition of a product in a database for testing. Although other flows are possible, this exemplary flow ensures that all data is added in a manner consistent with the database outline in the embodiment of FIG. 4 and that prerequisite data is entered into certain tables before other tables need to be added. Step 600 defines an entry in a "Manufacturer" table to represent the manufacturer of the test product. Step 602 defines an entry in a "Printer Model" table to represent the model of printer and associate it with their corresponding manufacturer. Step 604 defines an entry in a "Hw Version" table to represent the hardware version of the printer model. Step 606 defines an entry in a "Fw Version" table to represent the firmware version of the printer model. Step 608 defines an entry in an "Sq Version" table for the printer model to represent the version of the print manager server, which in one embodiment is a YSoft SafeQ server. Step 610 defines an entry in a "SafeQ Server" table corresponding to other information about the print manager server that could be used to manage the printer. Step 612 defines entries in the "MFPs" table, including definitions of the printer model, print manager server version, firmware version, hardware version, and screen size. Step 612 for defining the MFP entries is performed in the definition flow because there may be different multifunction printers of the same model with different print manager server versions, firmware versions, or hardware versions. Step 614 defines entries in the "Screen" table, representing the various screens of the interface in the embedded system of the test product. Step 616 defines entries in the "Button" table, representing the various buttons corresponding to each screen. Step 618 defines entries in the "Button Layout" table, representing the location and size of each button on the screen of each MFP.

[0038] In certain embodiments, the flow of defining a new product for testing may be performed manually, or a subset or even all of the steps may be assisted by a helper application or performed automatically. For example, in one embodiment, with an auto-training feature, the control application may use a robotic handler and a camera to navigate the screens of a new test product's embedded system by trying out the buttons and recording the results and the navigation flow between the screens. This adds information in the "Screen", "Button" and "Button Placement" tables to the database so that the screens can later be appropriately named.

[0039] FIG. 7 is a flow chart according to an embodiment of the present invention showing steps performed to calibrate a robot handler 118 and 218, etc., in an automated testing platform, consistent with the exemplary user interface shown in and described in connection with FIG. 5A. The purpose of the calibration is to ensure that the robot handler knows where it is relative to the embedded system being tested. Calibration of the robot handler may be basic or semi-automatic. The general process for initiating both basic and semi-automatic calibration methods is the same, identifying three corners of the screen that are mapped according to the flow in FIG. 7. However, while basic calibration only goes through the calibration flow once, semi-automatic calibration continues until the stage of automatic recalibration. In one embodiment, the calibration results can be verified by using a "try" function to have the robot handler tap selected buttons on the embedded system's screen.

[0040] In step 700, the robot is securely connected by using a robot control such as corresponding to 510 in the exemplary user interface of FIG. 5A. In step 702, the robot is physically positioned in front of the test product so that it can reach the on-screen components and any hardware components of the embedded system, such as buttons, required for testing. In step 704, the “Calibrate Robot” control of the user interface is activated, such as the button depicted in section 520 in the exemplary user interface of FIG. 5A. In step 706, a test product that has already been created is selected, for example, according to the flow of FIG. 6. The calibration interface displays a series of prompts on the screen for the user to follow. In step 708a, the robot handler is moved to the bottom left corner of the screen. In step 708b, the robot handler is moved to the top left corner of the screen. In step 708c, the robot handler is moved to the top right corner of the screen.

[0041] 8 is a screen image illustrating a semi-automatic calibration user interface that guides a user through an initial manual calibration comparable to the steps for a basic calibration. In this embodiment, the robot handler is equipped with a stylus as a manipulation device and is controlled via keyboard input. It should be understood that other manipulation devices and other forms of control can be used, including manual positioning, where possible with a robot handler that can receive manual positioning input and control the angles of servo motors using inverse kinematics.

[0042] The basic calibration now finishes with saving the calibration results and the user interface notifies that the robot handler has been calibrated to the test product. The calibration results consist of the x, y, and z coordinate values ​​of the three corners of the mapped screen, defining a plane in 3D space. In this embodiment, the coordinates are defined relative to x:0, y:0, z:0, which corresponds to the center of the first servo motor of the robot handler.

[0043] As described above, the semi-automatic calibration is performed through an initial basic calibration process followed by an automatic recalibration stage. FIG. 9 is a flow chart according to one embodiment of the present invention showing steps performed in a semi-automatic calibration method for automatic recalibration of a robot handler after an initial pass through the flow of FIG. 7. Step 730 is preferably performed before the flow of FIG. 7 and sets up a terminal connection to the control application of the test product. The control application outputs the (x, y) pixel coordinates of any detected taps on the screen of the embedded system. In step 732, the screen dimensions and resolution of the embedded system's screen are loaded into the control application. These are used to convert any pixel coordinates (x, y) to a position (x1, y1) on the screen in millimeters. In step 734, the control application uses these sources of information to position the robot handler using the data of the first mapped corner derived from the first pass through the semi-automatic calibration flow. Rather than positioning the stylus exactly as mapped, the robot handler starts initially at a position away from the screen (i.e., very close to its initial position). In step 736, the robot handler approaches the screen until a contact is output by the terminal connection. In step 738, the output pixel coordinates (x,y) are processed and converted to a physical location on the screen (x1,y1) in millimeters. The automatic recalibration process repeats steps 734 through 738 for each of the three mapped corners. Next, in step 740, the physical location data of the three mapped corners of the screen is used to recalculate the corrected physical locations (x,y,z) of the three mapped corners, along with the output pixel coordinate (x,y) data, replacing the data previously recorded in the first phase of the semi-automatic calibration process. The new data set resulting from the automatic recalibration process removes user error and provides absolute accuracy for positioning the robot handler relative to the screen of the embedded system in the test product.

[0044] FIG. 10 is a flow chart according to one embodiment illustrating steps performed for image acquisition and processing in an automated testing platform consistent with the system shown in and described with respect to FIG. 1. The automated testing platform includes image acquisition and processing and has the capability to recognize features in the image to provide visual feedback to a control application representative of the behavior of the tested product and / or the embedded system. In a system without image analysis capabilities, the robot may continue to process its steps without recognizing that something is wrong with a particular tested product, e.g., the paper tray is empty, the output tray is full, or an unexpected warning / error message is displayed on the screen.

[0045] In step 800, the control application 104 requests an image analysis of the embedded system 122 and / or the test product 120. In step 802, the camera 124 takes a picture of the embedded system 122 and / or the test product 120. In step 804, the camera 124 sends the image data to the image acquisition module 108. In step 806, the image acquisition module 108 sends the information to the image processing module 114, which may be local or may reside on a separate server connected to the image processing module via a network. The image processing module 114 then analyzes the image using two techniques, together with information already stored in the database 116 (e.g., screen definitions, button definitions, button placement definitions, etc.): (1) in step 808, using existing image data for various screens of an interface on an embedded system, and using descriptors that are unique regions within the images that can be computed by training a neural network to recognize, match, or otherwise compare images; (2) In step 810, an optical character recognition (OCR) engine is used to recognize and extract glyphs, characters, or symbols from the image. In some embodiments, an OCR engine may be used to provide information used in automatic identification of descriptors in step 808. The OCR engine may be a remote application accessible over a network and may be shared by multiple claimants. For example, multiple computing components may request the OCR engine to process an OCR task. In response, the OCR engine will perform the processing by using all available resources to perform the requested task or by occupying a limited set of resources, such as a single node.

[0046] The descriptors of step 808 may be further recalculated by the neural network pending changes to the image or any existing image data that is computed and analyzed at the start of image processing. Evaluation of the image may include comparing the computed or recalculated descriptors to any existing image data stored in the database 116 or any other resource accessible by the image processing module 114. The database 116 or such other resource may include, for example, a central storage application that serves as a widely accessible data repository of information including image data. Thus, multiple claimants may communicate with the database 116 or such other accessible resource to retrieve information. The information stored in the database 116 includes data specific to each screen of the interface, including images of each screen of each vendor's different product models. The evaluation may further include using probabilistic methods to analyze the screens based on image data from the database 116. Such methods may include, but are not limited to, BRISK or ORB algorithms operable on the descriptors for detection and matching purposes. The method detects certain descriptors of the image and compares them with image data from the database 116 or other resources accessible by the image processing module 114, and can identify whether there is a match based on the descriptor. Using this information, the above two techniques allow the robotic handler to recognize the screen that needs to be operated on and whether there is a warning, error, or other condition that the robotic handler needs to recover from. The descriptors are often sufficient to recognize the correct screen, meaning that after step 808 the image processing flow ends. If the descriptors are not sufficient to identify the screen, the flow proceeds to step 810. In step 810, OCR can be used to recognize and extract words from the image as a source of additional information to assist in the recognition of the correct screen.

[0047] FIG. 11 shows a sample test according to an embodiment of the present invention that can be executed on an automated test platform. Preparation of such tests is easy for end users, since the script language used is not a complex computer code, but a simple, structured language suitable for reading and understanding by people without special training. The test platform uses a shared test library module to translate the simple script into commands specific to the various components or modules of the test platform, using keywords defined in or made accessible by the shared test library module. Such commands may include, for example, instructions to obtain or capture information from a sensor or camera, instructions to perform data processing or analysis of image data, sensor signals or sensor data, instructions to register information in, obtain information from or manipulate information in a database, or instructions for a robotic handler. These commands may be processed and sent to the appropriate components or modules of the test platform via an interface corresponding to the components or modules of the test platform. The translation may be different for different vendors, printer models, or hardware / firmware versions of the test product. For example, a simple statement such as "if the user is authenticated with a PIN" may include different PIN screens or different PIN formats depending on the test product. This system has two primary advantages: (1) Testers do not need to be engineers with specific technical knowledge of how to operate a particular test product; (2) The testing platform can apply the same generic script to different testing products by transforming it as necessary based on product-specific information stored in a database. This system has advantages over existing testing methods in that it allows testers to design and perform verification tests as well as confirmation tests. Confirmation tests involve checking specific functions of a system and evaluating whether they function properly and conform to the system's product specifications to return expected results. Verification tests involve a different type of testing that focuses on whether the product meets the user's needs rather than conformance to specifications. Verification tests involve evaluation of high-level characteristics of the product, such as the overall speed of the system, the system's reliability over time, consistency of results across multiple tasks, and other areas that would be difficult or impossible for an individual tester to determine based on individual manual tests.

[0048] The automated test platform may run in certain states from which it can attempt to recover. There are two types: (1) expected states with specified recovery routes, and (2) unexpected states. An example of an expected state may be a case where the registration of a PIN code for the authentication of a test product was not successful because the entered PIN did not match the expected format of the test product. In such a case, the camera would take an image of an error window indicating the PIN code is incorrect, an error window of incorrect format, an error window not recognized, etc., so that the control application would notice the error status. In this event, the image processing module would recognize that the message indicates invalid recognition information, and the control application could command the robot handler to enter a PIN code of a different format.

[0049] There are also two types of unexpected conditions: (1) the kind of unexpected conditions that arise as a result of navigation to the embedded system's screens, where the controlling application can return to a known screen or state of the test flow, and (2) the kind of unexpected conditions that lead to an exit from the controlling application without any explicit recovery command. In the first case, the controlling application can guide the robot handler to press the appropriate button on the screen to navigate to the appropriate screen or state, and continue the test flow. For example, if the controlling application wants to test print jobs #1 and #3, but for some reason #2 and #4 are also selected, the application can command the robot handler to deselect #2 and #4 before running them. In the latter case, the controlling application can command the robot handler to press the appropriate "OK" or "Cancel" button on the screen to reset the flow to a known state and try again. If after another attempt, only to face the same unexpected error condition, the controlling application can check the machine's status and, if necessary, send a notification to the administrator to enter the appropriate recovery procedure later. It should be understood that it is not desirable to always recover from all unexpected conditions in an automated test platform, because at least some of these conditions will represent defects in the test product, rather than the control application, and the test product must be noted and repaired.

[0050] The present invention significantly improves the state of the art on standard test procedures by providing an automated testing platform for embedded systems. The consistency and speed of the robotic handler allows tests to be performed at a higher volume and with a higher level of precision compared to, for example, manual testing. Furthermore, the results of the test execution are more detailed and complete. Test scripts are fully reproducible with a level of precision, not only of the steps performed but also the exact timing, position, and even the force of button presses. These tests can measure certain product qualities that are difficult or impossible to determine with other forms of testing, such as the responsiveness of a user interface to an end user or the time required to perform various tasks. Finally, the universality of the scripting language and the ability of the control application to recover from certain unexpected situations allows greater flexibility for, for example, different printer models or hardware / firmware versions, allowing fewer test scripts to achieve the same level of testing coverage that was previously possible.

[0051] It is desirable to achieve accurate time measurements with respect to finding the time required to accomplish various tasks, especially when multiple components (each with its own mechanism for keeping track of time) are involved in the test process. Rather than adjusting and using the system time of each component, it is advantageous to keep track of the difference between a reference time for the component and the local system time. By tracking such differences and establishing each measurement time from a local component synchronized to the same reference time, the time measurements can be kept consistent. Maintaining the time difference at the application level rather than at the system level is also advantageous in minimizing disruption of other processes running on the system.

[0052] Other objects, advantages and embodiments of various aspects of the present invention will be apparent to those skilled in the art of the present invention and are within the scope of this description and the accompanying drawings. For example, and without limitation, structural or functional elements may be rearranged and method steps may be rearranged as is consistent with the present invention. Similarly, the principles of the present invention may be applied to other implementations that are within the scope of the present invention, even if not specifically described herein. For example, the principles of the present invention are not limited to testing printers, but could be used to test any product having an embedded system.

Claims

[Claim 1] A system for testing a function of an embedded system, comprising a computing device including a low-power computing unit and a computing unit, the low power computing unit is communicatively connected to the control application interface, the sensor interface, and the robot interface, and consumes less than 25 watts of power; receiving from the sensor interface a plurality of sensor signals generated during functional testing of the embedded system; providing sensor data corresponding to the plurality of sensor signals via the control application interface; receiving a plurality of commands for the functional test via the control application interface; providing, via the robot interface, a plurality of instructions for operation of a robot handler corresponding to at least one of the plurality of commands for the functional test; the computing unit is communicatively connected to the control application interface, the image processing interface, and the database interface; receiving the sensor data from the control application interface and image data from the image processing interface corresponding to a plurality of images of the embedded system acquired during the functional test; receiving from the database interface a number of tests that can be performed on the function; The system provides a plurality of commands for the functional test via the control application interface.

Images