Evaluation device and evaluation method

Abstract

To reduce a load related to screen acquisition in automatic evaluation for screen displays on a device with multiple screens.SOLUTION: An evaluation device includes a storage unit, a determination unit, an execution unit, and an acquisition unit. The storage unit stores a test program in which at least one command is described, the command including a first command that simulates a user operation for causing a device with multiple screens to execute an operation according to the user operation in a pseudo manner. The determination unit determines, from among the multiple screens, a relevant screen that is relevant to a command in an execution phase, out of all commands described in the test program. The execution unit executes a functional test of the device by sequentially executing all the commands. The acquisition unit captures a screen display on each of the multiple screens at a frame rate based on a result of the determination.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to an evaluation apparatus and an evaluation method.

Background Art

[0002] Conventionally, an automatic evaluation apparatus that automatically performs a functional test based on a pre-created test scenario has been known to evaluate screen displays and screen transitions in a display device (device).

[0003] For example, Patent Document 1 discloses a technique for testing a display screen by comparing and determining a screen display displayed by a system to be tested with pre-generated test data.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, when the device to be tested is a device having a plurality of screens, the processing load related to the acquisition of the screen display and the data volume to be acquired increase according to the number of screens. For this reason, when the number of screens of the device to be tested increases, there are problems such that a large amount of resources are required for the processing and recording related to the acquisition of the screens, or the operation becomes slow due to a high load on the resources.

[0006] One of the objectives of the present disclosure is to reduce the load related to screen acquisition in the automatic evaluation of screen displays in a device having a plurality of screens.

Means for Solving the Problems

[0007] The evaluation device according to the present disclosure includes a storage unit, a determination unit, an execution unit, and an acquisition unit. The storage unit stores a test program in which at least one command is described, including a first command that emulates a user operation for causing a device having a plurality of screens to pseudo-execute an operation according to a user operation. The determination unit determines a relevant screen among the plurality of screens, which is relevant to a command in an execution stage among all the commands described in the test program. The execution unit executes a function test of the device by sequentially executing all the commands. The acquisition unit acquires each of the screen displays on the plurality of screens at a frame rate based on the result of the determination.

Effect of the Invention

[0008] According to the present disclosure, in the automatic evaluation of the screen display in a device having a plurality of screens, the load related to screen acquisition can be reduced.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Embodiments for Carrying Out the Invention

[0010] Hereinafter, embodiments of the evaluation system, evaluation apparatus, evaluation method, program, and recording medium according to the present disclosure will be described with reference to the drawings.

[0011] In the description of the present disclosure, for components having the same or substantially the same functions as those described above with respect to the already presented figures, the same reference numerals may be given, and the description may be omitted as appropriate. Also, even when representing the same or substantially the same part, there are cases where the dimensions and ratios are represented differently in the drawings. Further, for example, from the viewpoint of ensuring the visibility of the drawings, only the main components are given reference numerals in the description of each drawing, and there are cases where components having the same or substantially the same functions as those described above in the already presented figures are not given reference numerals.

[0012] In the description of the present disclosure, components having the same or substantially the same functions may be described separately by adding alphanumeric characters to the end of the reference numerals. Alternatively, when not distinguishing a plurality of components having the same or substantially the same functions, they may be described integrally by omitting the alphanumeric characters attached to the end of the reference numerals.

[0013] FIG. 1 is a diagram showing an example of the configuration of the evaluation system 1 according to the embodiment. The evaluation system 1 according to the embodiment performs an automatic evaluation that automatically conducts an integrated functional test simulating a user's operation on a test target device 3 having a plurality of screens. As shown in FIG. 1, the evaluation system 1 includes a test target device 3 and an automatic evaluation device 5.

[0014] In the evaluation system 1 according to the embodiment, the test target device 3 and the automatic evaluation device 5 are communicably connected via, for example, a dedicated cable or the like. Note that the connection between the test target device 3 and the automatic evaluation device 5 is not limited to wired, and may be communicably connected wirelessly. Also, this communication may be performed via a telecommunication line such as the Internet.

[0015] The test target device 3 according to the embodiment is an example of a device to be automatically evaluated for screen display, that is, a device to be tested. The test target device 3 has a plurality of screens (display 96). When the test target device 3 is an in-vehicle device, the plurality of screens can be screens of various display devices such as in-vehicle infotainment (IVI), meters, HUD (Head Up Display), and electronic mirrors. FIG. 1 illustrates a first screen 31a, a second screen 31b, and a third screen 31c as the plurality of screens. Note that the number of screens is arbitrary and may be two or a plurality of four or more.

[0016] For example, the test target device 3 is an information processing device (in-vehicle device) that can be mounted on a vehicle. As an example, the test target device 3 may be an ECU (Electronic Control Unit) provided inside the vehicle or an OBU (On Board Unit). As an example, the test target device 3 may be a DCU (Domain Control Unit) such as a CDC (Cockpit Domain Controller) that integrates a plurality of ECUs. This CDC is configured to be able to execute each process such as IVI, meter control, display device control such as HUD (Head Up Display) and electronic mirrors, and ADAS (Advanced Driver-Assistance Systems). As an example, the test target device 3 may be an information presentation device that can be mounted on a vehicle, such as an external computer or a car navigation device installed near the dashboard of the vehicle.

[0017] Note that the test target device 3 may be configured integrally with an HMI (Human Machine Interface) or may be configured to be able to cooperate with an external HMI through communication.

[0018] The automatic evaluation device 5 according to the embodiment is an example of an evaluation device that executes a functional test for automatically evaluating the screen display on a device to be tested. For example, the automatic evaluation device 5 evaluates the display and transition of the screen of the device 3 to be tested according to the user's operation by executing a test program 511 (see FIG. 3).

[0019] For example, the automatic evaluation device 5 is an information processing device such as a PC (Personal Computer). Note that a part of the automatic evaluation device 5 may be realized by at least one server device provided on a network such as the Internet or a LAN (Local Area Network), or by the device 3 to be tested. Further, the automatic evaluation device 5 may be at least one server device constructed on a network. Further, the automatic evaluation device 5 may be a device incorporated in the device 3 to be tested.

[0020] FIG. 2 is a diagram showing an example of the hardware configuration of each device of the evaluation system 1 according to the embodiment. The device 3 to be tested and the automatic evaluation device 5 each have a processor 91, a main storage device 92, an auxiliary storage device 93, a communication interface 94, an input interface 95, and a display 96. The processor 91, the main storage device 92, the auxiliary storage device 93, the communication interface 94, the input interface 95, and the display 96 are communicably connected by, for example, an internal bus 99.

[0021] Note that the automatic evaluation device 5 may not have the input interface 95 and the display 96.

[0022] The processor 91 controls the overall operation in each device. As the processor 91, various processors such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), and an FPGA (Field Programmable Gate Array) can be appropriately used.

[0023] The main memory device 92 stores data that is temporarily being worked on in each device. As the main memory device 92, for example, a RAM (Random Access Memory) can be used.

[0024] The auxiliary storage device 93 stores various types of data and programs used in each device. As the auxiliary storage device 93, various storage media and storage devices such as a ROM (Read Only Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), and Flash memory can be appropriately used.

[0025] The communication interface 94 is a circuit for communicating with the outside in each device. For example, the communication interface 94 is a circuit for communicating with each other between the device under test 3 and the automatic evaluation device 5. As the communication interface 94, communication circuits for wired communication, communication circuits for wireless communication, and combinations thereof can be appropriately used. As the communication circuit for wireless communication, communication circuits corresponding to various standards such as 3G, 4G, 5G, Wi-Fi (registered trademark), and Bluetooth (registered trademark) can be appropriately used.

[0026] The input interface 95 is an example of an HMI provided in each device. The input interface 95 acquires a user's input operation in each device. As the input interface 95, input devices such as a keyboard and a touch panel can be appropriately used.

[0027] The display 96 is an example of the HMI provided in each device. The display 96 presents a display screen to the user. As the display 96, a liquid crystal display (LCD), an organic EL (Electo-Luminescence) display, a projector, etc. can be appropriately used. The display 96 may be configured as a touch panel display. In this case, the touch panel of the display 96 is provided, for example, on the surface of the display 96 and outputs information according to the touched position. The touch panel of this display 96 is an example of the input interface 95 that acquires the user's operation input.

[0028] Here, while referring to FIG. 1 again, an example of the functional configuration of each device of the evaluation system 1 according to the embodiment will be described.

[0029] In the device under test 3, the processor 91 realizes the functions as the screen display unit 32, the control unit 33, and the command reception unit 34 by executing the program read from the auxiliary storage device 93 and loaded into the main storage device 92.

[0030] The screen display unit 32 controls the screen display on the plurality of screens (the first screen 31a, the second screen 31b, and the third screen 31c) of the device under test 3 according to the control of the control unit 33.

[0031] The control unit 33 controls the operations of each part of the device under test 3. As an example, the control unit 33 receives various user operations on the device under test 3, such as touch operations and button operations, via the input interface 95. As an example, the control unit 33 controls the operations of each part of the device under test 3 based on the received various user operations or the commands from the automatic evaluation device 5 received by the command reception unit 34. This command includes information indicating the control target and information indicating the control content (see FIG. 3).

[0032] For example, based on a command (first command) that simulates a user operation from the automatic evaluation device 5, the control unit 33 pseudo-executes an operation corresponding to the user operation, including controlling the screen display on a plurality of screens by the screen display unit 32. Here, the pseudo-execution of an operation corresponding to a user operation means controlling the operations of each part of the device under test 3 according to the first command in the same manner as when a user operation on the device under test 3 is received via the input interface 95. In other words, the pseudo-execution of an operation corresponding to a user operation means controlling the operations of each part of the device under test 3 by regarding that the user operation simulated by the first command is received via the input interface 95.

[0033] As an example, the user operations to be simulated may include operations related to playing music or video on the IVI, operations of the car navigation system, etc. Also, the operations of the device under test 3 corresponding to such user operations may include not only the screen display on the IVI but also screen displays such as the display of audio information and map information on the meter.

[0034] The command reception unit 34 receives commands from the automatic evaluation device 5 via the communication interface 94.

[0035] In the automatic evaluation device 5, the processor 91 realizes the functions as the storage unit 51, the test target screen determination unit 52, the test execution unit 53, and the video acquisition unit 54 by executing a program read from the auxiliary storage device 93 and loaded into the main storage device 92.

[0036] The storage unit 51 is realized by at least one of, for example, the main storage device 92 and the auxiliary storage device 93. The storage unit 51 stores the test program 511, the reference data 512, and the video data 513.

[0037] The test program 511 is a script that shows a test scenario (test procedure) for performing an integrated functional test that simulates a user's operation on the device under test 3 having a plurality of screens. FIG. 3 is a diagram showing an example of the test program 511 according to the embodiment. FIG. 3 illustrates a case where any one of the first screen 31a, the second screen 31b, and the third screen 31c is an IVI screen and any other screen is a meter screen. As shown in FIG. 3, the test program 511 describes a plurality of commands indicating the procedure of the functional test of the device under test 3.

[0038] In the example of FIG. 3, after executing a command 61a (first command) that causes the IVI to pseudo-execute an operation corresponding to a user's touch operation at a position (x = 100, y = 200) on the screen, a sleep command 63 (third command) that instructs a waiting time of 5 seconds is executed. The procedure is illustrated. Further, thereafter, after executing a command 61b (first command) that causes the IVI to pseudo-execute an operation corresponding to a user's touch operation at a position (x = 400, y = 300) on the screen, a sleep command 63 that instructs a waiting time of 5 seconds is executed. The procedure is illustrated. Further, thereafter, a screenshot (image) showing the screen display is acquired (captured) from a video showing the screen display of the IVI acquired at the set frame rate, and a command 65a (second command) that generates the comparison result between the captured IVI screen display and the reference data 512 as "result1" is executed. The procedure is illustrated. Further, thereafter, a screenshot (image) showing the screen display is acquired (captured) from a video showing the screen display of the meter acquired at the set frame rate, and a command 65b (second command) that generates the comparison result between the captured meter screen display and the image of the reference data 512 as "result2" is executed. The procedure is illustrated.

[0039] As described above, the test program 511 describes at least one command including a first command that emulates a user operation for causing a device having a plurality of screens to pseudo-execute an operation according to a user operation. The test program 511 also describes a second command for comparing the acquired screen display of the device under test 3 with a reference image of reference data. The test program 511 also describes a waiting time.

[0040] Note that the test program 511 may be supplied from outside the automatic evaluation device 5 or may be input or edited via the input interface 95.

[0041] The reference data 512 is data of a reference image showing a screen display acquired in advance for each of a plurality of screens of the device under test 3. As an example, the reference data 512 includes, as a reference image for evaluating a screen display, a screen display accompanying an operation of the device under test 3 according to the user operation, which is acquired when the user operation emulated by the first command described in the test program 511 is actually performed.

[0042] The video data 513 is data of a screen display, that is, video, continuously acquired at a frame rate set in the functional test for each of a plurality of screens of the device under test 3.

[0043] The test target screen determination unit 52 determines which of the plurality of screens of the device under test 3 is related to the command executed by the test execution unit 53 among all the commands described in the test program 511. In other words, the test target screen determination unit 52 determines a related screen among the plurality of screens of the device under test 3 to which the command at the execution stage among all the commands described in the test program 511 is related.

[0044] As an example, when a waiting time (sleep state) is described in the test program 511, the test target screen determination unit 52 determines which of the plurality of screens of the test target device 3 the next command to be executed is related to during the waiting time. That is, when a waiting time is described in the test program 511, the test target screen determination unit 52 determines the related screen to which the next command to be executed is related during the waiting time. According to the configuration that interprets the next command during the waiting time in this way, it is possible to reduce the processing delay caused by the determination of the target screen.

[0045] Here, the screen related to the command (related screen) includes at least one of the screen based on the first command and the screen based on the second command. The screen based on the first command includes the screen of the control target operated by the execution of the first command, the screen whose display is affected by the execution of the first command, or the screen for confirming the influence on the screen display accompanying the execution of the first command. Also, the screen based on the second command includes the screen that performs the target screen display to be compared with the reference data 512 by the execution of the second command.

[0046] As an example, the test target screen determination unit 52 determines the screen related to the command based on the information indicating the device or screen described in the command. For example, based on the description of "device='IVI'" in the command 61a, the test target screen determination unit 52 determines that the command 61a is related to IVI. For example, based on the description of "device='Meter'" in the command 65b, the test target screen determination unit 52 determines that the command 65b is related to the meter.

[0047] The test execution unit 53 executes a functional test of the device under test 3 by sequentially executing all the commands described in the test program 511. Here, for the test execution unit 53 to execute a command means to execute the process indicated by the command. As an example, the test execution unit 53 executes a first command that simulates user operations on a device having a plurality of screens. Specifically, the test execution unit 53 transmits the first command to the device under test 3, thereby causing the device under test 3 to pseudo-execute an operation corresponding to the user operation.

[0048] In addition, the test execution unit 53 evaluates the results of the functional test of the device under test 3. As an example, the test execution unit 53 executes a second command that compares the acquired screen display of the device under test 3 with a reference image of reference data. Note that including the evaluation of the results of the functional test, it may be referred to as a functional test.

[0049] The video acquisition unit 54 continuously acquires screen displays on a plurality of screens of the device under test 3. Also, the video acquisition unit 54 can independently change the frequency (frame rate) of continuously acquiring the screen display for each of the plurality of screens. The video acquisition unit 54 performs a frame rate adjustment process of setting the frame rate for acquiring the screen display based on the determination result of the target screen by the test target screen determination unit 52 before the process indicated by the command is performed by the test execution unit 53 according to which command among the plurality of screens of the device under test 3 is to be controlled. Note that the frame rate set in this frame rate adjustment process is not the frame rate of the video used for the screen display on each of the plurality of screens in the device under test 3, but the frame rate when acquiring the screen display from the signal line or from the appearance.

[0050] In this way, the video acquisition unit 54 acquires each of the screen displays on the multiple screens of the device under test 3 at a frame rate based on the determination result of the target screen by the test target screen determination unit 52. Then, the video acquisition unit 54 stores the acquired video of the screen display in the storage unit 51 as video data 513.

[0051] Note that the storage of the acquired video of the screen display in the storage unit 51 may be continuously performed while acquiring each of the screen displays on the multiple screens of the device under test 3 at the set frame rate, or may be stored when the second command for comparison, such as commands 65a and 65b, is executed.

[0052] As an example, among the multiple screens of the device under test 3, the video acquisition unit 54 acquires the screen display on the screen (related screen) determined by the test target screen determination unit 52 to be related to the command executed by the test execution unit 53 at the first frame rate, and acquires the screen displays on the other screens of the related screen at a second frame rate lower than the first frame rate. That is, while continuously acquiring the screen display (video), the video acquisition unit 54 reduces the acquisition frequency (frame rate) of the screens that are not under control. Alternatively, the video acquisition unit 54 increases the frame rate of the related screen determined to be related by the test target screen determination unit 52 and decreases the frame rate of the other screens.

[0053] Here, increasing the frame rate means setting the frame rate for acquiring the screen display of that screen to a high frame rate (first frame rate) for the target screen. That is, increasing the frame rate includes increasing the frame rate to the first frame rate when the frame rate set at that time is not the first frame rate. Also, increasing the frame rate includes maintaining the first frame rate when the frame rate set at that time is the first frame rate.

[0054] Here, reducing the frame rate means setting the frame rate for obtaining the screen display of that screen to a low frame rate (second frame rate) for non-target screens, which is lower than the first frame rate. That is, reducing the frame rate includes reducing the frame rate to the second frame rate when the frame rate set at that time is not the second frame rate. Also, reducing the frame rate includes maintaining the second frame rate when the frame rate set at that time is the second frame rate. At this time, the lower the second frame rate, the greater the reduction in the load related to screen acquisition can be achieved.

[0055] As an example, the video acquisition unit 54 may set the second frame rate to zero and interrupt the acquisition of the screen display. That is, the video acquisition unit 54 may acquire the screen display at the first frame rate for the screen (related screen) determined to be related to the executed command among the plurality of screens of the device under test 3, and interrupt the acquisition of the screen display for other screens of the related screen. Thereby, the load related to screen acquisition can be further reduced.

[0056] As an example, the video acquisition unit 54 may vary the values of the first frame rate and / or the second frame rate for each screen.

[0057] As an example, the video acquisition unit 54 may determine whether to set the second frame rate to zero for each screen or for each state of the screen. For example, for a screen whose screen display can be turned on / off by a user operation such as IVI, the video acquisition unit 54 may set the second frame rate to zero. For example, for a screen in a state where the screen display is turned off, the video acquisition unit 54 may set the second frame rate to zero. For example, the video acquisition unit 54 may determine whether to set the second frame rate to zero based on information indicating candidate (non-target) screens that is predefined and stored in the storage unit 51. This information indicating the candidate screens may be preset, for example, by a person who plans / executes a functional test according to whether it is necessary to evaluate the screen display, for example. Of course, this information indicating the candidate screens may be described in the test program 511.

[0058] Here, with reference to the drawings, the screen acquisition according to the embodiment will be described more specifically. FIG. 4 is a diagram for explaining the control of the screen acquisition according to the embodiment. FIG. 4 illustrates a case where each of the first screen 31a, the second screen 31b, and the third screen 31c is a screen of any one of IVI, HUD, and a meter. Also, in the example of FIG. 4, an example is illustrated in which the first frame rate of a screen (related screen) determined to be related by the test target screen determination unit 52 is 30 fps, and the second frame rate of other screens is 5 fps. Also, FIG. 4 illustrates the control of the screen acquisition performed in a state where each screen of IVI, HUD, and the meter is set to the first frame rate (30 fps).

[0059] When the video acquisition unit 54 determines that the IVI is the target screen based on the first command 61 which is a control instruction to the IVI, while maintaining the frame rate for acquiring the screen display of the IVI at 30 fps (the first frame rate), it reduces the frame rate for acquiring the screen displays of other screens (HUD and the meter) to 5 fps (the second frame rate).

[0060] After that, when the meter is determined to be the target screen (related screen) based on the first command 67 which is a control instruction to the meter, the video acquisition unit 54 increases the frame rate for acquiring the screen display of the meter to 30 fps (the first frame rate), while maintaining the frame rate for acquiring the screen display of the HUD (other screen) at 5 fps (the second frame rate) and decreasing the frame rate for acquiring the screen display of the IVI (other screen) to 5 fps (the second frame rate).

[0061] Next, the flow of the evaluation process executed by the evaluation system 1 configured as described above will be explained. FIG. 5 is a flowchart showing an example of the flow of the evaluation process executed in the automatic evaluation device 5 according to the embodiment.

[0062] The automatic evaluation device 5 starts a loop process (S102 to S107) that is executed as many times as the number of commands described in the test program 511 (S101).

[0063] First, the automatic evaluation device 5 determines, based on the description of the test program 511, which screen among the plurality of screens the command to be executed targets, that is, which screen is the related screen (S102). Then, the automatic evaluation device 5 starts a loop process (S104 to S106) that is executed as many times as the number of screens of the test target device 3 (S103).

[0064] When the command to be executed is for the related screen (S104: Yes), the automatic evaluation device 5 increases the frame rate, which is the frequency of acquiring the screen display of the related screen as a continuous image (video) (S105).

[0065] When the command to be executed is not for the related screen (S104: No), the automatic evaluation device 5 decreases the frame rate for acquiring the screen display of that screen (S106).

[0066] After the processing of S105 or S106, the automatic evaluation device 5 determines (S107) whether the repetitive processing (S104 to S106) has been executed for all the screens of the device under test 3. The flow in FIG. 5 returns to the processing of S104 if the repetitive processing has not been executed for all the screens, and proceeds to the processing of S108 after the execution.

[0067] After the repetitive processing has been executed for all the screens, the automatic evaluation device 5 determines (S108) whether the repetitive processing (S102 to S107) has been executed the number of times of the commands described in the test program 511. The flow in FIG. 5 returns to the processing of S102 if the repetitive processing has not been executed for all the commands, and ends after the execution.

[0068] As described above, the automatic evaluation device 5 according to the embodiment is configured to read, in the function test of the device under test 3 having a plurality of screens, the screen (related screen) to which a command is to be transmitted at the beginning of the command processing, and set the frame rate for acquiring the screen display of the related screen of the target to a predetermined first frame rate. Further, the automatic evaluation device 5 is configured to set the frame rate for acquiring the screen display of other screens to a second frame rate lower than the predetermined frame rate.

[0069] According to this configuration, in the function test for continuously acquiring the screens of the device under test 3 having a plurality of screens, while maintaining a high frame rate of the related screen that is the control target in the test process to suppress the influence on the evaluation result, the load related to screen acquisition to the automatic evaluation device 5 can be reduced.

[0070] In addition, if the load related to screen acquisition can be reduced, it is possible to suppress the situation where the load exceeds the resources and the operation of the automatic evaluation device 5 becomes slow. For example, it is possible to suppress the occurrence of processes such as evacuating the video data 513 to a write destination such as the non-volatile auxiliary storage device 93 from the main storage device 92, and suppress the decrease in the processing speed. Suppressing the decrease in the processing speed contributes to the efficiency improvement of the functional test. Also, even when recording the screen during the test, it is possible to suppress the data amount of the video data 513 and reduce the storage capacity of the main storage device 92 and the auxiliary storage device 93. Thus, if the usage amount of the memory and storage can be reduced, even the automatic evaluation device 5 configured with low resources can perform automatic evaluation while acquiring the video of a plurality of screens.

[0071] (First Modification Example) Note that the analysis for determining the target screen based on the description of the test program 511 is not limited to being sequentially performed during the execution of the functional test, and may be performed for the whole of it before the functional test is executed, such as immediately before the execution. FIG. 6 is a flowchart showing another example of the flow of the evaluation process executed in the automatic evaluation device 5 according to the embodiment.

[0072] First, the automatic evaluation device 5 determines the related screens to be targeted by the entire test program 511 (S201). After that, the automatic evaluation device 5 starts the repetitive process (S203 to S204) to be executed for the number of screens of the test target device 3 (S202).

[0073] When the command never becomes the target screen in the entire test program 511 (S203: No), the automatic evaluation device 5 reduces the frame rate for acquiring the screen display of that screen (S204).

[0074] When the command targets a relevant screen even once in the entire test program 511 (S203: Yes), or after the process of S204, the automatic evaluation device 5 determines whether repetitive processes (S203 - S204) have been executed for all the screens of the device under test 3 (S205). The flow in Figure 6 returns to the process of S203 if the repetitive processes have not been executed for all the screens, and proceeds to the process of S206 after being executed.

[0075] After the processes immediately before execution (S201 - S205), the automatic evaluation device 5 executes the processes after the start of execution (S206). The processes after the start of execution are, for example, the processes of S101 - S108 in Figure 5.

[0076] In this way, the automatic evaluation device 5 may determine, at a stage before the functional test is executed (started), such as immediately before the execution of the functional test, which of the multiple screens of the device under test 3 each command described in the test program 511 is related to. In other words, the automatic evaluation device 5 may determine, before the functional test is executed, the relevant screen among the multiple screens of the device under test 3 to which any one of all the commands to be executed in the functional test is related.

[0077] According to this configuration, for screens that are not targeted throughout the functional test and for which it is never determined that the execution of a command is relevant, the frame rate can be set to a low value at the start of the functional test, so that the load related to screen acquisition can be further reduced.

[0078] (Second Modification Example) Note that the adjustment of the frame rate during the execution of the functional test may be performed not only based on the relevant screen related to the command being executed at that time, i.e., the execution-stage command, but also further based on whether it is a relevant screen related to the command scheduled to be executed later. FIG. 7 is a flowchart showing another example of the flow of the evaluation process executed in the automatic evaluation apparatus 5 according to the embodiment. Here, mainly the differences from the processes (S101 to S108) after the start of execution in FIG. 5 will be described.

[0079] First, the automatic evaluation apparatus 5 executes the processes immediately before execution (S201 to S205) (S301). Since the processes immediately before execution have been described above with reference to FIG. 6, the description thereof will be omitted here.

[0080] After the processes immediately before execution, the automatic evaluation apparatus 5 executes the processes after the start of execution (S101 to S105, S302, S106 to S108).

[0081] When the command to be executed is a relevant screen to be targeted (S104: Yes), the automatic evaluation apparatus 5 increases the frame rate, which is the frequency of acquiring the screen display of that screen as a continuous image (video), in the same manner as the flow in FIG. 5 (S105).

[0082] Also, when the command to be executed is not a relevant screen to be targeted (S104: No), and when there is a plan to become a relevant screen to be targeted based on the command scheduled to be executed later (S302: Yes), the automatic evaluation apparatus 5 also increases the frame rate of acquiring the screen display of that screen (S105). That is, even when the relevant screen is not related to the command being executed at that time, the automatic evaluation apparatus 5 does not decrease the frame rate of acquiring the screen display for the relevant screen related to the execution of the subsequent command.

[0083] For example, regarding the test program 511 in FIG. 3, the meter is not the target screen (associated screen) related to commands 61a, 61b, and 65a. On the other hand, the meter is the target screen (associated screen) related to command 65b. In this case, even when the meter is not the associated screen when the automatic evaluation device 5 executes commands 61a, 61b, and 65a, since it is the associated screen related to the execution of the subsequent command 65b, the frame rate for acquiring the screen display of the meter is not decreased.

[0084] Also, when the command to be executed by the automatic evaluation device 5 is not the target associated screen (S104: No), and there is no plan to become the target associated screen based on the commands scheduled to be executed later (S302: No), similar to the flow in FIG. 5, the frame rate for acquiring the screen display of that screen is decreased (S106).

[0085] Note that in the flow of FIG. 7 as well, the second frame rate may be set to zero and the acquisition of the screen display may be interrupted.

[0086] Note that even when the processing immediately before execution is not performed, the frame rate during the execution of the functional test may be further adjusted based on whether it is the associated screen related to the commands scheduled to be executed later. For example, in addition to the screen related to the command to be executed by the automatic evaluation device 5, for the screens that can be grasped as being the targets in the range where the analysis has progressed during the waiting time, the first frame rate may also be set (maintained).

[0087] In this way, during the execution of the functional test, the automatic evaluation device 5 may acquire the screen display on the screen determined to be related to the commands scheduled to be executed later, not limited to the commands in the execution stage, at the first frame rate, and acquire the screen display on other screens at the second frame rate.

[0088] According to this configuration, for the screens that are planned to be targeted later, since the frame rate for obtaining the screen display cannot be lowered, even if there is a delay in the determination of the target screen or the switching of the frame rate, the screen display of the target screen can be obtained at the desired frame rate and appropriate evaluation can be carried out.

[0089] In addition, in each of the above-described embodiments, "determining whether it is A" may mean "determining that it is A", or "determining that it is not A", or "determining whether it is A or not".

[0090] The program executed by each device of the evaluation system 1 in each of the above-described embodiments may be provided by being recorded on a computer-readable recording medium such as a CD-ROM, FD, CD-R, DVD, etc. in an installable format or an executable format file.

[0091] Also, the program executed by each device of the evaluation system 1 in each of the above-described embodiments may be configured to be stored on a computer connected to a network such as the Internet and downloaded via the network for providing. Also, the program executed by each device of the evaluation system 1 in each of the above-described embodiments may be configured to be provided or distributed via a network such as the Internet.

[0092] Also, the program executed by each device of the evaluation system 1 in each of the above-described embodiments may be configured to be provided by being pre-embedded in a ROM or the like.

[0093] For example, the program executed by the test target device 3 in the above-described embodiment has a module configuration including the above-described respective functional units (screen display unit 32, control unit 33, and command reception unit 34). As actual hardware, the processor 91 reads the program from the auxiliary storage device 93 and executes it, so that the above-described respective functional units are loaded onto the main storage device 92, and the above-described respective functional units are generated on the main storage device 92.

[0094] For example, the program executed by the automatic evaluation device 5 of each of the above-described embodiments has a module configuration including the above-described functional units (storage unit 51, test target screen determination unit 52, test execution unit 53, and video acquisition unit 54). As actual hardware, the processor 91 reads the program from the auxiliary storage device 93 and executes it, so that the above functional units are loaded onto the main storage device 92, and the above functional units are generated on the main storage device 92.

[0095] According to at least one of the embodiments described above, in the automatic evaluation of the screen display in a device having a plurality of screens, the load related to screen acquisition can be reduced.

[0096] Although some embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the gist of the invention. These embodiments and their modifications are included in the scope and gist of the invention, as well as in the invention described in the claims and its equivalent scope.

[0097] (Supplementary Note) With the description of the above embodiments, the following technology is disclosed. (1) A storage unit that stores a test program in which at least one command is described, the test program including a first command that simulates a user operation for pseudo-executing an operation according to a user operation on a device having a plurality of screens; A determination unit that determines a relevant screen among the plurality of screens, the relevant screen being related to a command at an execution stage among all the commands described in the test program; An execution unit that executes a functional test of the device by sequentially executing all the commands; An acquisition unit that acquires each of the screen displays on the plurality of screens at a frame rate based on the result of the determination. An evaluation device. (2) The acquisition unit acquires the screen display on the related screen among the plurality of screens at a first frame rate, and acquires the screen display on other screens at a second frame rate lower than the first frame rate. The evaluation device according to (1) above. (3) The acquisition unit acquires the screen display on the related screen among the plurality of screens at a first frame rate, and interrupts the acquisition of the screen display on other screens. The evaluation device according to (1) or (2) above. (4) Before the function test is executed, the determination unit determines the related screen to which any one of all the commands executed in the function test of the device is related. The acquisition unit sets the frame rate for acquiring the screen display on other screens than the related screen among the plurality of screens to a second frame rate lower than the first frame rate for acquiring the screen display on the related screen. The evaluation device according to (1) above. (5) During the execution of the function test, the acquisition unit acquires the screen display on the screen determined to be related to at least one of the commands in the execution stage and the commands scheduled to be executed later among the related screens at a first frame rate, and acquires the screen display on other screens at a second frame rate lower than the first frame rate. The evaluation device according to (4) above. (6) During the execution of the function test, the acquisition unit acquires the screen display on the screen determined to be related to at least one of the commands in the execution stage and the commands scheduled to be executed later among the related screens at a first frame rate, and interrupts the acquisition of the screen display on other screens. The evaluation device according to (4) or (5) above. (7) When a waiting time is described in the test program, the determination unit determines the relevant screen related to the command to be executed next among all the commands during the waiting time, among the commands in the execution stage. The evaluation device according to any one of (1) to (6) above. (8) The storage unit further stores a reference image as a criterion for evaluating the screen display. The at least one command further includes a second command that compares the screen display acquired by the acquisition unit with the reference image. The execution unit evaluates the result of the function test by executing the second command. The evaluation device according to any one of (1) to (7) above. (9) The determination unit determines at least one of the screen of the control target operated by the execution of the first command and the screen of the comparison target compared by the execution of the second command among the plurality of screens as the relevant screen. The evaluation device according to (8) above. (10) Storing a test program including at least one command including a first command that simulates the user operation for pseudo-executing an operation according to the user operation on a device having a plurality of screens. Determining the relevant screen among the plurality of screens related to the command in the execution stage among all the commands described in the test program. Executing a function test of the device by sequentially executing all the commands. Obtaining each of the screen displays on the plurality of screens at a frame rate based on the result of the determination. Evaluation method. (11) Determining a relevant screen among the plurality of screens, to which a command in an execution stage among at least one command including a first command that emulates a user operation for pseudo-executing an operation according to the user operation on a device having a plurality of screens described in a test program is relevant; Executing a functional test of the device by sequentially executing all commands described in the test program; Obtaining each screen display on the plurality of screens at a frame rate based on the result of the determination, including: Evaluation method. (12) Determining a relevant screen among the plurality of screens, to which a command in an execution stage among at least one command including a first command that emulates a user operation for pseudo-executing an operation according to the user operation on a device having a plurality of screens described in a test program is relevant; Executing a functional test of the device by sequentially executing all commands described in the test program; Causing a computer to obtain each screen display on the plurality of screens at a frame rate based on the result of the determination; Program. (13) A program executed by a computer, which is a recording medium (Computer Program Product) on which the program described in (12) above is recorded.

Explanation of Signs

[0098] 1 Evaluation system 3 Device under test (Device) 31a First screen 31b Second screen 31c Third screen 32 Screen display unit 33 Control unit 34 Command reception unit 5 Automatic evaluation device (Evaluation device) 51 Storage unit 511 Test Program 512 Reference Data 513 Video Data 52 Test Target Screen Determination Unit (Determination Unit) 53 Test Execution Unit (Execution Unit) 54 Video Acquisition Unit (Acquisition Unit) 91 Processor 92 Main Memory Device 93 Auxiliary Memory Device 94 Communication Interface 95 Input Interface 96 Display

Claims

1. A storage unit that stores a test program in which at least one command is described, the test program including a first command that emulates a user operation for causing a device having a plurality of screens to pseudo-execute an operation according to a user operation; A determination unit that determines a relevant screen among the plurality of screens, the relevant screen being related to a command in an execution stage among all the commands described in the test program; An execution unit that executes a function test of the device by sequentially executing all the commands; An acquisition unit that acquires each of the screen displays on the plurality of screens at a frame rate based on the result of the determination. An evaluation device.

2. The acquisition unit acquires the screen display on the relevant screen among the plurality of screens at a first frame rate, and acquires the screen display on other screens at a second frame rate lower than the first frame rate. The evaluation device according to Claim 1.

3. The acquisition unit acquires the screen display on the relevant screen among the plurality of screens at a first frame rate, and interrupts the acquisition of the screen display on other screens. The evaluation device according to Claim 1.

4. Before the function test is executed, the determination unit determines the relevant screen related to any one of all the commands executed in the function test of the device, The acquisition unit sets the frame rate for acquiring the screen display on other screens than the relevant screen among the plurality of screens to a second frame rate lower than the first frame rate for acquiring the screen display on the relevant screen. The evaluation device according to Claim 1.

5. During the execution of the function test, the acquisition unit acquires the screen display on a screen determined to be related to at least one of the command in the execution stage and the commands scheduled to be executed later among the relevant screens at a first frame rate, and acquires the screen display on other screens at a second frame rate lower than the first frame rate. The evaluation device according to Claim 4.

6. During the execution of the function test, the acquisition unit acquires the screen display on a screen determined to be related to at least one of the command in the execution stage and the commands scheduled to be executed later among the relevant screens at a first frame rate, and interrupts the acquisition of the screen display on other screens. The evaluation device according to Claim 4.

7. When the determination unit determines that a waiting time is described in the test program, during the waiting time, the determination unit determines the related screen related to the command to be executed next among the commands in the execution stage among all the commands. The evaluation device according to claim 1.

8. The storage unit further stores a reference image as a criterion for evaluating the screen display. The at least one command further includes a second command that compares the screen display acquired by the acquisition unit with the reference image. The execution unit evaluates the result of the function test by executing the second command. The evaluation device according to any one of claims 1 to 7.

9. The determination unit determines that at least one of the screen of the control target operated by the execution of the first command and the screen of the comparison target compared by the execution of the second command among the plurality of screens is the related screen. The evaluation device according to claim 8.

10. Storing a test program including at least one command including a first command that emulates the user operation for pseudo-executing an operation according to the user operation on a device having a plurality of screens; Determining a related screen among the plurality of screens related to the command in the execution stage among all the commands described in the test program; Executing a function test of the device by sequentially executing all the commands; Obtaining each of the screen displays on the plurality of screens at a frame rate based on the result of the determination. Evaluation method.

Images