Method for inspecting display device, inspection device, and inspection program

Abstract

To verify a form change in a stage before a robot retreats after the robot of an inspection device touch-operates an icon on a screen.SOLUTION: A display device, which serves as a monitor device of a car navigation system and displays an image including touch-operable icons, is fixed to a jig of an inspection device in a state before attachment to a vehicle. The operator at the distal end of the robot arm touches the icon to be inspected (S6). When the touch operation is detected (S7), the display surface of the display apparatus is photographed by two cameras 5A and 5B arranged on the left and right sides (S8) before the robot arm is retracted (S9). After each distortion correction (S13) is performed, a composite image is generated so as to compensate for a blind spot region due to the operator (S14). The icon image after the touch operation is extracted from the composite image (S17) and compared with the correct answer information to determine whether or not the icon image is normal (S18).SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an inspection of a display device that displays an image including a touch-operable icon, and more particularly to an inspection technique for confirming that a predetermined change in form occurs after the icon is touched. [Background technology]

[0002] In recent automobiles, a relatively large display device is placed in the center of the dash panel in front of the driver's seat, and various information, including map displays from the car navigation system, is often displayed on this display device.The display device often employs a configuration that allows touch operation by the driver, such as a capacitive touch panel, and also serves as an input device for various input operations, such as setting a destination in the car navigation system or selecting songs on the audio system.

[0003] In such display devices, a touchable icon is displayed as part of the image, and when the driver touches this icon, a specified operation is performed. However, it is desirable to check, for example, using an automated inspection device before installing the product in a vehicle, whether the icon is displayed correctly and whether it operates correctly when touched.

[0004] Patent Document 1 discloses a technology for inspecting the operation of mobile phone operation buttons (i.e., physical switches). The inspection device described in Patent Document 1 includes a plunger unit arranged facing the mobile phone's operation panel, which is fixed facing upward; a camera arranged next to the plunger unit and facing the mobile phone's screen; and a computer that determines whether the displayed content is acceptable. The plunger unit has releases made of wires, the number of which corresponds to the number of operation buttons. The releases press the operation buttons to be inspected, and the camera captures the changes on the screen at that time, and evaluates the captured images. In addition, a second embodiment is disclosed in which a robotic release operates buttons displayed on a PDA screen consisting of a touch panel. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-290852 Summary of the Invention [Problem to be solved by the invention]

[0006] As described above, in conventional inspection devices or inspection methods, the release that presses the operation button gets into the screen of the inspection object that the camera is trying to capture, creating a blind spot due to the release. Therefore, the image of the screen captured immediately after the release presses the operation button contains a blind spot due to the blind spot, making it impossible to properly inspect changes in the screen. In other words, it is not possible to inspect changes in the screen before the release leaves the screen and retreats upward. [Means for solving the problem]

[0007] The present invention provides a method for inspecting a display device that displays an image including an icon that can be touched by a user, the method comprising: The robot is positioned so that the controller at the end of the robot arm operates the above icon. At least two cameras are arranged at an angle inclined with respect to the display surface of the display device so as to mutually compensate for blind spots caused by the operation devices; A touch operation is performed on the coordinate position of the icon to be inspected using the operation device; detecting the touch operation by at least one of a sensor included in the display device and a sensor included in the robot; When the touch operation is detected, the display surface is photographed by the at least two cameras; The images captured by each camera are combined to generate a composite image of the display surface. Based on this composite image, it is determined whether the image displayed on the display screen after the operation is normal. [Effects of the Invention]

[0008] According to this invention, it is possible to reliably determine whether the image displayed on the display surface is normal immediately after the operator at the tip of the robot arm touches an icon, i.e., before the operator is retracted. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a functional block diagram of an inspection device according to an embodiment of the present invention. [Figure 2] FIG. 2 is an explanatory diagram showing a display example of a display device to be inspected. [Figure 3] 10 is a flowchart showing the processing flow of an example of an inspection. [Figure 4] FIG. 1 is an explanatory diagram of a control at the end of a robot arm. DETAILED DESCRIPTION OF THE INVENTION

[0010] An embodiment of the present invention will be described in detail below with reference to the drawings. Fig. 1 is a functional block diagram of an inspection device 1 according to the embodiment. First, a display device 2, which is an object to be inspected by the inspection device 1, will be described.

[0011] The display device 2 to be inspected in one embodiment is a monitor device, abbreviated as IVI (In-Vehicle Infotainment), that doubles as an input device and is located in the center of the dash panel in front of the driver's seat of a vehicle. This display device 2 displays various information, such as a car navigation system map, various setting screens, and vehicle information. Its display surface is a capacitive touch panel, allowing touch operations to be used to set a destination in the car navigation system, select audio, and perform various setting operations. For input operations, various icons are displayed as part of the image displayed on the display surface, and functions assigned to the icons are activated by touch operations by a user (usually the driver). Note that, in the present invention, the term "touch operation" is not limited to a narrowly defined touch operation that requires physical contact of a finger with the display surface, but may also include a non-contact touch operation in which an icon or the like is operated by bringing a finger close to the display surface.

[0012] The display device 2 is inspected before it is mounted on a vehicle. The inspection device 1 is equipped with a jig (not shown) that fixes the display device 2 in an appropriate position, for example, with the display surface facing diagonally upward. The display device 2 is connected to an IVI control terminal of the inspection device 1 via a harness and is virtually controlled by the inspection device 1. In other words, the inspection device 1 controls the input and output of the display device 2 in the same way that an on-board computer controls the input and output of the display device 2.

[0013] FIG. 2 shows an example of a screen display on the display device 2. This is a simplified illustration of a screen when a car navigation system is displaying a map for ease of explanation. Most of the screen is occupied by a map display section 21, and the vehicle's position is indicated on the road network by a triangular mark 22. A plurality of touch-operable icons 23 are arranged at the bottom and on the left and right edges of the screen. Each of these icons 23 is assigned a function, and touching an icon 23 initiates the corresponding function. At the same time, the appearance (color, shape, etc.) of the icon 23 changes to indicate that it has been touched. The inspection device 1 sequentially touches the icons 23 to be inspected among the various screen displays that the display device 2 can display, and determines whether the image on the display surface, including the icon image, after the touch operation is normal.

[0014] As shown in Figure 1, the inspection device 1 is composed of at least two cameras 5 arranged facing the display surface of the display device 2 fixed via a jig, a robot 6 arranged in a position where icons 23 can be touched, an operation control unit 7 that controls these cameras 5 and robot 6 as well as the display device 2 during inspection, a database 8 that stores necessary information about the object to be inspected such as design value data and the characteristics of the icons 23, and an image comparison processing unit 9 that processes images acquired by the cameras 5 and performs comparison judgment.

[0015] In a preferred embodiment, the camera 5 consists of two cameras 5A and 5B, each of which has essentially the same configuration. To avoid interference with the robot 6 moving on the display surface of the display device 2, the two cameras 5A and 5B are positioned outside the space above the display surface of the display device 2 and are tilted relative to the display surface. For example, assuming that the display surface of the display device 2 has a rectangular shape elongated from side to side as shown in FIG. 2, the first camera 5A is positioned on the left side of the display surface and the second camera 5B is positioned on the right side of the display surface, with their optical axes tilted relative to the display surface of the display device 2. In a preferred embodiment, the cameras are positioned as symmetrically as possible across the display device 2 to be inspected. Each of the cameras 5A and 5B has a field of view that allows them to capture the entire display surface. Distortion caused by the optical axes being tilted relative to the display surface is corrected by the image comparison processor 9, as described below.

[0016] Robot 6 is a relatively small, vertically articulated robot, with its base located on the upper or lower side of Figure 2. A thin controller capable of touching the capacitive touch panel of display device 2 is provided at the tip of a robot arm that moves in front of the display surface of display device 2 (in other words, on the display surface). This controller moves in a direction perpendicular to the display surface in accordance with the movement of the robot arm at any position on the display surface, for example, a position corresponding to icon 23, allowing touch operation. Because the two cameras 5A and 5B are arranged at an inclined angle on either side of the display surface as described above, they can mutually compensate for blind spots caused by the controller when the controller is above the display surface.

[0017] 4, the control at the tip of the robot arm includes a cylindrical support portion 31 supported by the robot arm, a cylindrical shaft portion 32 having a smaller diameter than support portion 31, and a tip portion 33 having an even smaller diameter provided at the tip of shaft portion 32. Tip portion 33 touches and operates the capacitive touch panel of display device 2. By configuring the diameter to decrease in stages from support portion 31 to tip portion 33 in this way, the blind spots that appear in cameras 5A, 5B when the control is on the display surface, and therefore the blind spot areas that appear in each captured image, are reduced.

[0018] The operation control unit 7 and image comparison processing unit 9 are configured by a so-called general-purpose computer system together with a storage unit, database 8. Although not shown, this computer system is configured to include input devices such as a keyboard and a pointing device, and output devices such as an LCD monitor.

[0019] The operation control unit 7 sequentially performs an operation check test for the icons 23, which will be described later, on a large number of icons 23 in an order that follows a predetermined test scenario. That is, it controls the display of a specific image on the display device 2, the touch operation of the icons 23 by the robot 6, the photographing of the display surface by the camera 5, and so on, all in a linked manner. The image comparison processing unit 9 corrects distortion in the image photographed by the camera 5, extracts an image of the icon 23 to be inspected (i.e., the icon 23 that the robot 6 touched), and determines whether the icon image that transitioned in response to the touch operation is normal after the touch operation.

[0020] 3 is a flowchart showing the flow of the inspection process executed by the inspection device 1, and below, inspection by the inspection device 1 will be described in more detail in accordance with this flowchart. When the inspection process starts, first in step 1, the image comparison processing unit 9 is started, and in step 2, the image comparison processing unit 9 acquires information about the icon 23 to be inspected this time by referring to the database 8. The information about the icon 23 may include coordinate data indicating the position of the icon 23 based on the design value data, the form (color, shape, etc.) of the icon 23 in its initial state before a touch operation, the form (color, shape, etc.) of the icon 23 after a touch operation, etc.

[0021] Next, in step 3, the robot 6 and camera 5 are activated. Next, in step 4, data on the operation position of the touch operation by the robot 6 is acquired. In detail, the coordinate position on the display surface of the representative point (for example, the center of gravity position) of the icon 23 to be inspected is acquired based on the design value data stored in the database 8, and assuming that the display device 2 is correctly fixed to the jig of the inspection device 1, the coordinate position of the representative point of the icon 23 on this display surface is converted into three-dimensional coordinates in space where the tip of the operator of the robot 6 should contact, and this is determined as the operation position.

[0022] Next, in step 5, the display device 2 is started up. Upon this start-up, an image including an icon 23 of the inspection target is displayed on the display surface.

[0023] After the display device 2 is started, the robot 6 starts operating in step 6, moves the operator to the operation position obtained in step 4, and performs a touch operation on the target icon 23. This touch operation of the operator on the display surface is detected by a touch sensor, i.e., a capacitance sensor, provided on the display surface, which is a capacitive touch panel. If the operator of the robot 6 is equipped with a sensor for detecting touch operations, the touch operation may be detected by this sensor on the robot 6 side. For example, it is desirable that the sensor on the robot 6 side be able to detect the touch operation even if the capacitance sensor on the display surface of the display device 2 is malfunctioning. In step 7, a signal indicating that a touch operation has been performed on the target coordinate point is acquired from the display device 2. Although not shown in the flowchart, if the coordinate point detected as the touch operation is different from the target coordinate point (the representative point of the target icon 23), an error signal is output and the inspection is interrupted.

[0024] When the touch sensor detects that a touch operation has been performed on the correct coordinate point, the operation control unit 7 immediately thereafter (after the very short time required for the screen to change) causes the first camera 5A and the second camera 5B to capture images of the display surface (step 8). Here, in order to obtain an image of the display surface that transitions substantially simultaneously with the touch operation of the icon 23, the two cameras 5A and 5B each capture still images. At this stage, the robot arm has not yet retracted, and the operator is positioned on the display surface. After the image is captured in step 8, the robot 6 is instructed to retract the robot arm (step 9). This retraction operation is performed so that the tip of the robot arm, including the operator, is retracted to a position where it does not block the view between the cameras 5A and 5B and the display surface.

[0025] In parallel with the retraction of the robot arm, when the touch sensor of the display device 2 detects a touch operation, a timer starts counting time and waits for a predetermined time to elapse (step 10). This predetermined time is set in advance as the time required for the retraction of the robot arm plus an appropriate margin of time.

[0026] After a predetermined time has elapsed, the process proceeds from step 10 to step 11, where the camera 5 again photographs the display surface. This is to obtain an image of the display surface in a state where the robot arm and the operator have retracted from the field of view of the camera 5, and the image is photographed, for example, by either the first camera 5A or the second camera 5B, or by both. Alternatively, an image of the display surface after the robot arm and the operator have retracted may be photographed by a third camera. Note that the image after the robot arm and the operator have retracted is not the subject of the present invention, and will not be described in detail below.

[0027] In step 12, the two sets of image data captured by the first camera 5A and the second camera 5B in step 8 before the robot arm and the operator are retracted are transferred from the cameras 5A and 5B to the image comparison processor 9. The image comparison processor 9 then performs distortion correction on the captured image data (step 13). That is, since the images captured by the cameras 5A and 5B, which are positioned at an angle to the display surface, result in the rectangular display surface appearing trapezoidal, distortion correction is performed corresponding to each angle of inclination. This distortion correction allows the images captured by each of the two cameras 5A and 5B to accurately represent the rectangular shape of the display surface.

[0028] Then, in the next step 14, the two images are combined to generate a composite image of the display surface. When generating this composite image, the one of the two images with the least blind spot is first determined as the main image. That is, the two image data captured by the first camera 5A and the second camera 5B before the robot arm and the controller are retracted contain blind spots where the display surface is not captured due to the robot arm and the controller. Which of the images from the first camera 5A and the second camera 5B has a larger blind spot depends on the positions of the robot arm and the controller on the display surface. As previously mentioned, assuming that the first camera 5A is located on the left side of the display surface and the second camera 5B is located on the right side of the display surface, if the controller is located to the left of the display surface, for example, the blind spot in the image from the first camera 5A is relatively large and the blind spot in the image from the second camera 5B is relatively small. Therefore, the image from the second camera 5B is designated as the main image, and the image from the first camera 5A is designated as the sub-image. Conversely, when the controller is positioned to the right of the display surface, the blind spot area in the image from first camera 5A is relatively small, and the blind spot area in the image from second camera 5B is relatively large. Therefore, the image from first camera 5A is used as the main image, and the image from second camera 5B is used as the sub-image. Which image is used as the main image can be determined by comparing the size of the blind spot areas in the actually acquired images, but it can also be determined from the coordinate position of the controller, including the robot arm.

[0029] When combining two images, the main image is used preferentially, and the sub-image is used in the blind spot area of ​​the main image to fill in the blind spot area of ​​the main image, and the two images are combined. Therefore, the boundary between the two combined images is approximately V-shaped along the side edge of the blind spot area of ​​the main image. As is well known, when multiple images are combined, the accuracy of the image at the boundary decreases. According to the above method, compared to a form in which, for example, the left half of the combined image is an image from the first camera 5A and the right half is an image from the second camera 5B, creating a boundary in the center, one of the main images occupies a larger proportion, which is advantageous for confirming screen transitions associated with touch operations of the icon 23.

[0030] Once the generation of the composite image is complete, the process proceeds from step 14 to step 15, where image recognition is used to determine whether or not an icon 23 exists in the generated composite image, more specifically, whether or not any part that can be considered to be an icon 23 is included in the vicinity corresponding to the coordinate points of the inspected icon 23. If no part that can be considered to be an icon 23 exists, this indicates an abnormality, such as the screen turning into a so-called blue screen and displaying nothing, and the process proceeds from step 15 to step 20, where the result is determined to be "NG" and a determination result to that effect is output.

[0031] If there is a portion that is deemed to be some kind of icon 23, the process proceeds from step 15 to step 16, where an icon region surrounding the icon 23 is set for more detailed image recognition. Then, in step 17, image recognition is performed on this icon region, and an image of the icon 23 being inspected immediately after the touch operation is extracted.

[0032] Next, in step 18, the extracted icon image immediately after the touch operation is compared with so-called correct answer information to determine whether the extracted icon image is normal. If it is normal, the process proceeds from step 18 to step 19, where it is determined to be "OK," and a determination result to that effect is output. If it is not normal, the process proceeds from step 18 to step 20, where it is determined to be "NG," and a determination result to that effect is output. These determination results are displayed, for example, on a monitor attached to the inspection device and stored in database 8. The correct answer information may include color, shape, brightness distribution, representative point coordinates, etc., which indicate the form of icon 23 after the touch operation based on the design value data. If this information between the extracted icon image and the correct answer information matches or is sufficiently similar to each other, the icon image transitioned by the touch operation is determined to be normal.

[0033] Similarly, the image of the display surface captured in step 11 after the robot arm and the operator have been retracted undergoes distortion correction, determination of the presence or absence of an icon 23, setting of the icon area, extraction of the icon image, and determination of whether the icon image is normal. After the robot arm and the operator have been retracted, processing can be performed based on the image from one camera 5, and image synthesis is not required. Note that the correct answer information at the stage when the robot arm and the operator have been retracted may differ from the correct answer information immediately after the touch operation.

[0034] In this way, in the above embodiment, by using images from the two cameras 5A and 5B arranged so as to mutually compensate for blind spots caused by the robot arm and the operator, it is possible to inspect whether the screen transition immediately after touching the icon 23 is normal. For example, even if the screen immediately after the touch operation changes further before the robot arm and the operator are retracted, it is possible to inspect the screen before the robot arm and the operator are retracted.

[0035] Next, a second embodiment will be described, which uses a different method of distortion correction. In the above-described embodiment, the main image and the sub-image are each corrected so that the display surface appears vertical before being combined. However, in the second embodiment, the sub-image is first corrected to match the distortion caused by the tilt of the main image. For example, if the image from the first camera 5A is the main image and the image from the second camera 5B is the sub-image, the sub-image from the second camera 5B is corrected to match the distortion of the main image from the first camera 5A, which exhibits a trapezoidal shape due to the tilt. In other words, the sub-image is corrected from its initial trapezoidal shape to an inverted trapezoidal shape, matching the trapezoidal distortion when viewed from the viewpoint of the first camera 5A. These images are then combined by compensating for blind spots in the main image with the sub-image. This generates a composite image containing distortion as if viewed from the viewpoint of the first camera 5A. The distortion of this composite image corresponding to the tilt of the first camera 5A is then corrected to generate the final composite image.

[0036] Next, a third embodiment will be described in which the main image and the sub-image are interchanged and the determination is performed twice to improve the inspection accuracy.

[0037] As described above, a main image and a sub-image are determined depending on the size of the blind spot area, and a first determination is made by the processing of steps 13 to 20. For example, the image from the first camera 5A is the main image, the image from the second camera 5B is the sub-image, and the first determination is made using a composite image obtained by combining the image so that the blind spot area in the main image is filled in with the sub-image.

[0038] Next, the main image and the sub-image are swapped, and a second determination is similarly performed by the processes of steps 13 to 20. That is, the image from second camera 5B is used as the main image, the image from first camera 5A is used as the sub-image, and the second determination is performed using a composite image synthesized so as to fill in the blind spot areas in the main image with the sub-image.

[0039] If both the first and second judgments are judged to be normal (step 19), the final judgment is that the device is normal. If either the first or second judgment is judged to be abnormal (step 20), the final judgment is NG.

[0040] The two types of composite images generated by swapping the main image and sub-image as described above may result in different judgment results when compared with the correct information because the main image, which occupies the majority of the area, is captured by different cameras 5A and 5B, and the boundary positions of the composite images along the blind spot areas are different. By performing judgment twice in this way, higher inspection accuracy can be achieved.

[0041] Although one embodiment of the present invention has been described above, the present invention is not limited to the above embodiment and can be modified in various ways. For example, the display device to be inspected is not limited to the above-mentioned vehicle IVI, but can be applied to inspect various display devices. Furthermore, the display device is not limited to one using a capacitive touch panel, but may be of other types such as a pressure-sensitive (resistive) touch panel. Furthermore, the icons to be inspected are not limited to icons on a map screen such as those shown in FIG. 2, but may be any icons such as icons for setting input or character icons that make up a character palette.

[0042] Furthermore, in the above embodiment, an example in which two cameras 5A and 5B are provided has been described, but it is also possible to perform inspection using three or more cameras. [Explanation of symbols]

[0043] 1...Inspection equipment 2...Display device 5. Camera 6. Robot 7...Operation control section 8. Database 9...Image comparison processing section 21...Map display section 23...Icon

Claims

1. 1. A method for inspecting a display device that displays an image including an icon that can be touched by a user, comprising: The robot is positioned so that the controller at the end of the robot arm operates the above icon. At least two cameras are arranged at an angle inclined with respect to the display surface of the display device so as to mutually compensate for blind spots caused by the operation devices; A touch operation is performed on the coordinate position of the icon to be inspected using the operation device; detecting the touch operation by at least one of a sensor included in the display device and a sensor included in the robot; When the touch operation is detected, the display surface is photographed by the at least two cameras; The images captured by each camera are combined to generate a composite image of the display surface. Based on this composite image, it is determined whether the image displayed on the display screen after the operation is normal. A method for inspecting a display device.

2. The images from the multiple cameras are corrected for distortions caused by the tilts of the respective images and then combined to generate the composite image. The method for inspecting a display device according to claim 1 .

3. Select one camera image as the main image, The images from other cameras are corrected to match the distortion caused by the tilt of the main image. The main image and the image from the other camera are combined, and distortion caused by tilt is corrected to generate the combined image. The method for inspecting a display device according to claim 1 .

4. The image of the camera with the least blind spot area on the display screen is selected as the main image, The blind spot area in the main image is filled in with an image from another camera to generate the composite image. The method for inspecting a display device according to claim 1 .

5. selecting an image from a first camera as a first main image, and filling in a blind spot area in the first main image due to the blind spot with an image from another camera to generate a first composite image; determining whether the image on the display screen after the operation is normal based on the first composite image; selecting an image from another second camera as a second main image, and filling in a blind spot area in the second main image due to the blind spot with an image from another camera to generate a second composite image; making a second determination as to whether the image on the display screen after the operation is normal based on the second composite image; The method for inspecting a display device according to claim 1 .

6. the display device is a monitor device for a car navigation system that displays a map of the car navigation system and allows input by touch operation, Before installation on the vehicle, the display device is fixed to a jig and inspected. The method for inspecting a display device according to claim 1 .

7. comparing the shape of the icon on the display surface immediately after the touch operation based on design value data with the icon image in the composite image to determine whether the icon image is normal; The method for inspecting a display device according to claim 1 .

8. An inspection device for a display device that displays an image including an icon that can be touched by a user, a robot arranged so that an operator at the tip of the robot arm operates the icon on a display surface of the display device fixed via a jig before installation; at least two cameras disposed at an inclined angle with respect to a display surface of the display device so as to mutually compensate for blind spots caused by the operation devices; a controller for controlling the display device, the camera, and the robot; Equipped with The above controller is A touch operation is performed on the coordinate position of the icon to be inspected using the operation device; detecting the touch operation by at least one of a sensor included in the display device and a sensor included in the robot; When the touch operation is detected, the display surface is photographed by the at least two cameras; The images captured by each camera are combined to generate a composite image of the display surface. Based on this composite image, it is determined whether the image displayed on the display screen after the operation is normal. Display device inspection equipment.

9. A display device inspection program that causes a computer of an inspection device to execute the display device inspection method according to claim 1.

Images