Image processing system and image processing method

Abstract

To accurately detect an object person's point of fixation on an observation object whose position and angle change.SOLUTION: An image processing system 1 comprises: an optical system 3 for object person detection for acquiring a face image necessary for detection of the angle and position of the line of sight of an object person Sp; an optical system 5 for object detection for acquiring an object image necessary for detection of the arrangement angle and position of an observation object So that is observed by the object person Sp; and a computer 9 that executes image processing on the face image and the object image. The computer 9 specifies the three-dimensional coordinates of a marker on the observation object So based on the object image, calculates the arrangement angle and position of the observation object So based on the three-dimensional coordinates of the marker; calculates the angle and position of the line of sight of the object person Sp based on the face image, and detects the object person Sp's point of fixation on a specific surface on the observation object So based on the angle and position of the line of sight and the arrangement angle and position of the observation object So.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing system and an image processing method for processing an image. [Background technology]

[0002] In recent years, face image processing devices that process face images captured using video cameras are becoming more and more popular (see, for example, Patent Documents 1 and 2 below). For example, such face image processing devices process face images to determine the three-dimensional coordinates of the pupils and the angle of the line of sight relative to the line connecting the camera to the center of the pupils, and detect the line of sight based on these. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 4500992 [Patent Document 2] Patent No. 4517049 Summary of the Invention [Problem to be solved by the invention]

[0004] In recent product manufacturing sites, product inspection is performed by inspectors or other subjects holding the product in their hands and visually inspecting it. Even in such a scene, the starting point and direction of the subject's gaze can be detected by using the devices described in Patent Documents 1 and 2 above. However, during product inspection, the position and angle of the observed object such as a product may constantly change, and even in such a case, it is desirable to detect the subject's gaze point on the observed object and output the detection information. Such detection information can be used to improve the inspection technique of inspectors or to improve the inspection efficiency of inspectors.

[0005] The present invention has been made in consideration of the above-mentioned problems, and aims to provide an image processing system and an image processing method that are capable of accurately detecting a subject's point of gaze on an object being observed whose position and angle change. [Means for solving the problem]

[0006] In order to solve the above problems, an image processing system according to one embodiment of the present invention comprises a subject detection optical system including a camera that captures an image of a subject's face and acquires a facial image necessary for detecting the angle and position of the subject's gaze, an object detection optical system including a camera that captures an image of an observed object being observed by the subject and acquires an object image necessary for detecting the placement angle and position of the observed object, and a calculation unit that performs image processing on the facial image and the object image, wherein the calculation unit identifies a three-dimensional position of a marker on the observed object based on the object image, calculates the placement angle and position of the observed object based on the three-dimensional position of the marker, calculates the subject's gaze angle and position based on the facial image, and detects the subject's gaze point on a specific surface on the observed object based on the gaze angle and position and the placement angle and position of the observed object.

[0007] Alternatively, an image processing method according to another aspect of the present invention is an image processing method executed by an image processing system, comprising the steps of: capturing an image of a subject's face to obtain a facial image necessary for detecting the angle and position of the subject's gaze; capturing an image of an observed object being observed by the subject to obtain an object image necessary for detecting the placement angle and position of the observed object; and performing image processing on the facial image and the object image, in which the step of performing image processing includes identifying a three-dimensional position of a marker on the observed object based on the object image, calculating the placement angle and position of the observed object based on the three-dimensional position of the marker, calculating the subject's gaze angle and position based on the facial image, and detecting the subject's gaze point on a specific surface on the observed object based on the gaze angle and position and the placement angle and position of the observed object.

[0008] In the image processing system or image processing method of the above-mentioned form, a face image of a subject is acquired, and an object image, which is an image of an object to be observed, is acquired. Then, the arrangement angle and position of the object to be observed are calculated using the three-dimensional positions of the markers on the object to be observed detected from the object image, and the angle and position of the line of sight of the subject are calculated from the face image, and the gaze point of the subject on the object to be observed is detected based on them. This makes it possible to accurately detect the gaze point of the subject on the object to be observed even when the position and angle of the object to be observed changes, such as when visually inspecting the object to be observed.

[0009] Here, the calculation unit may specify the three-dimensional positions of three or more markers on the observed object, and calculate the arrangement angle and position of the observed object based on the three-dimensional positions of the three or more markers. In this case, the arrangement angle and position of the observed object can be calculated with higher accuracy by using the positions of three or more independent markers, and the subject's gaze point on the observed object can be detected with higher accuracy.

[0010] In addition, a support member that integrally supports the subject detection optical system and the object detection optical system may be further provided. In such a configuration, it is possible to stabilize the relationship between the subject's line of sight detected using the subject detection optical system and the position of the observed object detected using the object detection optical system, and it is possible to accurately detect the subject's gaze point on the observed object.

[0011] The apparatus may further include a flat plate member supported by the support member and serving as a reference when the subject grasps the observation object. In this case, the position of the observation object relative to the object detection optical system is stabilized, so that the arrangement angle and position of the observation object can be detected reliably. As a result, the subject's gaze point on the observation object can be detected more reliably.

[0012] The calculation unit may also determine an equation of a specific surface in a three-dimensional coordinate system based on the arrangement angle and position of the observed object, and calculate the gaze point based on the equation and a vector representing the line of sight. In this way, the arrangement of the specific surface on the observed object can be specified by the equation, so that the gaze point of the subject on the specific surface on the observed object can be calculated with high accuracy.

[0013] The calculation unit may also determine an equation for each of the multiple divided surfaces of the specific surface based on the arrangement angle and position of the observed object, search for a surface having an intersection with the vector from among the multiple surfaces using the equations for the multiple surfaces, and determine the intersection between the searched surface and the vector as the gaze point. With this configuration, even when an observed object has a complex shape, the arrangement of the specific surface on the observed object can be specified by an equation for each of the multiple divided surfaces, so that the gaze point of the subject on the specific surface on the observed object can be calculated with high accuracy.

[0014] The calculation unit may store the positional relationship of the multiple true markers in advance, narrow down the three-dimensional positions of the multiple identified candidate markers to the multiple true markers, and calculate the arrangement angle and position of the observed object based on the narrowed-down three-dimensional positions of the multiple true markers. In this case, when multiple markers including a false marker are detected from the object image, the markers can be narrowed down to the true marker. As a result, the arrangement of the observed object can be detected without error, and the gaze point of the subject on a specific surface on the observed object can be detected without error.

[0015] The calculation unit may specify a three-dimensional position of the calibration target based on an image obtained in advance by the object detection optical system, and perform a calibration process in advance to calculate the angle and position of the line of sight based on the face image and the three-dimensional position of the calibration target. In this case, the detection accuracy of the subject's gaze point can be improved, and the subject's gaze point on a specific surface of the observed object can be detected with high accuracy. Effect of the Invention

[0016] According to the present invention, it is possible to accurately detect the subject's gaze point on an observation object whose position and angle change. [Brief description of the drawings]

[0017] [Figure 1] 1 is a side view showing an image processing system according to an embodiment. [Diagram 2] FIG. 2 is an external view of an observation object So that is a processing target of the image processing system of FIG. [Diagram 3] 1A to 1C are diagrams showing examples of the forms of markers MA, MB, and MC attached to an observation object So. [Figure 4] FIG. 13 is a diagram showing the form of an optotype displayed on an observation object So. [Diagram 5] FIG. 2 is a diagram illustrating a hardware configuration of a computer according to an embodiment. [Figure 6] 1 is a block diagram showing a functional configuration of a computer according to an embodiment. [Figure 7] 10 is a flowchart showing the procedure of a gaze point detection process performed by the image processing system according to the embodiment. [Figure 8] 2 is a side view of an observation object So that is a processing target of the image processing system according to the embodiment. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] Hereinafter, preferred embodiments of the image processing system according to the present invention will be described in detail with reference to the drawings. In the description of the drawings, the same or corresponding parts are designated by the same reference numerals, and duplicated description will be omitted.

[0019] First, referring to FIG. 1, the overall configuration of an image processing system 1 according to an embodiment will be described. FIG. 1 is a side view of the image processing system 1. The image processing system 1 is a processing system that detects the trajectory of the gaze point of an inspector on the surface of a plate-like product such as an electronic board when the inspector visually inspects the product and records the detection result. This image processing system 1 is a group of devices that capture images of a subject Sp who is an inspector and an observation object So which is a product to be inspected, to obtain a face image and an object image, and perform image processing on the face image and the object image. The image processing system 1 is configured to include an optical system 3 for detecting a subject, an optical system 5 for detecting an object, a table 7, a computer (calculation unit) 9, and a support member 11. Each component of the image processing system 1 will be described below.

[0020] The table 7 is a member having a structure in which a top plate portion 7a, which is a reference when the subject Sp grasps the observation object So during the examination, is supported by a support 7b. For example, the table 7 is provided to support the forearm of the subject Sp when the subject Sp grasps the observation object So, to stabilize the arrangement angle and position of the observation object So, and to cover and protect the subject detection optical system 3. The top plate portion 7a is preferably made of a light-transmitting material such as glass to prevent it from interfering with the imaging of the subject Sp with respect to the subject detection optical system 3. Note that, when the table 7 is provided mainly for the purpose of covering and protecting the subject detection optical system 3, the top plate portion 7a is shaped to extend toward the subject Sp side, and is made of a light-transmitting material such as glass. On the other hand, when the table 7 is provided primarily for the purpose of stabilizing the position and angle of the observation object So, the top plate portion 7a may be shaped so as not to impede imaging by the subject detection optical system 3 as shown in FIG. 1, and may be made of an opaque material.

[0021] The subject detection optical system 3 is fixed to one end side (left side in FIG. 1) of the top plate 7a of the table 7 under the table top plate 7a, and includes two cameras (stereo cameras) whose optical axes are adjusted to face the face of the subject Sp located away from the one end side of the top plate 7a. Each camera is calibrated beforehand after the subject detection optical system 3 is attached to the support member 11, and captures the face of the subject Sp, and acquires and outputs the face image. In this embodiment, the two cameras are CMOS (Complementary Metal Oxide Semiconductor) video cameras of the NTSC system, which is one of the interlace scan systems, and each camera is equipped with a near-infrared light source (for example, two types of light sources with emission wavelength bands of 850 nm and 940 nm) having a configuration described in Japanese Patent No. 4500992 in order to detect characteristic points such as the pupils, corneal reflection, or nostrils of the subject Sp. In response to a command from the computer 9, the subject detection optical system 3 turns on the near-infrared light source, captures an image of the face of the subject Sp, and outputs facial image data to the computer 9. These facial images are used by the computer 9 to detect the angle and position of the line of sight of the subject Sp.

[0022] The object detection optical system 5 includes two cameras (stereo cameras) that are fixed at a position away from the top plate 7a on the other end side (right side in FIG. 1) of the top plate 7a and have their optical axes adjusted to face the upper part of the top plate 7a. Each camera is calibrated beforehand after the object detection optical system 5 is attached to the support member 11, and captures an image of the observation object So held by the subject Sp from the opposite side of the surface facing the subject Sp, and acquires and outputs an object image. In this embodiment, the two cameras are CMOS video cameras of the NTSC system, which is one of the interlace scan systems, and each camera is equipped with a near-infrared light source (a light source with an emission wavelength band of 940 nm) to detect a marker on the observation object So. The object detection optical system 5 turns on the near-infrared light source and captures the observation object So in response to a command from the computer 9, and outputs data of the object image to the computer 9. These object images are used by the computer 9 to detect the arrangement angle and position of the observation object So.

[0023] The computer 9 is a data processing device that controls the subject detection optical system 3 and the object detection optical system 5, and performs image processing on the face images and object images acquired by the subject detection optical system 3 and the object detection optical system 5. The computer 9 may be constructed of a stationary or portable personal computer (PC), a workstation, or other types of computers. Alternatively, the computer 9 may be constructed by combining multiple computers of any type. When multiple computers are used, these computers may be connected via a communication network such as the Internet or an intranet.

[0024] The support member 11 is placed on a flat fixed surface such as a floor surface and integrally supports the table 7, the subject detection optical system 3, and the object detection optical system 5. That is, the support member 11 supports the subject detection optical system 3 at the lower part of one end side (left side in FIG. 1) of the top plate part 7a of the table 7 so that its optical axis faces diagonally upward and left in a direction away from the top plate part 7a. At this time, it is preferable that the support member 11 supports the subject detection optical system 3 so that the optical axis of the camera of the subject detection optical system 3 does not pass through the top plate part 7a. In addition, the support member 11 supports the object detection optical system 5 at a position away from the other end side (right side in FIG. 1) of the top plate part 7a of the table 7 so that its optical axis faces the upper part of the top plate part 7a. Here, the support member 11 may be configured so that its height from the fixed surface of the table 7 can be adjusted to match the height of the subject Sp, in which case the inclination of the optical axis of the camera of the subject detection optical system 3 can be adjusted to match the position of the face of the subject Sp, and the inclination of the optical axis of the camera of the object detection optical system 5 can be adjusted to match the height of the table 7.

[0025] When using the image processing system 1 with the above configuration, the subject Sp is positioned so that his / her face is within the shooting range of the subject detection optical system 3. Then, the subject Sp is made to hold the observation object So near the upper surface of the top plate portion 7a of the table 7 with his / her face facing upward, and to visually inspect the inspection surface, which is a specific surface of the observation object So.

[0026] FIG. 2 shows an example of the appearance of the observation object So, where (a) shows the appearance of the observation object So as viewed from the inspection target surface P1 side, and (b) shows the appearance of the observation object So as viewed from the surface P2 opposite the inspection target surface P1. The observation object So shown in FIG. 2 has an irregularly formed uneven surface overall, and the inspection target surface P1 is provided with an optotype M1 for calibrating the gaze detection process of the subject Sp. In addition, the surface P2 opposite the inspection target surface P1 is provided with three markers M1, which are used as a reference for calculating the arrangement angle and angle of the observation object So. A ,M B ,M CHere, a shape having an irregular concave-convex surface is given as an example of the shape of the observation object So, but the shape may be a flat plate shape having a flat surface, or a shape having a curved surface whose surface shape can be approximated by a mathematical formula, such as a cylindrical shape.

[0027] Marker M attached to the observation object So A ,M B ,M C As the marker, a marker of a color conspicuous on the object image such as white is preferable, and a light-emitting body such as an LED may be used as the marker, and when a light-emitting body is used, an LED with a low light-emitting part and low directivity is used so that the marker is reflected in a shape that enhances the center accuracy in the object image. In addition, when a near-infrared camera is used as the camera of the object detection optical system 5, a marker made of a retroreflective material is preferable, and when a color camera is used as the camera of the object detection optical system 5, a marker of a color conspicuous against the background (e.g., blue) may be used.

[0028] In parts (a) and (b) of FIG. 3, a marker M A ,M B ,M C In this way, the marker M A ,M B ,M C In the embodiment, an inner region of a predetermined shape (e.g., a rectangle) having a uniform color is surrounded by an outer region of a different color. In such an embodiment, as shown in part (a) of Fig. 3, it is preferable that the inner region has a color that easily reflects infrared rays (e.g., white) and the outer region has a color that easily absorbs infrared rays (e.g., black), and it is also preferable that the inner region has a color that easily absorbs infrared rays (e.g., black) and the outer region has a color that easily reflects infrared rays (e.g., white) as shown in part (b) of Fig. 3.

[0029] In addition, the calibration target M1 attached to the inspection target surface P1 of the observation target So may be a target of a fixed size, or may be a target M1 whose size changes once or repeatedly using a small liquid crystal display or the like. By using such a target M1, it is possible to make the target more likely to attract the subject's visual attention, so that the subject Sp can more accurately direct his or her gaze to the target M1. As a result, the calibration can be performed more accurately while reducing the burden on the subject Sp. FIG. 4 is a diagram showing an example of the mode of the target M1 displayed on the observation target So. In this way, it is preferable that the target M1 is displayed once or repeatedly in a mode in which it changes dynamically from a large size to a small size.

[0030] 5 is a block diagram showing a general hardware configuration of a computer 9. The computer 9 includes a CPU (processor) 101 that executes an operating system, application programs, etc., a main memory unit 102 consisting of ROM and RAM, an auxiliary memory unit 103 consisting of a hard disk or flash memory, etc., a communication control unit 104 consisting of a network card or a wireless communication module, input devices 105 such as a keyboard or a mouse, and output devices 106 such as a display or a printer.

[0031] Each functional element of computer 9, which will be described later, is realized by loading predetermined software onto CPU 101 or main memory unit 102, operating communication control unit 104, input device 105, output device 106, etc. under the control of CPU 101, and reading and writing data in main memory unit 102 or auxiliary memory unit 103. Data and databases necessary for processing are stored in main memory unit 102 or auxiliary memory unit 103.

[0032] 6, the computer 9 includes, as functional components, an imaging control unit 21, an image acquisition unit 23, a posture calculation unit 25, a line of sight calculation unit 27, and a gaze point detection unit 29. The functions of each component of the computer 9 will be described below.

[0033] After the gaze point detection process for the subject Sp is started, the imaging control unit 21 controls the imaging by the subject detection optical system 3 and the object detection optical system 5. At that time, as described in Japanese Patent No. 4500992, the imaging control unit 21 controls the lighting timing of the near-infrared light sources of two kinds of emission wavelengths attached to the stereo camera according to the imaging timing by the stereo camera of the subject detection optical system 3. By such control of the imaging control unit 21, in each camera constituting the stereo camera, an odd field image composed of odd-numbered horizontal pixel lines and an even field image composed of even-numbered horizontal pixel lines are obtained as one frame of face images obtained 30 times per second, and the odd field image and the even field image are generated by alternately photographing at intervals of 1 / 60 seconds as a face image in which the pupil is relatively bright (bright pupil image) and a face image in which the pupil is relatively dark (dark pupil image). However, the frame rate of the imaging by the stereo camera may be set appropriately.

[0034] The imaging control unit 21 also controls the imaging of the object detection optical system 5 while controlling the imaging of the object detection optical system 3. At this time, the imaging control unit 21 controls the timing of turning on the near-infrared light source attached to the stereo camera according to the imaging timing of the stereo camera of the object detection optical system 5. By such control of the imaging control unit 21, each camera constituting the stereo camera can obtain one frame of object image obtained simultaneously at 30 frames per second. Here, the frame rate of the imaging by the stereo camera can be appropriately set, but it is preferable to set it to the same as the frame rate of the imaging of the bright pupil image and the dark pupil image obtained by the object detection optical system 3. Note that when the subject Sp holds the observation object So on the table 7 when visually inspecting the observation object So, the surface P2 (part (b) in FIG. 2) of the observation object So opposite to the inspection object surface P1 faces the object detection optical system 5, so that the marker M is not included in the object image captured by the object detection optical system 5. A ,M B ,M C The image of the marker M is captured.A ,M B ,M C Prevents blurring of the image of the marker M A ,M B ,M C In order to accurately detect the position of the marker M on the image, it is preferable to set the exposure time of each camera of the object detection optical system 5 as short as possible (for example, 500 μsec). A ,M B ,M C When a light-emitting body such as an LED whose lighting can be controlled by the computer 9 is used as the marker M, the imaging control unit 21 synchronizes the lighting timing of the near-infrared light source with the lighting timing of the near-infrared light source. A ,M B ,M C It is preferable to control the LED so that the LED is turned on.

[0035] Image acquisition unit 23 acquires face images and object images in real time from the stereo cameras of subject detection optical system 3 and object detection optical system 5 via a wired or wireless data transfer interface. Then, image acquisition unit 23 passes each acquired face image to gaze calculation unit 27, and passes each acquired object image to posture calculation unit 25.

[0036] The orientation calculation unit 25 calculates the arrangement angle and position (orientation) of the observed object So based on a pair of object images simultaneously obtained by the stereo cameras of the object detection optical system 5. The orientation calculation function of the orientation calculation unit 25 for the observed object So will be described below. The orientation calculation unit 25 continuously executes the following orientation calculation process for each frame of the object images obtained continuously.

[0037] The orientation calculation unit 25 detects the position of the marker on each of the pair of object images. That is, the object image is converted into a binary image using a threshold value of pixel values, and the binary image is labeled. Then, the marker M A ,M B ,M CThe pose calculation unit 25 selects candidates for the reflected image of the marker from the labels by searching for images with an area and aspect ratio that are likely to be the reflected image of the marker M. A ,M B ,M C Alternatively, the object to be observed So can be selected by geometric shape pattern matching corresponding to the shape of the object to be observed (e.g., a rectangle), and geometric shape pattern matching is performed so as to be able to handle cases in which the posture of the object to be observed So rotates and the shape of the reflected image changes.

[0038] The orientation calculation unit 25 also obtains the center coordinates of the selected marker candidates on the images for the pair of object images, and performs a round-robin stereo matching on the center coordinates of the marker candidates on the pair of object images to obtain a predetermined three-dimensional coordinate system, i.e., a world coordinate system X W -Y W -Z W In addition, when the number of identified marker candidates is four or more, the orientation calculation unit 25 selects three true markers M from the three-dimensional coordinates of the identified four or more marker candidates. A ,M B ,M C At this time, the orientation calculation unit 25 narrows down the three-dimensional coordinates of the three markers M A ,M B ,M C The data showing the positional relationship between the markers M and M is stored, and the 3D coordinates of the candidate markers corresponding to the data are narrowed down as the 3D coordinates of the true marker. For example, A From marker M B Vector V pointing to A The data and marker M A From marker M C Vector V pointing to B The data of these vectors V A ,V B The 3D coordinates of the candidate marker corresponding to the true marker M A ,M B ,M CAs a result, the false markers M that are erroneously recognized as markers on the object of observation So due to image noise caused by reflection of near-infrared light or light from lighting provided in the inspection environment are eliminated. X (see part (b) of FIG. 2) can be excluded from the true markers.

[0039] In addition, the posture calculation unit 25 calculates the three narrowed down markers M A ,M B ,M C Using the three-dimensional coordinates of the world coordinate system X W -Y W -Z W The orientation of the observed object So in the world coordinate system X W -Y W -Z W In the object coordinate system X h -Y h -Z h That is, the orientation calculation unit 25 calculates the position and direction of the two markers M A ,M C The midpoint of the three-dimensional coordinates of the object coordinate system X h -Y h -Z h The coordinates of the center of O h and the three markers M A ,M B ,M C Let the normal vector of the plane passing through h The marker M is a directional vector that defines the axis. A From marker M B Vector V pointing to A And Marker M A From marker M C Vector V pointing to B The two mutually perpendicular vectors, X and X, are composed of the vectors X and X at a specified ratio. h Axis and Y h The orientation calculation unit 25 calculates the normal vector of the plane and the X axis as a direction vector. h Axis and Y h Based on the direction vector that defines the axis, W -Y W -Z WObject coordinate system X that defines the plane of the observed object So in h -Y h -Z h Y W axis,X W axis,Z W The rotation angles α, β, and γ around the axes are calculated. In this way, the orientation calculation unit 25 calculates the center coordinate O h and the rotation angles α, β, and γ are calculated as parameters specifying the orientation of the observed object So.

[0040] In addition, the posture calculation unit 25 determines the three-dimensional coordinates of the marker candidates in the world coordinate system X W -Y W -Z W Identify the three-dimensional coordinates of visual target M1 in.

[0041] The gaze calculation unit 27 performs gaze detection processing of the subject Sp for each face image obtained by the stereo camera of the subject detection optical system 3 in the same manner as the method described in Japanese Patent No. 4500992. At this time, the gaze calculation unit 27 performs gaze detection processing for each frame of the face image continuously obtained by the subject detection optical system 3.

[0042] In detail, the gaze calculation unit 27 performs stereo matching based on the position of the pupil detected on the face image, thereby calculating the gaze direction in the world coordinate system X W -Y W -Z W The gaze calculation unit 27 calculates the three-dimensional coordinates of the pupil position in the image. At this time, the gaze calculation unit 27 calculates a vector from the pinhole position to the pupil based on the face images of the two cameras constituting the stereo camera, and calculates the intersection of these two vectors, thereby calculating the three-dimensional coordinates of the pupil position. The gaze calculation unit 27 can also detect the closest point of the two vectors as the pupil position using the method described in International Publication WO2015 / 190204. In this case, the pupil position can be calculated even if there is no intersection of the two vectors due to an error.

[0043] Furthermore, the gaze calculation unit 27 calculates the gaze vector of the subject Sp using the method described in Japanese Patent No. 4500992 based on the pupil position calculated based on the face image and the positions of the pupil and corneal reflection on the face image detected based on the face image. Here, the gaze is a straight line connecting the pupil and the gaze point, and the gaze vector is expressed in the world coordinate system X W -Y W -Z W This is data that represents the position of the starting point of the line of sight and the angle (direction) of the line of sight.

[0044] Furthermore, the gaze calculation unit 27 may use the method described in International Publication WO2012 / 020760 to perform a process of calibrating coefficients used in calculating the gaze vector in advance, using the three-dimensional coordinates of the calibration target M1 (part (a) of FIG. 2) detected from the object image. However, this calibration process does not need to be performed every time the gaze detection process is performed, and it is sufficient to perform it at least once every time the subject Sp changes and every time the type of the observed object So changes.

[0045] For example, instead of the observation object So, a small liquid crystal display or the like (a display device having a configuration as shown in FIG. 8 described later) having a size equivalent to that of the observation object So may be used to display the visual target M1 in the display mode shown in FIG. 4, and the subject Sp may be asked to hold the small liquid crystal display. That is, the visual target M1 is first made to appear large on the small liquid crystal display, and the subject Sp is asked to focus on a small point at the center of the visual target M1, and then the visual target M1 is gradually reduced and displayed over a predetermined time (for example, about 1 second), and the subject Sp is asked to continue to focus on the center of the visual target M1. The gaze calculation unit 27 uses the face image obtained in such a state to perform a process of calibrating the coefficients used to calculate the gaze vector. After this calibration process, the gaze calculation unit 27 performs a process of calculating the gaze vector of the subject Sp in a state in which the subject Sp has changed his / her grip to the observation object So. In this way, by performing the calibration process while having the subject Sp focus on the visual target M1 displayed on a small display of the same size as the observed object So, the distance from the visual target M1 to the face of the subject Sp during the calibration process can be set to be equal to the distance from the observed object So to the face, thereby improving the accuracy of gaze point detection by the gaze point detection unit 29.

[0046] The gaze point detection unit 29 detects the gaze point of the subject Sp on the inspection target surface P1 of the observation target So using the posture data of the observation target So calculated by the posture calculation unit 25 and the gaze vector of the subject Sp calculated by the gaze calculation unit. Then, the gaze point detection unit 29 outputs the detected gaze point data to an output device 106 such as a display so that it can be viewed. Here, the gaze point data of the output target is expressed as the object coordinate system X h -Y h -Z h The gaze point detection unit 29 may store the detected gaze point data in a database in the auxiliary storage unit 103.

[0047] In detail, the gaze point detection unit 29 detects the object coordinate system X h -Y h -Z hUsing the position and direction of h -Y h -Z h This vector transformation converts the line of sight vector into a vector in the object coordinate system X h -Y h -Z h The world coordinate system X W -Y W -Z W The state of movement of the origin relative to the center coordinate O h The gaze point detection unit 29 then converts the gaze vector into the target coordinate system X h -Y h -Z h The three-dimensional coordinates of the gaze point of the subject Sp on the inspection surface P1 are detected by determining the intersection between the gaze vector and the inspection surface P1 based on an equation for specifying the shape and position of the inspection surface P1 in.

[0048] At this time, the gaze point detection unit 29 may detect the gaze point as follows. That is, the gaze point detection unit 29 detects the gaze point in the object coordinate system X h -Y h -Z h Based on the position and direction of the object surface P1 and the equation of the inspection object surface P1, the world coordinate system X W -Y W -Z W The equation of the inspection target surface P1 in the object coordinate system X h -Y h -Z h The world coordinate system X W -Y W -Z W The state of movement of the origin relative to the center coordinate O h The gaze point detection unit 29 performs coordinate transformation according to the movement state and the rotation state using the world coordinate system X W -Y W -Z WThe three-dimensional coordinates of the gaze point of the subject Sp on the inspection target surface P1 are detected by finding the intersection between the inspection target surface P1 and the gaze vector using the equation of the inspection target surface P1 in and the gaze vector of the subject Sp. For example, if the inspection target surface P1 has a cylindrical shape, the gaze point can be detected using an equation obtained by coordinate conversion of the equation representing the cylindrical shape. Furthermore, in the world coordinate system X W -Y W -Z W The 3D coordinates of the gaze point in the object coordinate system X h -Y h -Z h Convert to 3D coordinates in .

[0049] Here, when the inspection target surface P1 has an irregular surface shape having a concave and convex surface, an approximation equation for each surface divided into a mesh shape with a planar shape or a curved shape (for example, a spherical shape) may be stored in the auxiliary storage unit 103 in advance. In this case, the gaze point detection unit 29 obtains an equation for the inspection target surface P1 for each surface divided into a mesh shape, searches for a surface having an intersection with the gaze vector from among the divided surfaces, and calculates the intersection of the searched surface and the gaze vector as the gaze point. At this time, the gaze point detection unit 29 can extend the gaze vector in a direction from the viewpoint toward the tip at regular distance intervals, calculate the distance between the coordinates of the tip and the multiple surfaces, and search for a surface where the distance is equal to or less than a threshold value.

[0050] Next, the procedure of the gaze point detection process in the above-mentioned image processing system 1 will be described, and the image processing method according to this embodiment will be described in detail. FIG 7 is a flowchart showing the procedure of the gaze point detection process by the image processing system 1.

[0051] First, when the computer 9 receives an instruction input from the outside, the gaze point detection process is started, and the imaging control unit 21 of the computer 9 starts control of imaging of the subject Sp by the subject detection optical system 3 and control of imaging of the observed object So by the object detection optical system 5 (step S1). In response to this, the image acquisition unit 23 of the computer 9 continuously acquires face images and object images for each frame (step S2).

[0052] Thereafter, the orientation calculation unit 25 of the computer 9 executes an orientation calculation process for the object image for each frame, and obtains, as data representing the orientation of the observation object So, the object coordinate system X h -Y h -Z h Then, data indicating the moving and rotating states of the target person Sp is calculated (step S3). At the same time, the gaze calculation unit 27 of the computer 9 executes a gaze detection process for the face image for each frame, and calculates the gaze vector of the target person Sp (step S4).

[0053] Next, the gaze point detection unit 29 of the computer 9 calculates the object coordinate system X h -Y h -Z h Using the data showing the movement and rotation of the world coordinate system X W -Y W -Z W The equation of the inspection surface P1 of the observation object So in is calculated (step S5). Then, the gaze point detection unit 29 obtains an intersection between the line of sight vector of the subject Sp and the inspection surface P1, thereby detecting the three-dimensional coordinates of the gaze point of the subject Sp on the inspection surface P1 (step 6). The above processes from steps S3 to S6 are repeated for each frame of the face image and the object image captured at the same timing by the subject detection optical system 3 and the object detection optical system 5 (step S7). Finally, the gaze point detection unit 29 outputs data indicating the three-dimensional coordinates of the gaze point detected continuously for each frame of the face image and the object image to the output device 106 (step S8). At this time, for example, an image in which a marker indicating the gaze point of the subject Sp is superimposed on the image of the observation object So may be output. At this time, the data indicating the three-dimensional coordinates of the gaze point may be recorded in a database in the auxiliary storage unit 103 together with data indicating the posture of the observation object So.

[0054] According to the image processing system 1 according to the present embodiment described above, a face image of the subject Sp is acquired, and an object image, which is an image of the observation object So, is acquired. Then, three markers M on the observation object So detected from the object image are A ,M B ,M C The arrangement angle and position of the observation object So are calculated using the three-dimensional position of the object, the angle and position of the line of sight of the subject Sp are calculated from the face image, and the position of the gaze point of the subject Sp on the inspection surface P1 of the observation object So is detected based on these. This makes it possible to accurately detect the gaze point of the subject Sp with respect to the observation object So even when the position and angle of the observation object So change when performing a visual inspection of the observation object So, etc.

[0055] Here, the image processing system 1 further includes a support member 11 that integrally supports the subject detection optical system 3 and the object detection optical system 5. In this configuration, it is possible to stabilize the relationship between the line of sight vector of the subject Sp detected using the subject detection optical system 3 and the arrangement of the observed object So detected using the object detection optical system 5, and it is possible to accurately detect the position of the gaze point of the subject Sp relative to the observed object So.

[0056] The image processing system 1 further includes a table 7 that is supported by a support member 11 and serves as a reference when the subject Sp grasps the observation object So. In this case, the position of the observation object So relative to the object detection optical system 5 is stabilized, so that the arrangement angle and position of the observation object So can be detected reliably. As a result, the position of the gaze point of the subject Sp relative to the observation object So can be detected more reliably.

[0057] In addition, the computer 9 obtains an equation for the inspection surface P1 in the three-dimensional coordinate system based on the arrangement angle and position of the observation object So, and calculates the position of the gaze point based on this equation and the line of sight vector. In this way, the arrangement of the inspection surface P1 of the observation object So can be specified by the equation, so that the position of the gaze point of the subject Sp on the inspection surface P1 of the observation object So can be calculated with high accuracy.

[0058] In addition, the computer 9 determines an equation for each surface obtained by dividing the inspection target surface P1 into a plurality of surfaces based on the arrangement angle and position of the observation target So, searches for a surface having an intersection with the line of sight vector from among the plurality of surfaces using the equations for the plurality of surfaces, and determines the intersection between the searched surface and the line of sight vector as the position of the gaze point. With this configuration, even when the observation target So has a complex shape, the arrangement of the inspection target surface P1 of the observation target So can be specified by an equation for each of the divided surfaces, so that the position of the gaze point of the subject Sp on the inspection target surface P1 of the observation target So can be calculated with high accuracy.

[0059] The computer 9 also preliminarily determines three true markers M A ,M B ,M C The positional relationship of the four or more marker candidates is stored, and three true markers M are selected from the three-dimensional coordinates of the four or more marker candidates. A ,M B ,M C The three-dimensional coordinates of the three refined true markers M A ,M B ,M C The arrangement angle and position of the observation object So are calculated based on the three-dimensional coordinates of the object. In this case, when four or more markers including a false marker are detected from the object image, the true marker M is selected from among them. A ,M B ,M C As a result, the arrangement of the observation object So can be detected without error, and the position of the gaze point of the subject Sp on the inspection target surface P1 of the observation object So can be detected without error.

[0060] The present invention is not limited to the above-described embodiments.

[0061] For example, four or more markers for detecting the orientation of the observed object So may be attached on the inspection target surface P1 of the observed object So. In this case, the orientation calculation unit 25 of the computer 9 specifies the three-dimensional coordinates of the four or more markers, and calculates the orientation of the observed object So using these three-dimensional coordinates. For example, when using the three-dimensional coordinates of four markers, the orientation calculation unit 25 calculates the center of gravity of the four markers in the object coordinate system X h -Y h -Z h The coordinates of the center of O h The equation of the plane is calculated using the least squares method from the three-dimensional coordinates of the four markers, and the normal vector determined from the calculated equation of the plane is h In addition, when there are four true markers, the orientation calculation unit 25 narrows down the three-dimensional coordinates of the four true markers from the three-dimensional coordinates of five or more marker candidates in the same manner as in the above-described embodiment.

[0062] The marker for detecting the orientation of the observed object So, which is attached to the inspection surface P1 of the observed object So, may have a pattern that can be identified by image processing, such as a QR code (registered trademark) or a simplified code. Furthermore, multiple markers attached to the inspection surface P1 of the same observed object So may have different patterns. In this case, the orientation calculation unit 25 of the computer 9 can detect the positions of the reflected images of multiple markers on the inspection surface P1 in a state where each of them can be identified, by simple image processing. As a result, the processing time for narrowing down the three-dimensional coordinates of the true markers can be shortened.

[0063] On the other hand, in another modified example, a single marker for detecting the orientation of the observation object So may be attached on the inspection target surface P1 of the observation object So. For example, a marker having a form as shown in part (a) of FIG. 3, in which the shape of the four corners of the inner white rectangular shape are different from each other, may be attached at one point. As an example of a marker of this form, for example, a form in which the four corners are chipped in different ways is exemplified. The orientation calculation unit 25 of the computer 9 can identify the positions of the four corners of a single marker in the object image by the difference in their shapes. For example, the identification can be performed by rotating a template representing the shape and matching the template with the object image. As a result, the orientation calculation unit 25 can detect the three-dimensional coordinates of the four corners of a single marker, and can calculate the orientation of the observation object So using the detection results.

[0064] Furthermore, three or more characteristic points or one characteristic shape part originally existing on the observation object So may be used as the markers, without preparing special markers and attaching them to the observation object So. In this case, the orientation calculation unit 25 specifies the characteristic points or the characteristic shape part as the markers by image processing of the object image.

[0065] The observation object So to be imaged by the image processing system 1 may be a flat display device such as a liquid crystal display, a tablet terminal, or a small game device as shown in FIG. 8. In this case, for example, it may be possible to detect which part of the image displayed on the display the gaze is directed to, and the detection result may be reflected in the next processing content or image display. In this modification, when the image processing system 1 captures the image, a calibration target M1 may be dynamically displayed on the inspection target surface P1, which is the display surface of the observation object So. The coefficient used in calculating the gaze vector can be calibrated in advance by using the position of the target M1 on the face image. However, in order to accurately calibrate this coefficient, it is necessary to have the subject Sp correctly focus his / her gaze on the center of the target M1. Therefore, in order to achieve accurate calibration, it is preferable to display the target M1 so that it dynamically changes from a large size to a small size, as shown in FIG. 4. By displaying it in this way, the subject Sp can increase his / her concentration, and as a result, he / she can easily focus the gaze of both eyes on a small area at the center of the target M1. As a result, accurate calibration of the gaze vector can be achieved.

[0066] Furthermore, the subject detection optical system 3 or the subject detection optical system 5 is not limited to being composed of a stereo camera, and may be composed of a single camera. Alternatively, the subject detection optical system 3 and the subject detection optical system 5 may be composed of a single camera including a high-performance (high resolution, high speed, wide field of view, etc.) optical system, and may be configured to be capable of simultaneously photographing the subject Sp and the observed object So by the single camera in accordance with the positional relationship between the subject Sp and the observed object So. According to the image processing system 1 including such an optical system, the posture of the observed object So and the gaze direction of the subject Sp can be simultaneously detected.

[0067] Furthermore, the markers of the observation target of the image processing system 1 are not limited to being attached directly onto the observation target So. For example, when the observation target So is an electronic board or the like, a jig for holding it may be prepared, and multiple markers may be attached to the back surface of the jig.

[0068] The markers of the observation target of the image processing system 1 may be markers attached at four or more points to the observation target So having a three-dimensional shape. In this case, for example, even if one marker is hidden or falls out of the field of view of the object detection optical system 5 due to a rotation operation by the subject Sp during visual inspection, the three-dimensional positions of the remaining three markers can be detected.

[0069] 1, the image processing system 1 may include a housing (case) that houses the subject detection optical system 3, the object detection optical system 5, and the computer 9. In this case, the upper part of the subject detection optical system 3 in the housing is made of a material that is optically transparent at least in the wavelength band used for the light source, such as glass. This allows the camera of the subject detection optical system 3 to be protected (covered) while capturing an image of the subject Sp by the camera.

[0070] 1, the support member 11 integrally supports the table 7, the subject detection optical system 3, and the subject detection optical system 5, but the top plate portion 7a of the table 7 may be shaped to extend toward the subject detection optical system 5, the top plate portion 7a may support the subject detection optical system 5, and another table or housing may be integrally provided at an appropriate position below the top plate portion 7a on the subject Sp side, and the subject detection optical system 3 may be fixed to the table or housing. In other words, the role of the support member 11 may be played by the table 7 itself or a member integrally provided on the table 7.

[0071] In addition, in the image processing system 1, the table 7 may be omitted. [Explanation of symbols]

[0072] 1...image processing system, 3...object detection optical system, 5...object detection optical system, 7...table (flat member), 9...computer (calculation unit), 11...support member, M A ,M B ,M C...marker, P1...surface to be inspected (specific surface), So...object to be observed, Sp...subject.

Claims

1. a subject detection optical system including a camera for capturing an image of a subject's face, for acquiring a facial image necessary for detecting the angle and position of the subject's line of sight; an object detection optical system including a camera for capturing an image of an object being observed by the subject, the object detection optical system acquiring an object image necessary for detecting an arrangement angle and a position of the object being observed; a calculation unit that executes image processing on the face image and the object image; a support member that integrally supports the subject detection optical system and the subject detection optical system; A flat plate member supported by the support member and serving as a reference when the subject grasps the observation object; Equipped with The calculation unit is Identifying a three-dimensional position of a marker on the observed object based on the object image, and calculating a placement angle and a position of the observed object based on the three-dimensional position of the marker; Calculating an angle and a position of the line of sight of the subject based on the face image, and detecting a gaze point of the subject on a specific surface of the observed object based on the angle and position of the line of sight and a placement angle and position of the observed object. Image processing system.

2. A subject detection optical system including a camera for capturing an image of a subject's face, for acquiring a facial image necessary for detecting the angle and position of the subject's line of sight; an object detection optical system including a camera for capturing an image of an object being observed by the subject, the object detection optical system acquiring an object image necessary for detecting an arrangement angle and a position of the object being observed; a calculation unit that performs image processing on the face image and the object image, The calculation unit is Identifying a three-dimensional position of a marker on the observed object based on the object image, and calculating a placement angle and a position of the observed object based on the three-dimensional position of the marker; Calculating an angle and a position of the gaze of the subject based on the face image, and detecting a gaze point of the subject on a specific surface of the observed object based on the angle and position of the gaze and a placement angle and position of the observed object; A formula for the specific surface in a three-dimensional coordinate system is obtained based on the arrangement angle and position of the observed object, and the gaze point is calculated based on the formula and the vector representing the line of sight. Image processing system.

3. A subject detection optical system including a camera for capturing an image of a subject's face, for acquiring a facial image necessary for detecting the angle and position of the subject's line of sight; an object detection optical system including a camera for capturing an image of an object being observed by the subject, the object detection optical system acquiring an object image necessary for detecting an arrangement angle and a position of the object being observed; a calculation unit that performs image processing on the face image and the object image, The calculation unit is Identifying a three-dimensional position of a marker on the observed object based on the object image, and calculating a placement angle and a position of the observed object based on the three-dimensional position of the marker; Calculating an angle and a position of the gaze of the subject based on the face image, and detecting a gaze point of the subject on a specific surface of the observed object based on the angle and position of the gaze and a placement angle and position of the observed object; storing a positional relationship between a plurality of real markers in advance, narrowing down the three-dimensional positions of the identified plurality of candidate markers to the three-dimensional positions of the plurality of real markers, and calculating an arrangement angle and a position of the observed object based on the narrowed-down three-dimensional positions of the plurality of real markers; Image processing system.

4. A subject detection optical system including a camera for capturing an image of a subject's face, for acquiring a facial image necessary for detecting the angle and position of the subject's line of sight; an object detection optical system including a camera for capturing an image of an object being observed by the subject, the object detection optical system acquiring an object image necessary for detecting an arrangement angle and a position of the object being observed; a calculation unit that performs image processing on the face image and the object image, The calculation unit is Identifying a three-dimensional position of a marker on the observed object based on the object image, and calculating a placement angle and a position of the observed object based on the three-dimensional position of the marker; Calculating an angle and a position of the gaze of the subject based on the face image, and detecting a gaze point of the subject on a specific surface of the observed object based on the angle and position of the gaze and a placement angle and position of the observed object; a three-dimensional position of a calibration target is specified based on an image previously obtained by an object detection optical system, and a calibration process is previously performed to calculate the angle and position of the line of sight based on the face image and the three-dimensional position of the calibration target; Image processing system.

5. the calculation unit specifies three-dimensional positions of three or more markers on the observation object, and calculates an arrangement angle and a position of the observation object based on the three-dimensional positions of the three or more markers; The image processing system according to any one of claims 1 to 4.

6. the calculation unit calculates an equation for each of the plurality of surfaces obtained by dividing the specific surface into a plurality of surfaces based on a position and an arrangement angle of the observed object, searches for a surface having an intersection with the vector from among the plurality of surfaces using the equations for the plurality of surfaces, and calculates the intersection between the searched surface and the vector as the gaze point. The image processing system according to claim 2 .

7. An image processing method executed by an image processing system, comprising: A step of capturing an image of a face of a subject to obtain a face image necessary for detecting an angle and a position of the gaze of the subject; capturing an image of an object being observed by the subject to obtain an object image necessary for detecting an arrangement angle and a position of the object being observed; and executing image processing on the face image and the object image, In the step of performing image processing, Identifying a three-dimensional position of a marker on the observed object based on the object image, and calculating a placement angle and a position of the observed object based on the three-dimensional position of the marker; Calculating an angle and a position of the line of sight of the subject based on the face image, and detecting a gaze point of the subject on a specific surface of the observation object based on the angle and position of the line of sight and a placement angle and position of the observation object; A formula for the specific surface in a three-dimensional coordinate system is obtained based on the arrangement angle and position of the observed object, and the gaze point is calculated based on the formula and the vector representing the line of sight. Image processing methods.

8. An image processing method executed by an image processing system, comprising: A step of capturing an image of a face of a subject to obtain a face image necessary for detecting an angle and a position of the gaze of the subject; capturing an image of an object being observed by the subject to obtain an object image necessary for detecting an arrangement angle and a position of the object being observed; and executing image processing on the face image and the object image, In the step of performing image processing, Identifying a three-dimensional position of a marker on the observed object based on the object image, and calculating a placement angle and a position of the observed object based on the three-dimensional position of the marker; Calculating an angle and a position of the line of sight of the subject based on the face image, and detecting a gaze point of the subject on a specific surface of the observation object based on the angle and position of the line of sight and a placement angle and position of the observation object; storing a positional relationship between a plurality of real markers in advance, narrowing down the three-dimensional positions of the identified plurality of candidate markers to the three-dimensional positions of the plurality of real markers, and calculating an arrangement angle and a position of the observed object based on the narrowed-down three-dimensional positions of the plurality of real markers; Image processing methods.

9. An image processing method executed by an image processing system, comprising: A step of capturing an image of a face of a subject to obtain a face image necessary for detecting an angle and a position of the gaze of the subject; capturing an image of an object being observed by the subject to obtain an object image necessary for detecting an arrangement angle and a position of the object being observed; and executing image processing on the face image and the object image, In the step of performing image processing, Identifying a three-dimensional position of a marker on the observed object based on the object image, and calculating a placement angle and a position of the observed object based on the three-dimensional position of the marker; Calculating an angle and a position of the line of sight of the subject based on the face image, and detecting a gaze point of the subject on a specific surface of the observation object based on the angle and position of the line of sight and a placement angle and position of the observation object; a three-dimensional position of a calibration target is specified based on an image previously obtained by an object detection optical system, and a calibration process is previously performed to calculate the angle and position of the line of sight based on the face image and the three-dimensional position of the calibration target; Image processing methods.

Images