DETECTION SYSTEM, LEARNING METHOD, AND COMPUTER PROGRAM

Abstract

To appropriately detect each of a first part and a second part different in shape.SOLUTION: A detection system comprises learning means for a detection model that detects feature graphics corresponding to an outline of an iris or a pupil and feature points corresponding to an outline of eyelids from an image containing a living body. The learning means performs learning of the detection model by learning the feature graphics and feature points as a composite target.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to the technical fields of a detection system, a learning method, and a computer program for detecting a part of a living body from an image. [Background technology]

[0002] Known systems of this type detect areas around the eyes of a living body from an image. For example, Patent Document 1 discloses detecting a pupil circle and an iris circle from an image. Patent Document 2 discloses detecting a face from an image and detecting the eyes from position information of the face. Patent Document 3 discloses extracting feature points from a face image. Patent Document 4 discloses detecting a circular area from a ROI (Region of Interest) and detecting multiple areas that are candidates for the iris. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2003-317102 [Patent Document 2] Japanese Patent Application Laid-Open No. 2007-213377 [Patent Document 3] Japanese Patent Application Laid-Open No. 2014-075098 [Patent Document 4] Japanese Patent Application Publication No. 2018-045437 Summary of the Invention [Problem to be solved by the invention]

[0004] In detection using an image as input, methods for detecting feature points or feature shapes can be considered. However, the above cited documents do not mention detecting both feature points and feature shapes, and there is room for improvement.

[0005] An object of the present disclosure is to provide a detection system, a learning method, and a computer program that can solve the above-mentioned problems. [Means for solving the problem]

[0006] One aspect of the detection system of the present disclosure includes a detection model learning means for detecting a feature figure corresponding to the outline of the iris or pupil and a feature point corresponding to the outline of the eyelid from an image including a living body, and the learning means learns the detection model by learning the feature figure and the feature point as a composite target.

[0007] One aspect of the learning method of this disclosure is a method for learning a detection model in which at least one computer detects a feature figure corresponding to the outline of the iris or pupil and a feature point corresponding to the outline of the eyelid from an image including a living body, and the detection model is trained by learning the feature figure and the feature point as a composite target.

[0008] One aspect of the computer program of this disclosure is a method for training a detection model that detects, from an image containing a living body, a feature figure corresponding to the outline of an iris or a pupil and a feature point corresponding to the outline of an eyelid, by using at least one computer to execute the training method, in which the detection model is trained by learning the feature figure and the feature point as a composite target. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a block diagram showing a hardware configuration of a detection system according to a first embodiment. [Figure 2] 1 is a block diagram showing a functional configuration of a detection system according to a first embodiment. [Figure 3] 4 is a flowchart showing the flow of operations of the detection system according to the first embodiment. [Figure 4] FIG. 2 is a diagram showing an example of a characteristic graphic detected by the detection system according to the first embodiment. [Figure 5]FIG. 10 is a block diagram showing the functional configuration of a detection system according to a second embodiment. [Figure 6] 10 is a flowchart showing the flow of operations of the detection system according to the second embodiment. [Figure 7] 10A to 10C are diagrams illustrating an example of detection of characteristic figures and characteristic points by the detection system according to the second embodiment. [Figure 8] FIG. 10 is a block diagram showing the functional configuration of a detection system according to a third embodiment. [Figure 9] 10 is a flowchart showing the flow of operations of the detection system according to the third embodiment. [Figure 10] FIG. 13 is a diagram illustrating an example of a gaze estimation method by a detection system according to a fourth embodiment. [Figure 11] FIG. 10 is a block diagram showing the functional configuration of a detection system according to a fifth embodiment. [Figure 12] 10 is a flowchart showing the flow of operations of the detection system according to the fifth embodiment. [Figure 13] FIG. 13 is a diagram illustrating a specific example of the operation of the detection system according to the fifth embodiment. [Figure 14] FIG. 13 is a block diagram showing the functional configuration of a detection system according to a sixth embodiment. [Figure 15] 13 is a flowchart showing the flow of operations of the detection system according to the sixth embodiment. [Figure 16] FIG. 10 is a diagram (part 1) showing an example of displaying feature points and feature figures on a display unit. [Figure 17] FIG. 10 is a diagram (part 2) showing an example of displaying feature points and feature figures on the display unit. [Figure 18] FIG. 10 is a diagram (part 3) showing an example of displaying characteristic points and characteristic figures on the display unit. [Figure 19] FIG. 10 is a diagram (part 4) showing an example of displaying characteristic points and characteristic figures on the display unit. [Figure 20] FIG. 5 is a diagram (part 5) showing an example of displaying characteristic points and characteristic figures on the display unit. [Figure 21] FIG. 13 is a block diagram showing the functional configuration of a detection system according to a seventh embodiment. [Figure 22] 13 is a flowchart showing the flow of operations of the detection system according to the seventh embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of a detection system, a detection method, and a computer program will be described with reference to the drawings.

[0011] First Embodiment A detection system according to a first embodiment will be described with reference to FIGS.

[0012] (Hardware configuration) First, the hardware configuration of the detection system according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the hardware configuration of the detection system according to the first embodiment.

[0013] 1, a detection system 10 according to the first embodiment includes a processor 11, a RAM (Random Access Memory) 12, a ROM (Read Only Memory) 13, and a storage device 14. The detection system 10 may further include an input device 15 and an output device 16. The processor 11, RAM 12, ROM 13, storage device 14, input device 15, and output device 16 are connected via a data bus 17.

[0014] The processor 11 loads a computer program. For example, the processor 11 is configured to load a computer program stored in at least one of the RAM 12, the ROM 13, and the storage device 14. Alternatively, the processor 11 may load a computer program stored in a computer-readable storage medium using a storage medium reader (not shown). The processor 11 may acquire (i.e., load) the computer program from a device (not shown) located outside the detection system 10 via a network interface. The processor 11 controls the RAM 12, the storage device 14, the input device 15, and the output device 16 by executing the loaded computer program. In particular, in this embodiment, when the processor 11 executes the loaded computer program, a functional block for detecting a portion of a living body from an image is realized within the processor 11. The processor 11 may be one of a central processing unit (CPU), a graphics processing unit (GPU), a field-programmable gate array (FPGA), a demand-side platform (DSP), and an application-specific integrated circuit (ASIC), or a plurality of such devices may be used in parallel.

[0015] The RAM 12 temporarily stores computer programs executed by the processor 11. The RAM 12 temporarily stores data that is temporarily used by the processor 11 while the processor 11 is executing the computer programs. The RAM 12 may be, for example, a D-RAM (Dynamic RAM).

[0016] The ROM 13 stores computer programs executed by the processor 11. The ROM 13 may also store fixed data. The ROM 13 may be, for example, a programmable ROM (P-ROM).

[0017] The storage device 14 stores data that is to be saved long-term by the detection system 10. The storage device 14 may operate as a temporary storage device for the processor 11. The storage device 14 may include, for example, at least one of a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and a disk array device.

[0018] The input device 15 is a device that receives input instructions from a user of the detection system 10. The input device 15 may include, for example, at least one of a keyboard, a mouse, and a touch panel.

[0019] The output device 16 is a device that outputs information related to the detection system 10 to the outside. For example, the output device 16 may be a display device (for example, a display) that can display information related to the detection system 10.

[0020] (Functional configuration) Next, the functional configuration of the detection system 10 according to the first embodiment will be described with reference to Fig. 2. Fig. 2 is a block diagram showing the functional configuration of the detection system according to the first embodiment.

[0021] 2, a detection system 10 according to the first embodiment is configured to detect a part of a living body from an image. The detection system 10 includes an image acquisition unit 110 and a detection unit 120 as processing blocks or physical processing circuits for realizing the function. The image acquisition unit 110 and the detection unit 120 can be realized by, for example, the above-mentioned processor 11 (see FIG. 1).

[0022] The image acquisition unit 110 is configured to be able to acquire an image (i.e., an image to be detected) input to the detection system 10. The image acquisition unit 110 may include a storage unit that stores the acquired image. Information related to the image acquired by the image acquisition unit 110 is configured to be output to the detection unit 120.

[0023] The detection unit 120 is configured to detect a part of a living body from an image acquired by the image acquisition unit 110. Specifically, the detection unit 110 is configured to detect a characteristic figure corresponding to a first part of the living body and a characteristic point corresponding to a second part of the living body. Here, the "first part" refers to a part of the living body that has a substantially circular shape. On the other hand, the "second part" refers to a part of the living body that is located around the first part. It is sufficient to set in advance which part of the living body is the first part and which part is the second part. In this case, multiple parts of different types may be set as the first part, and multiple parts of different types may be set as the second part. The detection unit 120 may have a function to output information about the detected characteristic points and characteristic figures.

[0024] (Operation flow) Next, the flow of operations of the detection system 10 according to the first embodiment will be described with reference to Fig. 3. Fig. 3 is a flowchart showing the flow of operations of the detection system according to the first embodiment.

[0025] As shown in FIG. 3, when the detection system 10 according to the first embodiment operates, first, the image acquisition unit 110 acquires an image (step S101).

[0026] Next, the detection unit 120 detects a characteristic figure corresponding to the first portion from the image acquired by the image acquisition unit 110 (step S102). The detection unit 120 further detects a characteristic point corresponding to the second portion from the image acquired by the image acquisition unit 110 (step S103). The detected characteristic figure and characteristic point can be expressed by coordinates, a mathematical formula, or the like. Note that the processes of steps S102 and S103 may be executed one after the other, or may be executed simultaneously in parallel. In other words, the order of detection of the characteristic figure corresponding to the first portion and the characteristic point corresponding to the second portion is not limited, and the characteristic figure and the characteristic point may be detected simultaneously.

[0027] (Examples of characteristic shapes) Next, a specific example of a characteristic graphic detected by the detection system 10 according to the first embodiment will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of a characteristic graphic detected by the detection system according to the first embodiment.

[0028] As shown in FIG. 4, in the detection system 10 according to the first embodiment, the detection unit 120 detects a circle (including an ellipse) as a characteristic graphic corresponding to the second portion. The circle detected by the detection unit 120 may include a perfect circle, a vertically elongated ellipse, a horizontally elongated ellipse, and a diagonal ellipse (i.e., an ellipse rotated at an arbitrary angle). The type of circle to be actually detected may be set in advance. For example, a shape corresponding to the portion of the living body to be detected may be set in advance. The detection unit 120 may also be configured to be able to detect a circle with a missing portion or a circle with a hidden portion.

[0029] (Technical Effects) Next, the technical effects obtained by the detection system 10 according to the first embodiment will be described.

[0030] 1 to 3, in the detection system 10 according to the first embodiment, a characteristic figure corresponding to a first portion and a characteristic point corresponding to a second portion are detected from an image. That is, the first portion and the second portion are detected by different methods. This makes it possible to appropriately detect the first portion and the second portion, which have different shape characteristics, from an image of a living body.

[0031] Second Embodiment A detection system 10 according to the second embodiment will be described with reference to Figures 5 to 7. The second embodiment differs from the first embodiment described above only in some configurations and operations, and for example, the hardware configuration may be the same as that of the first embodiment (see Figure 1). Therefore, in the following, descriptions of parts that overlap with the embodiments already described will be omitted as appropriate.

[0032] (Functional configuration) First, the functional configuration of the detection system 10 according to the second embodiment will be described with reference to Fig. 5. Fig. 5 is a block diagram showing the functional configuration of the detection system according to the second embodiment. Note that in Fig. 5, the same elements as those shown in Fig. 2 are denoted by the same reference numerals.

[0033] As shown in Fig. 5, the detection system 10 according to the second embodiment includes an image acquisition unit 110, a detection unit 120, and an iris authentication unit 130 as processing blocks for realizing its functions or as physical processing circuits. That is, the detection system 10 according to the second embodiment is configured to further include the iris authentication unit 130 in addition to the components of the first embodiment (see Fig. 2). The iris authentication unit 130 can be realized by, for example, the above-mentioned processor 11 (see Fig. 1).

[0034] The iris authentication unit 130 is configured to be able to perform iris authentication using the feature points and feature figures detected by the detection unit 120. The iris authentication unit 130 is configured to identify an iris area based on, for example, feature points corresponding to the eyelids, which are an example of the first part, and feature figures corresponding to the iris and pupils, which are an example of the second part (see FIG. 7), and to perform iris authentication processing using the identified iris area. The iris authentication unit 130 may also have a function to output the results of iris authentication. The iris authentication unit 130 may also be configured to perform part of the iris authentication processing outside the system (for example, on an external server, in the cloud, etc.).

[0035] (Operation flow) Next, the flow of operation of the detection system 10 according to the second embodiment will be described with reference to Fig. 6. Fig. 6 is a flowchart showing the flow of operation of the detection system according to the second embodiment. Note that in Fig. 6, the same processes as those shown in Fig. 3 are denoted by the same reference numerals.

[0036] 6, when the detection system 10 according to the second embodiment operates, the image acquisition unit 110 first acquires an image (step S101). Then, the detection unit 120 detects a characteristic figure corresponding to a first portion from the image acquired by the image acquisition unit 110 (step S102). The detection unit 120 further detects a characteristic point corresponding to a second portion from the image acquired by the image acquisition unit 110 (step S103).

[0037] Next, the iris authentication unit 130 identifies the eyelid region (i.e., the region where the eyelids are present) from the feature points corresponding to the eyelids, and generates a mask of the eyelid region (step S201). The eyelid region mask is used to remove the eyelid region that is not required for iris authentication (in other words, that does not have iris information). After that, the iris authentication unit 130 identifies the iris region (i.e., the region from which iris information can be obtained) from the feature figures corresponding to the iris and pupil, and performs iris authentication using the iris region (step S202). Note that the specific processing content of iris authentication can be appropriately adopted from existing technologies, so a detailed description thereof will be omitted here. (Example of detection around the eyes) Next, detection of a characteristic graphic and a characteristic point by the detection system 10 according to the second embodiment will be described with reference to Fig. 7. Fig. 7 is a diagram showing an example of detection of a characteristic graphic and a characteristic point by the detection system according to the second embodiment.

[0038] As shown in FIG. 7, in the detection system 10 according to the second embodiment, the "iris and pupil" is detected as the first part, and the "eyelid" is detected as the second part.

[0039] The detection unit 120 detects a circle corresponding to the iris and a circle corresponding to the pupil. Note that the detection unit 120 may detect only one of the circle corresponding to the iris and the circle corresponding to the pupil. Because the shape of the iris and pupil is close to a circle, they are suitable for detection as characteristic figures of approximately circles. If the iris or pupil were to be detected using characteristic points (e.g., points on the circumference), the number and positions of the points would depend on the system design, which would directly affect the detection accuracy. However, if the iris or pupil is detected as a circle, the position of the iris or pupil can be determined as a circle formula. Because the circle formula is uniquely determined, it is independent of the system design and does not affect the detection accuracy. From these points of view, it can be said that the iris is suitable for detection using a circle.

[0040] The detection unit 120 also detects multiple feature points indicating the position (contour) of the eyelids. In the example shown in the figure, the detection unit 120 detects two feature points corresponding to the inner and outer corners of the eye, three feature points corresponding to the upper eyelid, and three feature points corresponding to the lower eyelid. Note that the number of feature points described above is merely an example, and fewer or more feature points may be detected. Eyelids vary considerably among individuals, for example, there are large variations in shape between individuals, such as single eyelids, double eyelids, slanted eyes, and droopy eyes. Therefore, they are suitable for detection as feature points rather than as feature figures. Note that although the shape of eyelids varies among individuals, they are commonly located near the iris and pupil. Therefore, by detecting them together with feature figures, they can be relatively easily detected as feature points.

[0041] (Technical Effects) Next, the technical effects obtained by the detection system 10 according to the second embodiment will be described.

[0042] As described with reference to Figures 5 to 7, the detection system 10 according to the second embodiment performs iris authentication using the detected feature figures and feature points. In this embodiment, in particular, multiple parts around the eyes are properly detected, making it possible to properly perform iris authentication. In the example shown in Figure 5, the coordinates of the inner and outer corners of the eye and the radii of the iris and pupil circles are known, so the ratio between the distance from the inner to outer corner of the eye and the difference between the radii of the two circles may be used for data matching or weighting in iris authentication.

[0043] Third Embodiment A detection system 10 according to the third embodiment will be described with reference to Figures 8 and 9. The third embodiment differs from the above-described embodiments only in some configurations and operations, and for example, the hardware configuration may be the same as that of the first embodiment (see Figure 1). Therefore, in the following, descriptions of parts that overlap with the embodiments already described will be omitted as appropriate.

[0044] (Functional configuration) First, the functional configuration of the detection system 10 according to the third embodiment will be described with reference to Fig. 8. Fig. 8 is a block diagram showing the functional configuration of the detection system according to the third embodiment. In Fig. 8, the same elements as those shown in Figs. 2 and 5 are denoted by the same reference numerals.

[0045] As shown in Fig. 8, the detection system 10 according to the fifth embodiment includes an image acquisition unit 110, a detection unit 120, and a gaze estimation unit 140 as processing blocks for realizing its functions or as physical processing circuits. That is, the detection system 10 according to the third embodiment is configured to further include the gaze estimation unit 140 in addition to the components of the first embodiment (see Fig. 2). Note that the gaze estimation unit 140 can be realized by, for example, the above-mentioned processor 11 (see Fig. 1).

[0046] The gaze estimation unit 140 is configured to be able to perform gaze direction estimation using the feature points and feature figures detected by the detection unit 120. Specifically, the gaze estimation unit 140 is configured to be able to perform a process of estimating the gaze direction based on the "feature points corresponding to the eyelids" and the "feature figures corresponding to the iris and pupil" (see FIG. 7) described in the second embodiment. Note that the feature points corresponding to the eyelids and the feature figures corresponding to the iris and pupil may be detected as described in the second embodiment. The gaze estimation unit 140 may have a function of outputting the result of gaze estimation. Furthermore, the gaze estimation unit 140 may be configured to perform part of the gaze estimation process outside the system (for example, by an external server, cloud, etc.).

[0047] (Operation flow) Next, the flow of operation of the detection system 10 according to the third embodiment will be described with reference to Fig. 9. Fig. 9 is a flowchart showing the flow of operation of the detection system according to the third embodiment. Note that in Fig. 9, the same processes as those shown in Figs. 3 and 6 are denoted by the same reference numerals.

[0048] 9, when the detection system 10 according to the third embodiment operates, the image acquisition unit 110 first acquires an image (step S101). Then, the detection unit 120 detects a characteristic figure corresponding to a first portion from the image acquired by the image acquisition unit 110 (step S102). The detection unit 120 further detects a characteristic point corresponding to a second portion from the image acquired by the image acquisition unit 110 (step S103).

[0049] Next, the gaze estimation unit 140 estimates the gaze direction based on the characteristic graphic and the characteristic points (step S301). Note that the specific processing content of the gaze direction estimation will be explained in detail in the fourth embodiment described later.

[0050] (Technical Effects) Next, the technical effects obtained by the detection system 10 according to the third embodiment will be described.

[0051] 8 and 9, the detection system 10 according to the third embodiment performs gaze estimation using the detected feature figures and feature points. In particular, in this embodiment, the parts used for estimating the gaze direction (for example, parts around the eyes) are properly detected, so that the gaze direction can be properly estimated.

[0052] <Fourth embodiment> A detection system 10 according to a fourth embodiment will be described with reference to Fig. 10. The fourth embodiment shows a more specific configuration of the third embodiment (i.e., a specific method for estimating the gaze direction), and the configuration and operation flow may be the same as those of the third embodiment (see Figs. 8 and 9). Therefore, in the following, descriptions of parts that overlap with those already described will be omitted as appropriate.

[0053] (Calculation of relative position) A method for estimating a gaze direction by the detection system 10 according to the fourth embodiment will be described with reference to Fig. 10. Fig. 10 is a diagram showing an example of a method for estimating a gaze direction by the detection system according to the fourth embodiment.

[0054] 10, the detection system 10 according to the fourth embodiment estimates the gaze direction using feature points corresponding to the eyelids and feature figures corresponding to the iris and pupil. Note that the feature points corresponding to the eyelids and the feature figures corresponding to the iris and pupil may be detected as described in the second embodiment (see FIG. 7).

[0055] Feature points 1 and 2 in the figure correspond to the inner and outer corners of the eyes, respectively. Feature points 3 and 4 are points on the midline of feature points 1 and 2 that intersect with the eyelids. Therefore, as long as the face direction does not change, feature points 1 to 4 will be in the same position no matter which direction the eyeballs are facing. Note that if the direction of the eyeballs changes, circles 5 and 6, which are feature figures corresponding to the iris and pupil, will move. Therefore, the direction of the eyeballs (i.e., the gaze direction) can be estimated by using the relative relationship between the positions of the eyelids, which can be identified from each feature point, and the positions of the iris and pupil, which can be identified from each feature figure.

[0056] To estimate the gaze, it is necessary to determine in advance the relationship between the relative position of the eyelids and the position of the eyeballs and the current position of the gaze. This relationship may be calculated as a function or created as a table.

[0057] When calculating gaze direction, the image is first normalized using circle 6, which corresponds to the iris. Next, the intersection of the line connecting feature points 1 and 2 with the line connecting feature points 3 and 4 is set as the origin, and the relative positional relationship between the eyelid and eyeball is calculated from information on how many pixels away from this origin in the x and y directions. The calculated positional relationship between the eyelid and eyeball is then used to estimate gaze direction.

[0058] (Technical Effects) Next, the technical effects obtained by the detection system 10 according to the fourth embodiment will be described.

[0059] 10, in the detection system 10 according to the fourth embodiment, the gaze direction is estimated from the positional relationship between the eyelid and the eyeball. In particular, in this embodiment, the relative positional relationship between the eyelid and the iris and pupil can be appropriately calculated from the feature points corresponding to the eyelid and the feature figures corresponding to the iris and pupil, so that the gaze direction can be appropriately estimated.

[0060] Fifth Embodiment A detection system 10 according to the fifth embodiment will be described with reference to Figures 11 to 13. The fifth embodiment differs from the above-described embodiments only in part of its configuration and operation, and for example, the hardware configuration may be the same as that of the first embodiment (see Figure 1). Therefore, in the following, descriptions of parts that overlap with the embodiments already described will be omitted as appropriate.

[0061] (Functional configuration) First, the functional configuration of the detection system 10 according to the fifth embodiment will be described with reference to Fig. 11. Fig. 11 is a block diagram showing the functional configuration of the detection system according to the fourth embodiment. In Fig. 11, the same elements as those shown in Figs. 2, 5, and 8 are denoted by the same reference numerals.

[0062] 11, the detection system 10 according to the fifth embodiment includes an image acquisition unit 110, a detection unit 120, a rotation angle estimation unit 150, and an image rotation unit 160 as processing blocks for realizing its functions or as physical processing circuits. That is, the detection system 10 according to the fifth embodiment is configured to further include the rotation angle estimation unit 150 and the image rotation unit 160 in addition to the components of the first embodiment (see FIG. 2). Note that the rotation angle estimation unit 150 and the image rotation unit 160 can be realized by, for example, the above-mentioned processor 11 (see FIG. 1).

[0063] The rotation angle estimation unit 150 can estimate the rotation angle (i.e., tilt) of the image acquired by the image acquisition unit 110 based on the feature points detected by the detection unit 120. For example, as shown in the second embodiment, if feature points of the eyelids are detected (see FIG. 7, etc.), the image acquisition unit 110 estimates the rotation angle of the image from the detected tilt of the eyelids. Note that the rotation angle estimation unit 150 may estimate the rotation angle of the image by taking into account feature figures in addition to the feature points detected by the detection unit 120. For example, as shown in the second embodiment, if feature points of the eyelids and feature figures of the iris or pupil are detected (see FIG. 7, etc.), the image acquisition unit 110 may estimate the rotation angle of the image from the positional relationship between the detected eyelids and the iris and pupil.

[0064] Image rotation unit 160 is configured to be able to rotate the image acquired by image acquisition unit 110 based on the rotation angle estimated by rotation angle estimation unit 150. In other words, image rotation unit 160 is configured to be able to perform tilt correction of the image based on the estimated rotation angle. Image rotation unit 160 may have a function of storing the rotated image as a corrected image.

[0065] (Operation flow) Next, the flow of operation of the detection system 10 according to the fifth embodiment will be described with reference to Fig. 12. Fig. 12 is a flowchart showing the flow of operation of the detection system according to the fifth embodiment. Note that in Fig. 12, the same processes as those shown in Figs. 3, 6, and 9 are denoted by the same reference numerals.

[0066] 12, when the detection system 10 according to the fifth embodiment operates, the image acquisition unit 110 first acquires an image (step S101). Then, the detection unit 120 detects a characteristic figure corresponding to a first portion from the image acquired by the image acquisition unit 110 (step S102). The detection unit 120 further detects a characteristic point corresponding to a second portion from the image acquired by the image acquisition unit 110 (step S103).

[0067] Next, the rotation angle estimation unit 150 estimates the rotation angle of the image based on the detected feature points (step S401). Then, the image rotation unit 160 rotates the image according to the estimated rotation angle (step S402). In particular, the image rotation unit 160 rotates the image around the center of the approximate circle detected as the feature figure as the rotation axis.

[0068] (Specific operation example) Next, a specific example of operation (i.e., an example of operation of rotating an image) by the detection system 10 according to the fifth embodiment will be described with reference to Fig. 13. Fig. 13 is a diagram showing a specific example of operation by the detection system according to the fifth embodiment.

[0069] As shown in FIG. 13, the detection system 10 according to the fifth embodiment estimates the rotation angle of an image from the eyelid feature points. In the example shown in the figure, it can be seen that the image is tilted to the left (counterclockwise). The rotation angle can be calculated, for example, by comparing the position of a previously set normal feature point with the position of the currently detected feature point. However, existing technology can be used as appropriate to estimate the rotation angle based on the feature points.

[0070] Next, the image rotation unit 160 rotates the image by the estimated rotation angle. In the example shown in the figure, the image rotation unit 160 rotates the image to the right (clockwise). In particular, the image rotation unit 160 rotates the image around the center of the circle corresponding to the iris or pupil detected as the characteristic figure, which serves as the rotation axis. Note that if multiple characteristic figures are detected (for example, if the irises or pupils of both eyes are detected), the image rotation unit 160 may rotate the image around the center of one of the characteristic figures, which serves as the rotation axis. (Technical Effects) Next, the technical effects obtained by the detection system 10 according to the fifth embodiment will be described.

[0071] As described with reference to FIGS. 11 to 13 , in the detection system 10 according to the fifth embodiment, the rotation angle of the image is estimated based on the detected feature points, and the image is rotated around the center of the feature figure as the rotation axis. In this way, even if the image acquired by the image acquisition unit 110 is tilted, the tilt can be appropriately corrected. Note that the rotated image can also be used for, for example, the iris authentication described in the third embodiment or the gaze estimation described in the fourth embodiment. In this case, since the tilt is corrected by rotating the image, it is possible to perform iris authentication and gaze estimation with higher accuracy.

[0072] Sixth Embodiment A detection system 10 according to the sixth embodiment will be described with reference to Fig. 14 to Fig. 20. The sixth embodiment differs only in part of the configuration and operation from the above-described embodiments, and for example, the hardware configuration may be the same as that of the first embodiment (see Fig. 1). Therefore, in the following, descriptions of parts that overlap with the embodiments already described will be omitted as appropriate.

[0073] (Functional configuration) First, the functional configuration of the detection system 10 according to the sixth embodiment will be described with reference to Fig. 14. Fig. 14 is a block diagram showing the functional configuration of the detection system according to the sixth embodiment. Note that in Fig. 14, the same elements as those shown in Figs. 2, 5, 8, and 11 are denoted by the same reference numerals.

[0074] 14, the detection system 10 according to the sixth embodiment includes, as processing blocks for realizing its functions or as physical processing circuits, an image acquisition unit 110, a detection unit 120, and a display unit 170. That is, the detection system 10 according to the sixth embodiment is configured to further include the display unit 170 in addition to the components of the first embodiment (see FIG. 2).

[0075] The display unit 170 is configured as, for example, a monitor having a display. The display unit 170 may be configured as a part of the output device 16 shown in Fig. 1. The display unit 170 is configured to be able to display information about the characteristic figures and characteristic points detected by the detection unit 120. The display unit 170 may be configured to be able to change its display mode, for example, by operation by the system user.

[0076] (Operation flow) Next, the flow of operation of the detection system 10 according to the sixth embodiment will be described with reference to Fig. 15. Fig. 15 is a flowchart showing the flow of operation of the detection system according to the seventh embodiment. Note that in Fig. 15, the same processes as those shown in Figs. 3, 6, 9, and 12 are denoted by the same reference numerals.

[0077] 15, when the detection system 10 according to the sixth embodiment operates, the image acquisition unit 110 first acquires an image (step S101). Then, the detection unit 120 detects a characteristic figure corresponding to a first portion from the image acquired by the image acquisition unit 110 (step S102). The detection unit 120 further detects a characteristic point corresponding to a second portion from the image acquired by the image acquisition unit 110 (step S103).

[0078] Next, the display unit 170 displays information about the detected characteristic figures and characteristic points (step S501). The display unit 170 may display not only information directly related to the characteristic figures and characteristic points, but also information that can be estimated from the characteristic figures and characteristic points.

[0079] (Display example) Next, display examples of the detection system 10 according to the sixth embodiment will be described with reference to Fig. 16 to Fig. 20. Note that the display examples described below may be used in appropriate combination.

[0080] (1st display example) The first display example will be described with reference to Fig. 16. Fig. 16 is a diagram (part 1) showing a display example of feature points and feature figures on the display unit.

[0081] As shown in Fig. 16, the display unit 170 may display an image in which a characteristic graphic and characteristic points are superimposed on it. In this case, the display unit 170 may display only the characteristic graphic or only the characteristic points. The display unit 170 may be configured to switch between displaying and not displaying the characteristic graphic and characteristic points, for example, in response to a user operation. When a user operation is performed, the display unit 170 may display an operation button (i.e., a button for switching the display) below the image, etc.

[0082] The display unit 170 may further display information indicating the positions of the characteristic figures and the characteristic points (for example, the coordinates of the characteristic points, the formulas of the characteristic figures, etc.) in addition to the characteristic figures and the characteristic points. The display unit 170 may also display the characteristic figures and the characteristic points by coloring or drawing boundaries so that the ranges of the areas that can be identified from the characteristic figures and the characteristic points (in the illustrated example, the eyelid area, the iris area, and the pupil area) can be identified.

[0083] (2nd display example) The second display example will be described with reference to Fig. 17. Fig. 17 is a diagram (part 2) showing a display example of feature points and feature figures on the display unit.

[0084] As shown in Fig. 17, the display unit 170 may display the original image (i.e., the input image) and the detection result (i.e., an image in which the characteristic graphic and the characteristic points are drawn on the input image) side by side. In this case, the display unit 170 may display only either the original image or the detection result, for example, in response to a user operation. Furthermore, the display modes of the original image and the detection result may be changed separately.

[0085] (3rd display example) The third display example will be described with reference to Fig. 18. Fig. 18 is a diagram (part 3) showing a display example of feature points and feature figures on the display unit. Note that the display example of Fig. 18 is based on the second embodiment (i.e., a configuration including the iris authentication unit 130).

[0086] 18, the display unit 170 may display a registered image for iris authentication and the current captured image (i.e., an image in which a feature graphic and feature points are drawn on an input image) side by side. In this case, the display unit 170 may display only either the registered image or the captured image, for example, in response to a user operation. Also, the display modes of the registered image and the captured image may be changed separately.

[0087] (4th display example) The fourth display example will be described with reference to Fig. 19. Fig. 19 is a diagram (part 4) showing a display example of feature points and feature figures on a display unit. The display example of Fig. 19 is based on the third and fourth embodiments (i.e., a configuration including the gaze estimation unit 140).

[0088] 19, the display unit 170 may display the gaze direction estimation result in addition to the image in which the characteristic figures and characteristic points are superimposed. Specifically, the display unit 170 may display an arrow indicating the gaze direction as shown in the figure. In this case, the arrow may be displayed longer or larger as the deviation of the gaze from the front increases.

[0089] (5th display example) The fifth display example will be described with reference to Fig. 20. Fig. 20 is a diagram (part 5) showing a display example of feature points and feature figures on a display unit. The display example in Fig. 20 is based on the fifth embodiment (i.e., a configuration including a rotation angle estimation unit 150 and an image rotation unit 160).

[0090] As shown in Fig. 20, the display unit 170 may display an image before rotation (i.e., an image before tilt correction) and an image after rotation (i.e., an image after tilt correction) side by side. In this case, the display unit 170 may display only one of the image before rotation or the image after rotation, for example, in response to a user operation. Furthermore, the display mode of each of the image before rotation and the image after rotation may be changed separately.

[0091] (Technical Effects) Next, the technical effects obtained by the detection system 10 according to the sixth embodiment will be described.

[0092] 14 to 20, the detection system 10 according to the sixth embodiment displays information about the detected characteristic figures and characteristic points. Therefore, the detection results of the characteristic figures and characteristic points and the results of various processes using the characteristic figures and characteristic points can be presented to the user in an easy-to-understand manner.

[0093] Seventh Embodiment A detection system 10 according to the seventh embodiment will be described with reference to Figures 21 and 22. The seventh embodiment differs only in part of the configuration and operation from the above-described embodiments, and for example, the hardware configuration may be the same as that of the first embodiment (see Figure 1). Therefore, in the following, descriptions of parts that overlap with the embodiments already described will be omitted as appropriate.

[0094] (Functional configuration) First, the functional configuration of the detection system 10 according to the seventh embodiment will be described with reference to Fig. 21. Fig. 21 is a block diagram showing the functional configuration of the detection system according to the seventh embodiment. Note that in Fig. 21, the same elements as those shown in Figs. 2, 5, 8, 11, and 14 are denoted by the same reference numerals.

[0095] 21, the detection system 10 according to the seventh embodiment includes, as processing blocks for realizing its functions or as physical processing circuits, an image acquisition unit 110, a detection unit 120, and a learning unit 180. That is, the detection system 10 according to the seventh embodiment is configured to further include a learning unit 180 in addition to the components of the first embodiment (see FIG. 2).

[0096] The learning unit 180 is configured to be able to learn a model (e.g., a neural network model) for detecting characteristic figures and feature points. When learning is performed by the learning unit 180, the image acquisition unit 110 acquires images that are training data. Then, the learning unit 180 performs learning using the characteristic figures and feature points detected from the training data by the detection unit 120. That is, the learning unit 180 performs learning using the detected characteristic figures and feature points as a composite target. More specifically, the learning unit 180 performs learning by comparing correct answer data of the characteristic figures and feature points input as training data with the characteristic figures and feature points detected by the detection unit 120. The learning unit 180 may be configured to perform part of the learning process outside the system (e.g., on an external server, cloud, etc.).

[0097] The detection system 10 according to the seventh embodiment may have a function of expanding the input training data. For example, the image acquisition unit 110 may expand the data by changing the brightness, shifting vertically or horizontally, enlarging or reducing, rotating, or the like.

[0098] (Operation flow) Next, the flow of operation of the detection system 10 according to the seventh embodiment will be described with reference to Fig. 22. Fig. 22 is a flowchart showing the flow of operation of the detection system according to the seventh embodiment. Note that in Fig. 22, the same processes as those shown in Figs. 3, 6, 9, 12, and 15 are denoted by the same reference numerals.

[0099] 22, when the detection system 10 according to the seventh embodiment operates, the image acquisition unit 110 first acquires an image (step S101). Then, the detection unit 120 detects a characteristic figure corresponding to a first portion from the image acquired by the image acquisition unit 110 (step S102). The detection unit 120 further detects a characteristic point corresponding to a second portion from the image acquired by the image acquisition unit 110 (step S103).

[0100] Next, the learning unit 180 calculates an error function from the detected characteristic figures and characteristic points (step S601). Specifically, the learning unit 180 calculates the distance between a vector indicating the detected characteristic figures and characteristic points and a vector indicating the characteristic figures and characteristic points of the training data (i.e., correct answer data), thereby calculating the error between them. As a method for calculating the error, for example, the L1 norm or the L2 norm can be used, but other methods may also be used.

[0101] Next, the learning unit 180 performs error backpropagation based on the error and calculates the gradient of the parameters of the detection model (step S602). After that, the learning unit 180 updates (optimizes) the parameters of the detection model based on the calculated gradient (step S603). As an optimization method, for example, methods such as SDG (Stochastic Gradient Descent) and Adam can be used, but optimization may also be performed using other methods. When optimizing the parameters, the learning unit 180 may perform regularization such as weight decay. Furthermore, if the detection model is a neural network, it may include a layer that performs regularization such as dropout or batchnorm.

[0102] It should be noted that the series of learning processes described above (ie, steps S601 to S603) is merely an example, and learning may be performed using other methods as long as the feature figures and feature points can be used as composite targets.

[0103] Finally, the learning unit 180 determines whether learning has ended (step S604). The learning unit 180 determines whether learning has ended, for example, based on whether the processing up to this point has been looped a predetermined number of times. If it is determined that learning has ended (step S604: YES), the series of processing ends. On the other hand, if it is determined that learning has not ended (step S604: NO), the processing is repeated from step S101.

[0104] (Technical Effects) Next, the technical effects obtained by the detection system 10 according to the seventh embodiment will be described.

[0105] 21 and 22, in the detection system 10 according to the seventh embodiment, learning is performed using the feature graphics and feature points as composite targets. This makes it possible to optimize the detection model for the feature graphics and feature points and achieve more appropriate detection.

[0106] <Modification> Here, a modified example of the seventh embodiment will be described. Note that since the modified example has substantially the same configuration and operation as the seventh embodiment already described, the following will describe in detail only the parts that are different from the seventh embodiment, and will omit the description of other parts as appropriate.

[0107] In the detection system 10 according to the modified example, the learning unit 180 executes a learning process using information about the distribution of the relative positional relationships between feature figures and feature points. For example, the learning unit 180 uses the distribution of the positions of the iris detected as a feature figure and the positions of the eyelids detected as a feature point to learn a model for detecting feature figures and feature points.

[0108] If learning is not performed using the relative positional relationship between the feature shapes and feature points, the iris and eyelids will each be detected independently (i.e., the relative positional relationship will not be taken into consideration), which could result in parts that are not irises being detected as irises, or parts that are not eyelids being detected as eyelids.

[0109] However, according to the detection system 10 of the modified example, learning is performed taking into consideration the relative positional relationship between the characteristic figure and the characteristic points, making it possible to more appropriately optimize the detection model for the characteristic figure and the characteristic points.

[0110] <Additional Notes> The above-described embodiment may be further described as follows, but is not limited to the following.

[0111] (Appendix 1) The detection system described in Appendix 1 is characterized by comprising an acquisition means for acquiring an image including a living body, and a detection means for detecting a characteristic figure corresponding to a first, approximately circular part of the living body from the image, and detecting a characteristic point corresponding to a second part of the living body surrounding the first part.

[0112] (Appendix 2) The detection system described in Appendix 2 is the detection system described in Appendix 1, further comprising an iris authentication means for performing iris authentication processing on the living body based on the feature figure corresponding to at least one of the iris and pupil, which are the first part, and the feature point corresponding to the eyelid, which is the second part.

[0113] (Appendix 3) The detection system described in Appendix 3 is the detection system described in Appendix 1 or 2, further comprising a gaze estimation means that executes a gaze estimation process to estimate the gaze of the living body based on the feature figure corresponding to at least one of the iris and pupil, which are the first part, and the feature point corresponding to the eyelid, which is the second part.

[0114] (Appendix 4) The detection system described in Supplementary Note 4 is the detection system described in Supplementary Note 3, wherein the gaze estimation means estimates the gaze of the living body based on a relative positional relationship between the feature figure corresponding to at least one of the iris and the pupil and the feature point corresponding to the eyelid.

[0115] (Appendix 5) The detection system described in Supplementary Note 5 is the detection system described in any one of Supplementary Notes 1 to 4, further comprising: a rotation angle estimation means for estimating a rotation angle of the image using the feature points corresponding to the eyelids, which are the second part; and an image rotation means for rotating the image by the rotation angle around a rotation axis that is the center of the feature figure corresponding to at least one of the iris and pupils, which are the first part.

[0116] (Appendix 6) The detection system described in Supplementary Note 6 is the detection system described in any one of Supplementary Notes 1 to 5, further comprising a display means for displaying each of the feature points and the feature figures in a distinguishable display mode.

[0117] (Appendix 7) The detection system described in Appendix 7 is the detection system described in any one of Appendixes 1 to 6, further comprising a learning means that executes a learning process of the detection means using the feature points and feature figures detected from the image that is training data.

[0118] (Appendix 8) The detection system described in Appendix 8 is the detection system described in Appendix 6, characterized in that the learning means performs the learning process using information regarding the relative positional relationship between the characteristic figure and the characteristic point.

[0119] (Appendix 9) The detection method described in Appendix 9 is a detection method characterized by acquiring an image including a living body, detecting a feature figure corresponding to a first, approximately circular part of the living body from the image, and detecting a feature point corresponding to a second part of the living body surrounding the first part.

[0120] (Appendix 10) The computer program described in Appendix 10 is a computer program characterized by operating a computer to acquire an image including a living body, detect a characteristic figure corresponding to a first, approximately circular part of the living body from the image, and detect a characteristic point corresponding to a second part of the living body surrounding the first part.

[0121] (Appendix 11) The recording medium described in Supplementary Note 11 is a recording medium having the computer program described in Supplementary Note 10 recorded thereon.

[0122] This disclosure may be modified as appropriate within the scope of the claims and the gist or idea of ​​the invention that can be read from the entire specification, and detection systems, learning methods, and computer programs incorporating such modifications are also included in the technical idea of ​​this disclosure. [Explanation of symbols]

[0123] 10. Detection System 11 processors 110 Image acquisition unit 120 Detector 130 Iris Recognition Unit 140 Gaze estimation section 150 Rotation angle estimation unit 160 Image Rotation Unit 170 Display section 180 Learning Department

Claims

1. a detection model learning means for detecting a feature figure corresponding to the outline of an iris or a pupil and a feature point corresponding to the outline of an eyelid from an image including a living body; the learning means learns the detection model by learning the detection model to simultaneously detect the feature graphic and the feature points as a composite target. A detection system comprising:

2. 2. The detection system according to claim 1, wherein the characteristic figures correspond to the contours of the iris and pupil.

3. 3. The detection system according to claim 2, wherein the characteristic figures are a circle corresponding to the iris and a circle corresponding to the pupil.

4. At least one computer A method for learning a detection model for detecting a feature figure corresponding to an iris or pupil contour and a feature point corresponding to an eyelid contour from an image including a living body, the method comprising: training the detection model so that the detection model simultaneously detects the feature graphic and the feature points as a composite target; A learning method characterized by:

5. 5. The learning method according to claim 4, wherein said characteristic figures correspond to the contours of the iris and pupil.

6. 6. The learning method according to claim 5, wherein the characteristic figures are a circle corresponding to the iris and a circle corresponding to the pupil.

7. by at least one computer, A method for learning a detection model for detecting a feature figure corresponding to an iris or pupil contour and a feature point corresponding to an eyelid contour from an image including a living body, the method comprising: training the detection model so that the detection model simultaneously detects the feature graphic and the feature points as a composite target; A computer program that implements a learning method.

Images