DETECTION SYSTEM, DETECTION METHOD, AND COMPUTER PROGRAM

Abstract

This detection system (10) comprises: an acquisition means (110) for acquiring an image that includes a living organism; and a detection means (120) for detecting, from the image, a feature diagram that corresponds to an approximately circular first portion of the living organism, and detecting feature points that correspond to a second portion of the living organism in the periphery of the first portion. Such a detection system makes it possible to appropriately detect each of the first and second portions, which differ in terms of features of shapes.

Description

[Technical field]

[0001] The present disclosure relates to the technical fields of a detection system, a detection method, and a computer program for detecting a part of a living body from an image. [Background technology]

[0002] As this type of system, there is known a system that detects the area around the eyes of a living body from an image. For example, Patent Document 1 discloses detecting a pupil circle and an iris circle from an image. Patent Document 2 discloses detecting a face from an image and detecting the eyes from the position information of the face. Patent Document 3 discloses extracting feature points from a face image. Patent Document 4 discloses detecting a circular area from a ROI (Region Of Interest) and detecting multiple areas that are candidates for the iris. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2003-317102 A [Patent Document 2] JP 2007-213377 A [Patent Document 3] JP 2014-075098 A [Patent Document 4] JP 2018-045437 A Summary of the Invention [Problem to be solved by the invention]

[0004] In detection using an image as an input, a method of detecting feature points or feature shapes can be considered. However, the above cited documents do not mention the detection of both feature points and feature shapes, and there is room for improvement.

[0005] An object of the present disclosure is to provide a detection system, a detection method, and a computer program capable of solving the above-mentioned problems. [Means for solving the problem]

[0006] One aspect of the detection system of the present disclosure includes an acquisition means for acquiring an image including a living body, and a detection means for detecting a feature figure corresponding to a substantially circular first portion of the living body from the image, and detecting a feature point corresponding to a second portion of the living body surrounding the first portion. Equipped with.

[0007] One aspect of the detection method disclosed herein involves obtaining an image including a living body, detecting a characteristic figure from the image that corresponds to a substantially circular first portion of the living body, and detecting characteristic points that correspond to a second portion of the living body surrounding the first portion.

[0008] One aspect of the computer program of the present disclosure operates a computer to acquire an image including a living body, detect a characteristic figure from the image corresponding to a substantially circular first portion of the living body, and detect characteristic points corresponding to a second portion of the living body surrounding the first portion. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram showing a hardware configuration of a detection system according to a first embodiment. [Diagram 2] 1 is a block diagram showing a functional configuration of a detection system according to a first embodiment. [Diagram 3] 4 is a flowchart showing the flow of operations of the detection system according to the first embodiment. [Figure 4] 3A to 3C are diagrams illustrating an example of a characteristic pattern detected by the detection system according to the first embodiment. [Diagram 5] FIG. 11 is a block diagram showing a functional configuration of a detection system according to a second embodiment. [Figure 6]10 is a flowchart showing the flow of operations of a detection system according to a second embodiment. [Figure 7] 13A to 13C are diagrams illustrating an example of detection of characteristic patterns and characteristic points by the detection system according to the second embodiment. [Figure 8] FIG. 11 is a block diagram showing a functional configuration of a detection system according to a third embodiment. [Figure 9] 13 is a flowchart showing the flow of operations of the detection system according to the third embodiment. [Figure 10] FIG. 13 is a diagram illustrating an example of a gaze estimation method by a detection system according to a fourth embodiment. [Figure 11] FIG. 13 is a block diagram showing a functional configuration of a detection system according to a fifth embodiment. [Figure 12] 13 is a flowchart showing the flow of operations of the detection system according to the fifth embodiment. [Figure 13] FIG. 13 is a diagram showing a specific example of the operation of the detection system according to the fifth embodiment. [Figure 14] FIG. 13 is a block diagram showing a functional configuration of a detection system according to a sixth embodiment. [Figure 15] 13 is a flowchart showing the flow of operations of the detection system according to the sixth embodiment. [Figure 16] FIG. 13 is a diagram (part 1) showing an example of displaying characteristic points and characteristic figures on a display unit. [Figure 17] FIG. 2 is a diagram (part 2) showing an example of displaying characteristic points and characteristic figures on the display unit. [Figure 18] FIG. 11 is a diagram (part 3) showing an example of displaying characteristic points and characteristic figures on the display unit. [Figure 19] FIG. 4 is a diagram showing a fourth example of displaying characteristic points and characteristic figures on the display unit. [Figure 20] FIG. 5 is a diagram showing a fifth example of displaying characteristic points and characteristic figures on the display unit. [Figure 21] FIG. 13 is a block diagram showing a functional configuration of a detection system according to a seventh embodiment. [Figure 22] 20 is a flowchart showing the flow of operations of the detection system according to the seventh embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, embodiments of a detection system, a detection method, and a computer program will be described with reference to the drawings.

[0011] First Embodiment A detection system according to a first embodiment will be described with reference to FIGS. 1 to 4. FIG.

[0012] (Hardware configuration) First, the hardware configuration of the detection system according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the hardware configuration of the detection system according to the first embodiment.

[0013] 1, the detection system 10 according to the first embodiment includes a processor 11, a RAM (Random Access Memory) 12, a ROM (Read Only Memory) 13, and a storage device 14. The detection system 10 may further include an input device 15 and an output device 16. The processor 11, the RAM 12, the ROM 13, the storage device 14, the input device 15, and the output device 16 are connected via a data bus 17.

[0014] The processor 11 reads a computer program. For example, the processor 11 is configured to read a computer program stored in at least one of the RAM 12, the ROM 13, and the storage device 14. Alternatively, the processor 11 may read a computer program stored in a computer-readable storage medium using a storage medium reading device (not shown). The processor 11 may acquire (i.e., read) a computer program from a device (not shown) disposed outside the detection system 10 via a network interface. The processor 11 controls the RAM 12, the storage device 14, the input device 15, and the output device 16 by executing the computer program that the processor 11 reads. In particular, in this embodiment, when the processor 11 executes the computer program that the processor 11 reads, a functional block for detecting a part of a living body from an image is realized in the processor 11. In addition, as the processor 11, one of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), an FPGA (field-programmable gate array), a DSP (Demand-Side Platform), and an ASIC (Application Specific Integrated Circuit) may be used, or a plurality of them may be used in parallel.

[0015] The RAM 12 temporarily stores a computer program executed by the processor 11. The RAM 12 temporarily stores data that is temporarily used by the processor 11 when the processor 11 is executing the computer program. The RAM 12 may be, for example, a D-RAM (Dynamic RAM).

[0016] The ROM 13 stores computer programs executed by the processor 11. The ROM 13 may also store other fixed data. The ROM 13 may be, for example, a programmable ROM (P-ROM).

[0017] The storage device 14 stores data to be stored long-term by the detection system 10. The storage device 14 may operate as a temporary storage device for the processor 11. The storage device 14 may include, for example, at least one of a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and a disk array device.

[0018] The input device 15 is a device that receives an input instruction from a user of the detection system 10. The input device 15 may include, for example, at least one of a keyboard, a mouse, and a touch panel.

[0019] The output device 16 is a device that outputs to the outside information related to the detection system 10. For example, the output device 16 may be a display device (for example, a display) that can display information related to the detection system 10.

[0020] (Functional configuration) Next, the functional configuration of the detection system 10 according to the first embodiment will be described with reference to Fig. 2. Fig. 2 is a block diagram showing the functional configuration of the detection system according to the first embodiment.

[0021] 2, a detection system 10 according to the first embodiment is configured to detect a part of a living body from an image. The detection system 10 includes an image acquisition unit 110 and a detection unit 120 as a processing block for realizing the function or as a physical processing circuit. The image acquisition unit 110 and the detection unit 120 can be realized by, for example, the above-mentioned processor 11 (see FIG. 1).

[0022] The image acquiring unit 110 is configured to be able to acquire an image (i.e., an image to be detected) input to the detection system 10. The image acquiring unit 110 may include a storage unit that stores the acquired image. Information related to the image acquired by the image acquiring unit 110 is configured to be output to the detection unit 120.

[0023] The detection unit 120 is configured to be able to detect a part of the living body from the image acquired by the image acquisition unit 110. Specifically, the detection unit 110 is configured to be able to detect a feature figure corresponding to a first part of the living body and a feature point corresponding to a second part of the living body. The "first part" here is a part of the living body having a substantially circular shape. On the other hand, the "second part" is a part located around the first part of the living body. It is only necessary to set in advance which part of the living body is the first part and which part is the second part. In this case, a plurality of parts of different types may be set as the first part, and a plurality of parts of different types may be set as the second part. The detection unit 120 may have a function of outputting information on the detected feature points and feature figures.

[0024] (Operation flow) Next, the flow of operations of the detection system 10 according to the first embodiment will be described with reference to Fig. 3. Fig. 3 is a flowchart showing the flow of operations of the detection system according to the first embodiment.

[0025] As shown in FIG. 3, when the detection system 10 according to the first embodiment operates, first, the image acquisition unit 110 acquires an image (step S101).

[0026] Next, the detection unit 120 detects a characteristic figure corresponding to the first portion from the image acquired by the image acquisition unit 110 (step S102). The detection unit 120 further detects a characteristic point corresponding to the second portion from the image acquired by the image acquisition unit 110 (step S103). The detected characteristic figure and characteristic point can be expressed by coordinates, a mathematical formula, or the like. Note that the processes of steps S102 and S103 may be executed one after the other, or may be executed simultaneously in parallel. In other words, the order of detection of the characteristic figure corresponding to the first portion and the characteristic point corresponding to the second portion is not limited, and the characteristic figure and the characteristic point may be detected simultaneously.

[0027] (Specific examples of characteristic shapes) Next, a specific example of a characteristic graphic detected by the detection system 10 according to the first embodiment will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of a characteristic graphic detected by the detection system according to the first embodiment.

[0028] As shown in FIG. 4, in the detection system 10 according to the first embodiment, the detection unit 120 detects a circle (including an ellipse) as a characteristic figure corresponding to the second portion. The circle detected by the detection unit 120 may include a vertically elongated ellipse, a horizontally elongated ellipse, and a diagonal ellipse (i.e., an ellipse rotated at an arbitrary angle) in addition to a perfect circle. The type of circle to be actually detected may be set in advance. For example, a shape corresponding to the part of the living body to be detected may be set. The detection unit 120 may be configured to be capable of detecting a circle with a missing part or a circle with a hidden part.

[0029] (Technical effect) Next, technical effects obtained by the detection system 10 according to the first embodiment will be described.

[0030] As described in Fig. 1 to Fig. 3, in the detection system 10 according to the first embodiment, a characteristic figure corresponding to the first portion and a characteristic point corresponding to the second portion are detected from an image. That is, the first portion and the second portion are detected by different methods. This makes it possible to appropriately detect the first portion and the second portion, which have different shape characteristics, from an image of a living body.

[0031] <Second embodiment> A detection system 10 according to the second embodiment will be described with reference to Fig. 5 to Fig. 7. The second embodiment differs from the first embodiment only in some configurations and operations, and for example, the hardware configuration may be the same as that of the first embodiment (see Fig. 1). Therefore, in the following, descriptions of parts that overlap with the embodiments already described will be omitted as appropriate.

[0032] (Functional configuration) First, the functional configuration of the detection system 10 according to the second embodiment will be described with reference to Fig. 5. Fig. 5 is a block diagram showing the functional configuration of the detection system according to the second embodiment. In Fig. 5, the same components as those shown in Fig. 2 are denoted by the same reference numerals.

[0033] As shown in Fig. 5, the detection system 10 according to the second embodiment includes an image acquisition unit 110, a detection unit 120, and an iris authentication unit 130 as processing blocks for realizing the functions thereof or as physical processing circuits. That is, the detection system 10 according to the second embodiment is configured to further include the iris authentication unit 130 in addition to the components of the first embodiment (see Fig. 2). The iris authentication unit 130 can be realized by, for example, the above-mentioned processor 11 (see Fig. 1).

[0034] The iris authentication unit 130 is configured to be able to perform iris authentication using the feature points and feature figures detected by the detection unit 120. The iris authentication unit 130 is configured to specify an iris area based on, for example, feature points corresponding to the eyelids, which are an example of the first part, and feature figures (see FIG. 7) corresponding to the iris and pupils, which are an example of the second part, and to perform iris authentication processing using the iris area. The iris authentication unit 130 may have a function of outputting the result of iris authentication. In addition, the iris authentication unit 130 may be configured to perform part of the iris authentication processing outside the system (for example, on an external server, cloud, etc.).

[0035] (Operation flow) Next, the flow of operations of the detection system 10 according to the second embodiment will be described with reference to Fig. 6. Fig. 6 is a flowchart showing the flow of operations of the detection system according to the second embodiment. In Fig. 6, the same processes as those shown in Fig. 3 are denoted by the same reference numerals.

[0036] 6, when the detection system 10 according to the second embodiment operates, the image acquisition unit 110 first acquires an image (step S101). After that, the detection unit 120 detects a characteristic figure corresponding to the first portion from the image acquired by the image acquisition unit 110 (step S102). The detection unit 120 further detects a characteristic point corresponding to the second portion from the image acquired by the image acquisition unit 110 (step S103).

[0037] Next, the iris authentication unit 130 identifies the eyelid area (i.e., the area where the eyelids are present) from the feature points corresponding to the eyelids, and generates a mask for the eyelid area (step S201). The eyelid area mask is used to remove the eyelid area that is not necessary for iris authentication (in other words, does not have iris information). After that, the iris authentication unit 130 identifies the iris area (i.e., the area from which iris information can be obtained) from the feature figures corresponding to the iris and pupil, and executes iris authentication using the iris area (step S202). Note that the specific processing content of iris authentication can be appropriately adopted from existing technologies, so a detailed description thereof will be omitted here. (Example of detection around the eyes) Next, detection of a characteristic graphic and a characteristic point by the detection system 10 according to the second embodiment will be described with reference to Fig. 7. Fig. 7 is a diagram showing an example of detection of a characteristic graphic and a characteristic point by the detection system according to the second embodiment.

[0038] As shown in FIG. 7, in the detection system 10 according to the second embodiment, the "iris and pupil" is detected as the first part, and the "eyelid" is detected as the second part.

[0039] The detection unit 120 detects a circle corresponding to the iris and a circle corresponding to the pupil. The detection unit 120 may detect only one of the circle corresponding to the iris and the circle corresponding to the pupil. The iris and the pupil are suitable for detection as characteristic figures of approximately circles because their shapes are close to circles. If the iris and the pupil are to be detected as characteristic points (for example, points on a circumference), the number and positions of the points depend on the system design, which directly affects the detection accuracy. However, if the iris and the pupil are detected as circles, the positions of the iris and the pupil can be determined as the formula of the circle. Since the formula of the circle is uniquely determined, it does not depend on the system design and does not affect the detection accuracy. From these points of view, it can be said that the iris is suitable for detection as a circle.

[0040] The detection unit 120 detects a plurality of feature points indicating the position (contour) of the eyelids. In the example shown in the figure, the detection unit 120 detects two feature points corresponding to the inner corner and outer corner of the eye, three feature points corresponding to the upper eyelid, and three feature points corresponding to the lower eyelid. The number of feature points described above is merely an example, and fewer or more feature points may be detected. Eyelids vary relatively greatly among individuals, and there are large differences in shape between individuals, such as single eyelids, double eyelids, slanted eyes, droopy eyes, and the like. Therefore, eyelids are suitable for detection as feature points rather than as feature figures. Although eyelids vary in shape between individuals, they are commonly located around the iris and pupil. Therefore, if they are detected together with feature figures, they can be relatively easily detected as feature points.

[0041] (Technical effect) Next, technical effects obtained by the detection system 10 according to the second embodiment will be described.

[0042] As described in Fig. 5 to Fig. 7, in the detection system 10 according to the second embodiment, iris authentication is performed using the detected feature figures and feature points. In particular, in this embodiment, since a plurality of parts around the eyes are properly detected, it is possible to properly perform iris authentication. In the example shown in Fig. 5, since the coordinates of the inner corner and outer corner of the eye and the radii of the circles of the iris and pupil are known, the ratio of the difference between the distance from the inner corner to the outer corner of the eye and the radii of the two circles may be used for data matching or weighting in iris authentication.

[0043] <Third embodiment> A detection system 10 according to the third embodiment will be described with reference to Fig. 8 and Fig. 9. The third embodiment differs from the above-mentioned embodiments only in part of the configuration and operation, and for example, the hardware configuration may be the same as that of the first embodiment (see Fig. 1). Therefore, in the following, the description of the parts that overlap with the embodiments already described will be omitted as appropriate.

[0044] (Functional configuration) First, the functional configuration of the detection system 10 according to the third embodiment will be described with reference to Fig. 8. Fig. 8 is a block diagram showing the functional configuration of the detection system according to the third embodiment. In Fig. 8, the same components as those shown in Figs. 2 and 5 are denoted by the same reference numerals.

[0045] As shown in Fig. 8, the detection system 10 according to the fifth embodiment includes an image acquisition unit 110, a detection unit 120, and a gaze estimation unit 140 as processing blocks for realizing the functions thereof or as physical processing circuits. That is, the detection system 10 according to the third embodiment is configured to further include a gaze estimation unit 140 in addition to the components of the first embodiment (see Fig. 2). Note that the gaze estimation unit 140 can be realized by, for example, the above-mentioned processor 11 (see Fig. 1).

[0046] The gaze estimation unit 140 is configured to be capable of estimating the gaze direction using the feature points and feature figures detected by the detection unit 120. Specifically, the gaze estimation unit 140 is configured to be capable of estimating the gaze direction based on the "feature points corresponding to the eyelids" and the "feature figures corresponding to the iris and pupil" (see FIG. 7) described in the second embodiment. Note that the feature points corresponding to the eyelids and the feature figures corresponding to the iris and pupil may be detected as described in the second embodiment. The gaze estimation unit 140 may have a function of outputting the result of the gaze estimation. In addition, the gaze estimation unit 140 may be configured to execute a part of the gaze estimation process outside the system (for example, on an external server, cloud, etc.).

[0047] (Operation flow) Next, the flow of operations of the detection system 10 according to the third embodiment will be described with reference to Fig. 9. Fig. 9 is a flowchart showing the flow of operations of the detection system according to the third embodiment. In Fig. 9, the same processes as those shown in Figs. 3 and 6 are denoted by the same reference numerals.

[0048] 9, when the detection system 10 according to the third embodiment operates, the image acquisition unit 110 first acquires an image (step S101). After that, the detection unit 120 detects a characteristic figure corresponding to the first portion from the image acquired by the image acquisition unit 110 (step S102). The detection unit 120 further detects a characteristic point corresponding to the second portion from the image acquired by the image acquisition unit 110 (step S103).

[0049] Next, the gaze direction estimation unit 140 estimates the gaze direction based on the feature graphic and the feature points (step S301). Note that the specific processing content of the gaze direction estimation will be described in detail in the fourth embodiment described later.

[0050] (Technical effect) Next, technical effects obtained by the detection system 10 according to the third embodiment will be described.

[0051] 8 and 9, in the detection system 10 according to the third embodiment, gaze estimation is performed using the detected feature figures and feature points. In particular, in this embodiment, since the parts used for estimating the gaze direction (for example, parts around the eyes) are appropriately detected, it is possible to appropriately estimate the gaze direction.

[0052] <Fourth embodiment> A detection system 10 according to the fourth embodiment will be described with reference to Fig. 10. Note that the fourth embodiment shows a more specific configuration of the above-mentioned third embodiment (i.e., a specific method for estimating the gaze direction), and the configuration and operation flow may be the same as those of the third embodiment (see Figs. 8 and 9). Therefore, in the following, descriptions of parts that overlap with parts already explained will be omitted as appropriate.

[0053] (Calculation of relative position) A method of estimating a gaze direction by the detection system 10 according to the fourth embodiment will be described with reference to Fig. 10. Fig. 10 is a diagram showing an example of a method of estimating a gaze direction by the detection system according to the fourth embodiment.

[0054] As shown in Fig. 10, the detection system 10 according to the fourth embodiment estimates the gaze direction using feature points corresponding to the eyelids and feature figures corresponding to the iris and pupil. Note that the feature points corresponding to the eyelids and the feature figures corresponding to the iris and pupil may be detected as described in the second embodiment (see Fig. 7).

[0055] Feature points 1 and 2 in the figure are feature points corresponding to the inner corner and outer corner of the eye, respectively. Furthermore, feature points 3 and 4 are points on the midline of feature points 1 and 2 that intersect with the eyelid. For this reason, feature points 1 to 4 will be in the same position no matter which direction the eyeball faces, unless the direction of the face changes. Note that if the direction of the eyeball changes, circles 5 and 6, which are feature figures corresponding to the iris and pupil, move. Therefore, the direction of the eyeball (i.e., the gaze direction) can be estimated by using the relative relationship between the position of the eyelid, which can be identified from each feature point, and the position of the iris and pupil, which can be identified from each feature figure.

[0056] To estimate the gaze direction, it is necessary to obtain in advance the relationship between the relative relationship between the eyelid position and the eyeball position, and the current position at which the person is looking. This relationship may be calculated as a function, or may be created as a table.

[0057] When calculating the gaze direction, the image is first normalized using circle 6, which corresponds to the iris. Next, the intersection point of the line connecting feature point 1 and feature point 2 with the line connecting feature point 3 and feature point 4 is set as the origin, and the relative positional relationship between the eyelid and the eyeball is calculated from information on how many pixels away from this origin in the x and y directions. The gaze direction is then estimated using the calculated positional relationship between the eyelid and the eyeball.

[0058] (Technical effect) Next, technical effects obtained by the detection system 10 according to the fourth embodiment will be described.

[0059] 10, in the detection system 10 according to the fourth embodiment, the gaze direction is estimated from the positional relationship between the eyelids and the eyeball. In particular, in this embodiment, the relative positional relationship between the eyelids and the iris and pupil can be appropriately calculated from the feature points corresponding to the eyelids and the feature figures corresponding to the iris and pupil, so that the gaze direction can be appropriately estimated.

[0060] <Fifth embodiment> A detection system 10 according to the fifth embodiment will be described with reference to Fig. 11 to Fig. 13. The fifth embodiment differs from the above-mentioned embodiments only in part of the configuration and operation, and for example, the hardware configuration may be the same as that of the first embodiment (see Fig. 1). Therefore, in the following, the description of the parts that overlap with the embodiments already described will be omitted as appropriate.

[0061] (Functional configuration) First, the functional configuration of the detection system 10 according to the fifth embodiment will be described with reference to Fig. 11. Fig. 11 is a block diagram showing the functional configuration of the detection system according to the fourth embodiment. In Fig. 11, the same elements as those shown in Figs. 2, 5, and 8 are denoted by the same reference numerals.

[0062] 11, the detection system 10 according to the fifth embodiment includes an image acquisition unit 110, a detection unit 120, a rotation angle estimation unit 150, and an image rotation unit 160 as processing blocks for realizing the functions thereof or as physical processing circuits. That is, the detection system 10 according to the fifth embodiment is configured to further include the rotation angle estimation unit 150 and the image rotation unit 160 in addition to the components of the first embodiment (see FIG. 2). Note that the rotation angle estimation unit 150 and the image rotation unit 160 can be realized by, for example, the above-mentioned processor 11 (see FIG. 1).

[0063] The rotation angle estimation unit 150 can estimate the rotation angle (i.e., tilt) of the image acquired by the image acquisition unit 110 based on the feature points detected by the detection unit 120. For example, as shown in the second embodiment, when the feature points of the eyelids are detected (see FIG. 7, etc.), the image acquisition unit 110 estimates the rotation angle of the image from the tilt of the detected eyelids. The rotation angle estimation unit 150 may estimate the rotation angle of the image taking into account the feature figures in addition to the feature points detected by the detection unit 120. For example, as shown in the second embodiment, when the feature points of the eyelids and the feature figures of the iris or pupil are detected (see FIG. 7, etc.), the image acquisition unit 110 may estimate the rotation angle of the image from the positional relationship between the detected eyelids and the iris and pupil.

[0064] Image rotation section 160 is configured to be capable of rotating the image acquired by image acquisition section 110 based on the rotation angle estimated by rotation angle estimation section 150. That is, image rotation section 160 is configured to be capable of executing tilt correction of the image based on the estimated rotation angle. Image rotation section 160 may have a function of storing the rotated image as a corrected image.

[0065] (Operation flow) Next, the flow of operations of the detection system 10 according to the fifth embodiment will be described with reference to Fig. 12. Fig. 12 is a flowchart showing the flow of operations of the detection system according to the fifth embodiment. In Fig. 12, the same processes as those shown in Figs. 3, 6, and 9 are denoted by the same reference numerals.

[0066] 12, when the detection system 10 according to the fifth embodiment operates, the image acquisition unit 110 first acquires an image (step S101). After that, the detection unit 120 detects a characteristic figure corresponding to the first portion from the image acquired by the image acquisition unit 110 (step S102). The detection unit 120 further detects a characteristic point corresponding to the second portion from the image acquired by the image acquisition unit 110 (step S103).

[0067] Next, the rotation angle estimation unit 150 estimates the rotation angle of the image based on the detected feature points (step S401). Then, the image rotation unit 160 rotates the image according to the estimated rotation angle (step S402). In particular, the image rotation unit 160 rotates the image around the center of the approximate circle detected as the feature figure as the rotation axis.

[0068] (Specific operation example) Next, a specific operation example (i.e., an operation example of rotating an image) by the detection system 10 according to the fifth embodiment will be described with reference to Fig. 13. Fig. 13 is a diagram showing a specific operation example by the detection system according to the fifth embodiment.

[0069] As shown in FIG. 13, in the detection system 10 according to the fifth embodiment, the rotation angle of the image is estimated from the feature points of the eyelids. In the example shown in the figure, it can be seen that the image is tilted to the left (counterclockwise). The numerical value of the rotation angle can be calculated, for example, by comparing the position of the feature points in a normal state that is set in advance with the position of the feature points detected this time. However, as a method for estimating the rotation angle based on the feature points, it is possible to adopt existing technology as appropriate.

[0070] Next, the image rotation unit 160 rotates the image by the estimated rotation angle. In the example shown in the figure, the image rotation unit 160 rotates the image to the right (clockwise). In particular, the image rotation unit 160 rotates the image around the center of a circle corresponding to the iris or pupil detected as a characteristic figure, as the rotation axis. Note that when multiple characteristic figures are detected (for example, when the irises or pupils of both eyes are detected), the image rotation unit 160 may rotate the image around the center of one of the characteristic figures as the rotation axis. (Technical effect) Next, technical effects obtained by the detection system 10 according to the fifth embodiment will be described.

[0071] As described in FIG. 11 to FIG. 13, in the detection system 10 according to the fifth embodiment, the rotation angle of the image is estimated based on the detected feature points, and the image is rotated around the center of the feature figure as the rotation axis. In this way, even if the image acquired by the image acquisition unit 110 is tilted, the tilt can be appropriately corrected. Note that the rotated image can also be used for, for example, the iris authentication described in the third embodiment or the gaze estimation described in the fourth embodiment. In this case, since the tilt is corrected by rotating the image, it is possible to perform iris authentication and gaze estimation with higher accuracy.

[0072] Sixth Embodiment A detection system 10 according to the sixth embodiment will be described with reference to Fig. 14 to Fig. 20. The sixth embodiment differs from the above-mentioned embodiments only in part of the configuration and operation, and for example, the hardware configuration may be the same as that of the first embodiment (see Fig. 1). Therefore, in the following, the description of parts that overlap with the embodiments already described will be omitted as appropriate.

[0073] (Functional configuration) First, the functional configuration of the detection system 10 according to the sixth embodiment will be described with reference to Fig. 14. Fig. 14 is a block diagram showing the functional configuration of the detection system according to the sixth embodiment. In Fig. 14, the same elements as those shown in Figs. 2, 5, 8, and 11 are denoted by the same reference numerals.

[0074] 14, the detection system 10 according to the sixth embodiment includes an image acquisition unit 110, a detection unit 120, and a display unit 170 as processing blocks for realizing the functions thereof or as physical processing circuits. That is, the detection system 10 according to the sixth embodiment is configured to further include a display unit 170 in addition to the components of the first embodiment (see FIG. 2).

[0075] The display unit 170 is configured, for example, as a monitor having a display. The display unit 170 may be configured as a part of the output device 16 shown in Fig. 1. The display unit 170 is configured to be capable of displaying information on the characteristic figures and characteristic points detected by the detection unit 120. The display unit 170 may be configured to be capable of changing the display mode, for example, by an operation by a system user.

[0076] (Operation flow) Next, the flow of operations of the detection system 10 according to the sixth embodiment will be described with reference to Fig. 15. Fig. 15 is a flowchart showing the flow of operations of the detection system according to the seventh embodiment. In Fig. 15, the same processes as those shown in Figs. 3, 6, 9, and 12 are denoted by the same reference numerals.

[0077] 15, when the detection system 10 according to the sixth embodiment operates, the image acquisition unit 110 first acquires an image (step S101). After that, the detection unit 120 detects a characteristic figure corresponding to the first portion from the image acquired by the image acquisition unit 110 (step S102). The detection unit 120 further detects a characteristic point corresponding to the second portion from the image acquired by the image acquisition unit 110 (step S103).

[0078] Next, the display unit 170 displays information about the detected characteristic figures and characteristic points (step S501). The display unit 170 may display not only information directly related to the characteristic figures and characteristic points, but also information that can be estimated from the characteristic figures and characteristic points.

[0079] (Example) Next, display examples of the detection system 10 according to the sixth embodiment will be described with reference to Fig. 16 to Fig. 20. Note that the display examples described below may be used in appropriate combination.

[0080] (1st display example) The first display example will be described with reference to Fig. 16. Fig. 16 is a diagram (part 1) showing a display example of characteristic points and characteristic figures on the display unit.

[0081] As shown in Fig. 16, the display unit 170 may display an image with a characteristic figure and characteristic points superimposed thereon. In this case, the display unit 170 may display only the characteristic figure or only the characteristic points. The display unit 170 may switch between drawing the characteristic figure and the characteristic points and not drawing them, for example, in response to a user operation. When a user operation is performed, the display unit 170 may display an operation button (i.e., a button for switching the display) below the image, etc.

[0082] The display unit 170 may further display information indicating the positions of the characteristic figures and the characteristic points (for example, the coordinates of the characteristic points, the formula of the characteristic figures, etc.) in addition to the characteristic figures and the characteristic points. The display unit 170 may also display the characteristic figures and the characteristic points by painting them with colors or drawing boundaries so that the ranges of the areas that can be specified from the characteristic figures and the characteristic points (in the illustrated example, the eyelid area, the iris area, and the pupil area) can be identified.

[0083] (2nd display example) The second display example will be described with reference to Fig. 17. Fig. 17 is a diagram (part 2) showing a display example of characteristic points and characteristic figures on the display unit.

[0084] As shown in Fig. 17, the display unit 170 may display an original image (i.e., an input image) and a detection result (i.e., an image in which a characteristic figure and characteristic points are drawn on the input image) side by side. In this case, the display unit 170 may display only either the original image or the detection result, for example, in response to a user operation. Also, the display mode of each of the original image and the detection result may be changed separately.

[0085] (3rd display example) The third display example will be described with reference to Fig. 18. Fig. 18 is a diagram (part 3) showing a display example of characteristic points and characteristic figures on a display unit. Note that the display example of Fig. 18 is based on the second embodiment (i.e., a configuration including the iris authentication unit 130).

[0086] As shown in Fig. 18, the display unit 170 may display a registered image for iris authentication and a current captured image (i.e., an image in which a characteristic figure and characteristic points are drawn on an input image) side by side. In this case, the display unit 170 may display only either the registered image or the captured image, for example, in response to a user operation. Also, the display mode of each of the registered image and the captured image may be changed separately.

[0087] (4th display example) The fourth display example will be described with reference to Fig. 19. Fig. 19 is a diagram (part 4) showing a display example of feature points and feature figures on a display unit. Note that the display example of Fig. 19 is based on the third and fourth embodiments (i.e., a configuration including the gaze estimation unit 140).

[0088] As shown in Fig. 19, the display unit 170 may display the estimated result of the gaze direction in addition to the image with the characteristic figures and feature points superimposed thereon. Specifically, the display unit 170 may display an arrow indicating the gaze direction as shown in the figure. In this case, the arrow may be displayed longer or larger as the deviation of the gaze from the front increases.

[0089] (5th display example) The fifth display example will be described with reference to Fig. 20. Fig. 20 is a diagram (part 5) showing a display example of feature points and feature figures on a display unit. Note that the display example of Fig. 20 is based on the fifth embodiment (i.e., a configuration including a rotation angle estimation unit 150 and an image rotation unit 160).

[0090] As shown in Fig. 20, the display unit 170 may display an image before rotation (i.e., an image before tilt correction) and an image after rotation (i.e., an image after tilt correction) side by side. In this case, the display unit 170 may display only one of the image before rotation or the image after rotation, for example, in response to a user operation. Also, the display mode of each of the image before rotation and the image after rotation may be changed separately.

[0091] (Technical effect) Next, technical effects obtained by the detection system 10 according to the sixth embodiment will be described.

[0092] 14 to 20, the detection system 10 according to the sixth embodiment displays information about the detected characteristic figures and characteristic points. Therefore, the detection results of the characteristic figures and characteristic points, and the results of various processes using the characteristic figures and characteristic points can be presented to the user in an easily understandable manner.

[0093] Seventh embodiment A detection system 10 according to the seventh embodiment will be described with reference to Fig. 21 and Fig. 22. The seventh embodiment differs from the above-mentioned embodiments only in part of the configuration and operation, and for example, the hardware configuration may be the same as that of the first embodiment (see Fig. 1). Therefore, in the following, the description of parts that overlap with the embodiments already described will be omitted as appropriate.

[0094] (Functional configuration) First, the functional configuration of the detection system 10 according to the seventh embodiment will be described with reference to Fig. 21. Fig. 21 is a block diagram showing the functional configuration of the detection system according to the seventh embodiment. In Fig. 21, the same reference numerals are used to designate the same elements as those shown in Figs. 2, 5, 8, 11, and 14.

[0095] 21, the detection system 10 according to the seventh embodiment includes an image acquisition unit 110, a detection unit 120, and a learning unit 180 as processing blocks for realizing the functions thereof or as physical processing circuits. That is, the detection system 10 according to the seventh embodiment is configured to further include a learning unit 180 in addition to the components of the first embodiment (see FIG. 2).

[0096] The learning unit 180 is configured to be able to learn a model (e.g., a neural network model) for detecting a characteristic figure and a characteristic point. When learning is performed by the learning unit 180, the image acquisition unit 110 acquires an image that is training data. Then, the learning unit 180 performs learning using the characteristic figure and the characteristic point detected by the detection unit 120 from the training data. That is, the learning unit 180 performs learning with the detected characteristic figure and the characteristic point as a composite target. More specifically, the learning unit 180 performs learning by comparing the correct answer data of the characteristic figure and the characteristic point input as training data with the characteristic figure and the characteristic point detected by the detection unit 120. The learning unit 180 may be configured to perform a part of the learning process outside the system (e.g., on an external server, a cloud, etc.).

[0097] The detection system 10 according to the seventh embodiment may have a function of expanding the training data to be input. For example, the image acquisition unit 110 may expand the data by changing the brightness, shifting vertically and horizontally, enlarging or reducing the size, rotating, or the like.

[0098] (Operation flow) Next, the flow of operations of the detection system 10 according to the seventh embodiment will be described with reference to Fig. 22. Fig. 22 is a flowchart showing the flow of operations of the detection system according to the seventh embodiment. In Fig. 22, the same processes as those shown in Figs. 3, 6, 9, 12, and 15 are denoted by the same reference numerals.

[0099] 22, when the detection system 10 according to the seventh embodiment operates, the image acquisition unit 110 first acquires an image (step S101). After that, the detection unit 120 detects a characteristic figure corresponding to the first portion from the image acquired by the image acquisition unit 110 (step S102). The detection unit 120 further detects a characteristic point corresponding to the second portion from the image acquired by the image acquisition unit 110 (step S103).

[0100] Next, the learning unit 180 calculates an error function from the detected feature figures and feature points (step S601). Specifically, the learning unit 180 calculates the distance between a vector indicating the detected feature figures and feature points and a vector indicating the feature figures and feature points of the training data (i.e., correct answer data), thereby calculating the error between them. As a method for calculating the error, for example, the L1 norm or the L2 norm can be used, but other methods may also be used.

[0101] Next, the learning unit 180 performs error backpropagation based on the error and calculates the gradient of the parameters of the detection model (step S602). After that, the learning unit 180 updates (optimizes) the parameters of the detection model based on the calculated gradient (step S603). As an optimization method, for example, a method such as SDG (Stochastic Gradient Descent) or Adam can be used, but optimization may be performed using other methods. When optimizing the parameters, the learning unit 180 may perform regularization such as weight decay. Furthermore, when the detection model is a neural network, it may include a layer that performs regularization such as dropout or batchnorm.

[0102] It should be noted that the series of learning processes described above (that is, steps S601 to S603) is merely an example, and learning may be performed using other methods as long as the feature figures and feature points can be used as composite targets.

[0103] Finally, the learning unit 180 determines whether learning has been completed (step S604). The learning unit 180 determines whether learning has been completed, for example, based on whether the process up to this point has been looped a predetermined number of times. If it is determined that learning has been completed (step S604: YES), the series of processes ends. On the other hand, if it is determined that learning has not been completed (step S604: NO), the process is repeated from step S101.

[0104] (Technical effect) Next, technical effects obtained by the detection system 10 according to the seventh embodiment will be described.

[0105] 21 and 22, in the detection system 10 according to the seventh embodiment, learning is performed using the characteristic figures and the characteristic points as a composite target. Therefore, it is possible to optimize the detection model of the characteristic figures and the characteristic points and realize more appropriate detection.

[0106] <Modification> Here, a modified example of the seventh embodiment described above will be described. Note that since the modified example has substantially the same configuration and operation as the seventh embodiment already described, the following will describe in detail the parts that are different from the seventh embodiment, and will appropriately omit the description of the other parts.

[0107] In the detection system 10 according to the modified example, the learning unit 180 executes a learning process using information on the distribution of the relative positional relationship between the feature figure and the feature point. For example, the learning unit 180 learns a model for detecting the feature figure and the feature point using the distribution of the positions of the iris detected as the feature figure and the positions of the eyelids detected as the feature point.

[0108] If learning were not performed using the relative positional relationship between the feature shapes and feature points, the iris and eyelids would each be detected independently (i.e., the relative positional relationship would not be taken into consideration), and there is a possibility that a part that is not the iris would be detected as the iris, or a part that is not the eyelid would be detected as the eyelid.

[0109] However, according to the detection system 10 of the modified example, learning is performed taking into account the relative positional relationship between the characteristic figure and the characteristic points, making it possible to more appropriately optimize the detection model for the characteristic figure and the characteristic points.

[0110] <Additional Notes> The above-described embodiment may be further described as follows, but is not limited to the following.

[0111] (Appendix 1) The detection system described in Appendix 1 is a detection system characterized by comprising an acquisition means for acquiring an image including a living body, and a detection means for detecting a characteristic figure corresponding to a substantially circular first part of the living body from the image, and detecting a characteristic point corresponding to a second part surrounding the first part of the living body.

[0112] (Appendix 2) The detection system described in Appendix 2 is the detection system described in Appendix 1, further comprising an iris authentication means for executing iris authentication processing on the living body based on the characteristic figure corresponding to at least one of the iris and pupil, which are the first part, and the characteristic point corresponding to the eyelid, which is the second part.

[0113] (Appendix 3) The detection system described in Appendix 3 is the detection system described in Appendix 1 or 2, further comprising a gaze estimation means for executing a gaze estimation process to estimate the gaze of the living body based on the feature figure corresponding to at least one of the iris and the pupil which are the first part, and the feature point corresponding to the eyelid which is the second part.

[0114] (Appendix 4) The detection system described in Appendix 4 is the detection system described in Appendix 3, characterized in that the gaze estimation means estimates the gaze of the living body based on a relative positional relationship between the feature figure corresponding to at least one of the iris and the pupil, and the feature point corresponding to the eyelid.

[0115] (Appendix 5) The detection system described in Appendix 5 is the detection system described in any one of Appendixes 1 to 4, further comprising: a rotation angle estimation means for estimating a rotation angle of the image using the feature point corresponding to the second part, that is, the eyelid; and an image rotation means for rotating the image by the rotation angle around a rotation axis that is a center of the feature figure corresponding to at least one of the first part, that is, the iris and the pupil.

[0116] (Appendix 6) The detection system described in Supplementary Note 6 is the detection system described in any one of Supplementary Notes 1 to 5, further comprising a display means for displaying each of the feature points and the feature figures in a distinguishable display mode.

[0117] (Appendix 7) The detection system described in Appendix 7 is the detection system described in any one of Appendixes 1 to 6, further comprising a learning means for executing a learning process of the detection means using the feature points and the feature figures detected from the image, which is training data.

[0118] (Appendix 8) The detection system described in Appendix 8 is the detection system described in Appendix 6, characterized in that the learning means performs the learning process using information regarding the relative positional relationship between the characteristic figure and the characteristic point.

[0119] (Appendix 9) The detection method described in Appendix 9 is a detection method characterized by acquiring an image including a living body, detecting a characteristic figure corresponding to a substantially circular first part of the living body from the image, and detecting a characteristic point corresponding to a second part of the living body surrounding the first part.

[0120] (Appendix 10) The computer program described in Appendix 10 is a computer program characterized by operating a computer to obtain an image including a living body, detect a characteristic figure corresponding to a substantially circular first part of the living body from the image, and detect a characteristic point corresponding to a second part of the living body surrounding the first part.

[0121] (Appendix 11) The recording medium according to claim 11 is a recording medium having the computer program according to claim 10 recorded thereon.

[0122] This disclosure may be modified as appropriate without departing from the spirit or concept of the invention as can be read from the claims and the entire specification, and detection systems, detection methods, and computer programs incorporating such modifications are also included in the technical concept of this disclosure. [Explanation of symbols]

[0123] 10. Detection System 11 Processors 110 Image acquisition unit 120 Detection unit 130 Iris Recognition Unit 140 Gaze estimation section 150 Rotation angle estimation unit 160 Image Rotation Unit 170 Display section 180 Learning Department

Claims

1. An acquisition means for acquiring an image including a living body; a detection means for simultaneously detecting, from the image, a feature figure corresponding to at least one of an iris and a pupil of the living body and a feature point corresponding to an eyelid of the living body as a composite target; A detection system comprising:

2. a learning means for learning a detection model configured with a neural network so as to simultaneously detect the characteristic figure and the characteristic point as a composite target from the image, 2. The detection system according to claim 1, wherein said detection means detects said feature graphic and said feature points simultaneously by using said detection model learned by said learning means.

3. 3. The detection system according to claim 2, wherein the learning means learns a detection model constituted by a neural network using information regarding a distribution of relative positional relationships between the characteristic figure and the characteristic points so as to simultaneously detect the characteristic figure and the characteristic points as a composite target from the image.

4. A detection system described in any one of claims 1 to 3, characterized in that it further comprises an iris authentication means for performing iris authentication processing on the living body based on the characteristic figure and the characteristic points.

5. A detection system described in any one of claims 1 to 4, characterized in that it further comprises a gaze estimation means for performing a gaze estimation process to estimate the gaze of the living body based on the characteristic figure and the characteristic points.

6. 6. The detection system according to claim 5, wherein the gaze estimating means estimates the gaze of the living body based on a relative positional relationship between the characteristic figure and the characteristic point.

7. A rotation angle estimation means for estimating a rotation angle of the image using the feature points; an image rotation means for rotating the image by the rotation angle with the center of the characteristic figure as a rotation axis; 7. The detection system according to claim 1, further comprising:

8. 8. The detection system according to claim 1, further comprising a display unit that displays each of the feature points and the feature graphics in a distinguishable display mode.

9. Acquire an image including a living body; A feature figure corresponding to at least one of an iris and a pupil of the living body and a feature point corresponding to an eyelid of the living body are simultaneously detected as a composite target from the image. A detection method comprising:

10. A detection model configured with a neural network is trained so as to simultaneously detect the characteristic figure and the characteristic points as a composite target from the image; 10. The detection method according to claim 9, further comprising the steps of: detecting the feature pattern and the feature points simultaneously using the detection model.

11. 11. The detection method according to claim 10, further comprising: learning a detection model formed of a neural network using information regarding a distribution of relative positional relationships between the characteristic figure and the characteristic points so as to simultaneously detect the characteristic figure and the characteristic points as a composite target from the image.

12. Acquire an image including a living body; A feature figure corresponding to at least one of an iris and a pupil of the living body and a feature point corresponding to an eyelid of the living body are simultaneously detected as a composite target from the image. A computer program for causing a computer to operate in such a manner.

13. A detection model configured with a neural network is trained so as to simultaneously detect the characteristic figure and the characteristic points as a composite target from the image; 13. The computer program according to claim 12, wherein the detection model is used to simultaneously detect the feature graphic and the feature points.

14. The computer program according to claim 13, further comprising: learning a detection model configured with a neural network using information regarding a distribution of relative positional relationships between the characteristic figure and the characteristic points so as to simultaneously detect the characteristic figure and the characteristic points as a composite target from the image.

Images