Image processing device, image processing method, and program

Abstract

To provide an image processing device capable of accurately determining the center position of a pupil.SOLUTION: An image processing device detects object pixels 50 having a luminance value included in a luminance range corresponding to a pupil and being continuously adjacent to each other, from each frame of an image in which an eye of a subject person 3 is captured. The image processing device extracts, from among a plurality of contour pixels of the object pixels 50, a first pixel 61 where a coordinate value of an axis along a lateral direction of the subject person 3 is the minimum, a second pixel 62 where a coordinate value of the axis is the maximum, one or more third pixels 63 positioned below the first pixel 61, and one or more fourth pixels 64 positioned below the second pixel 62. The image processing device obtains an approximate circle 70 of the first pixel 61, the second pixel 62, the one or more third pixels 63, and the one or more fourth pixels 64, and determines a center 71 of the approximate circle 70 as a center position of the pupil.SELECTED DRAWING: Figure 9

Description

[Technical Field]

[0001] This disclosure relates to an image processing apparatus, an image processing method, and a program. [Background technology]

[0002] Traditionally, dizziness disorders have been diagnosed and differentiated by analyzing eye movements (hereinafter also referred to as nystagmus) from images of the subject's eyes. Images are obtained, for example, by photographing the eyes illuminated with infrared light using Frenchell glasses. However, since only a portion of the eyeball is recorded in the image, the central position of the pupil is usually identified, and nystagmus is analyzed from the trajectory of the identified central position.

[0003] Japanese Patent Publication No. 2002-000567 (Patent Document 1) discloses an image processing method for identifying the central position of a pupil in an image. This image processing method involves, for each image, arbitrarily detecting three contour points that constitute the outline of the pupil, and performing the steps of calculating the center coordinates of a circle passing through the three detected contour points multiple times, thereby identifying the centroid coordinates of the obtained group of center coordinates as the central position of the pupil. [Prior art documents] [Patent Documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2002-000567 [Overview of the project] [Problems that the invention aims to solve]

[0005] In the image processing method described in Patent Document 1, three contour points that constitute the outline of the pupil are arbitrarily detected. However, the entire pupil is not always visible in each frame of the image. In particular, in elderly people, part of the pupil may be covered by drooping eyelids. In such cases, the identification method described in Patent Document 1 may detect the boundary point between the pupil and the eyelid covering the pupil as a contour point. The detection of such contour points reduces the accuracy of the identified central position of the pupil.

[0006] This disclosure has been made in view of the above circumstances, and its purpose is to provide an image processing device, an image processing method, and a program that can accurately identify the central position of the pupil. [Means for solving the problem]

[0007] An image processing device relating to one aspect of this disclosure comprises an acquisition unit, a detection unit, an extraction unit, and a pupil position determination unit. The acquisition unit acquires an image of the subject's eye. The detection unit detects a group of target pixels from the image, in which pixels having a brightness value included in the brightness range corresponding to the pupil are consecutive. The extraction unit extracts from a plurality of contour pixels constituting the contour of the group of target pixels the first pixel whose axis coordinate value is smallest along the left-right direction of the subject, the second pixel whose axis coordinate value is largest, one or more third pixels located below the first pixel, and one or more fourth pixels located below the second pixel. The pupil position determination unit finds an approximate circle of the first pixel, the second pixel, one or more third pixels, and one or more fourth pixels, and determines the center of the approximate circle as the center position of the pupil.

[0008] The image processing method according to one aspect of the present disclosure includes four steps. The first step is to obtain an image showing the eyes of the subject. The second step is to detect a group of target pixels in the image in which pixels having luminance values included in the luminance range corresponding to the pupil are continuous. The third step is to extract, from among a plurality of contour pixels constituting the contour of the group of target pixels, a first pixel having the minimum coordinate value of the axis along the left - right direction of the subject, a second pixel having the maximum coordinate value of the axis, one or more third pixels located below the first pixel, and one or more fourth pixels located below the second pixel. The fourth step is to calculate an approximate circle from the first pixel, the second pixel, the one or more third pixels, and the one or more fourth pixels, and determine the center of the approximate circle as the center position of the pupil.

[0009] A program according to one aspect of the present disclosure causes a computer to execute the above - mentioned image processing method.

Advantages of the Invention

[0010] According to the present disclosure, the center position of the pupil can be accurately specified.

Brief Description of the Drawings

[0011] [Figure 1] It is a diagram showing the configuration of the system according to the embodiment. [Figure 2] It is a diagram showing an example of the hardware configuration of the image processing apparatus shown in FIG. 1. [Figure 3] It is a diagram showing an example of the functional configuration of the image processing apparatus shown in FIG. 1. [Figure 4] It is a diagram showing an example of the extraction process of pixels having luminance values included in the luminance range corresponding to the pupil. [Figure 5] It is a diagram showing an example of the labeling process. [Figure 6] It is a diagram showing an example of the group of target pixels detected by the detection unit. [Figure 7] It is a diagram showing another example of the group of target pixels detected by the detection unit. [Figure 8] It is a diagram for explaining the process of the extraction unit. [Figure 9] It is a diagram for explaining the processing of the pupil position determination unit. [Figure 10] It is a diagram showing an example of a user interface screen provided by the providing unit. [Figure 11] It is a diagram showing another example of a user interface screen provided by the providing unit. [Figure 12] It is a flowchart showing the processing flow of the image processing apparatus. [Figure 13] It is a diagram showing an example of the center position of the pupil specified by the image processing method according to the first reference form. [Figure 14] It is a diagram for explaining the correction processing performed in the image processing method according to the first reference form.

Embodiments for Carrying Out the Invention

[0012] Embodiments of the present invention will be described in detail with reference to the drawings. For the same or corresponding parts in the drawings, the same reference numerals are given and the description thereof will not be repeated.

[0013] (Reference Form) Before explaining the embodiments of the present invention, a reference form of an image processing method for specifying the center position of the pupil and its problems will be described. As an image processing method for specifying the center position of the pupil, in addition to the method described in Patent Document 1, image processing methods according to the first and second reference forms can be considered.

[0014] The image processing method according to the first reference form includes the first to third processes. The first process is a process of setting a luminance range in which pixels where the pupil appears in the video can take values according to a user operation. The second process is a process of detecting pixels having luminance values within the set luminance range from each frame of the video. The third process is a process of specifying the area centroid of the detected pixel group as the center position of the pupil.

[0015] Figure 13 shows an example of the pupil center position identified by the image processing method according to the first reference embodiment. In the example shown in Figure 13, the pixel group 150 detected by the second processing includes not only pixels that show the pupil, but also pixels that show the area near the outer corner of the eye. This is because the area near the outer corner of the eye is in the shadow of the eyelid, and the brightness value of the pixels that show this area is included in the brightness range set in the first processing. Because the pixel group 150 includes pixels that show the area near the outer corner of the eye, the center position 171 identified in the third processing is shifted from the center of the pupil. Thus, in the case of the image processing method according to the first reference embodiment, a problem arises in that the accuracy of identifying the pupil center position may decrease.

[0016] Furthermore, if part of the pupil is covered by the eyelid, the area centroid of the pixel group corresponding to the remaining part of the pupil is identified as the center position of the pupil. As a result, the accuracy of identifying the center position of the pupil decreases. Therefore, it is conceivable to exclude frames showing pupils partially covered by the eyelid as error frames. However, in this case, in images of elderly subjects with drooping eyelids, the number of frames excluded as error frames increases, leading to the problem of being unable to properly diagnose and differentiate dizziness disorders.

[0017] As shown in Figure 13, the image may contain areas 180 with increased brightness due to reflected infrared light. If these areas 180 are included within the region where the pupil is captured, the accuracy of identifying the pupil's center position decreases. In addition, generally, the brightness value of pixels capturing the outer edge of the pupil is higher than that of pixels capturing the center of the pupil. Therefore, the brightness value of pixels capturing the outer edge of the pupil may fall outside the brightness range set by the first processing, and the pixels capturing the outer edge may not be detected as part of the pixel group 150. As a result, the accuracy of identifying the pupil's center position decreases. Taking these factors into consideration, in the image processing method according to the first reference embodiment, it is conceivable to perform a correction process after the second processing to correct the pixel group 150 detected from the target frame using frames before and after the target frame.

[0018] Figure 14 is a diagram illustrating the correction process. The upper part of Figure 14 shows the target frame before the correction process is performed, and the lower part of Figure 14 shows the target frame after the correction process is performed.

[0019] As shown in the upper part of Figure 14, within the area where the pupil is captured, there is a portion 180 where the brightness is increased due to reflected infrared light. Furthermore, there is a portion 182 where the brightness value is higher than the brightness range, even though the outer edge of the pupil is captured. Therefore, in the second processing, only the pixels included in the region where portions 180 and 182 are missing from the area where the pupil is captured are detected as a group of pixels 150 with brightness values ​​within the brightness range. The area centroid of the pixel group 150 shown in the upper part of Figure 14 is offset from the center position of the pupil. Therefore, when the third processing is performed using the pixel group 150 shown in the upper part of Figure 14, the accuracy of identifying the center position of the pupil decreases.

[0020] Therefore, a correction process is performed to fill in the missing parts by overlaying the outline of the pupil in the preceding and succeeding frames and applying correlation processing. As a result, as shown in the lower part of Figure 14, the pixel group 150 is corrected to match the area in which the pupil is captured. By performing a third process using the corrected pixel group 150, the accuracy of identifying the center position of the pupil is improved.

[0021] However, the computational load required for the correction process is high. This leads to the problem that it takes a long time to determine the center position of the pupil.

[0022] The image processing method according to the second reference embodiment uses AI (Artificial Intelligence) including high-resolution image processing, high-performance computing, and high-load logic to convert a frame showing a pupil partially covered by eyelids, eyelashes, etc., into a frame showing the entire pupil, and then uses the converted frame to determine the central position of the pupil. In order to realize the image processing method according to the second reference embodiment, it is necessary to prepare a large amount of training data and a high-performance computer.

[0023] In recent years, the demand for home-based medical care has increased, and research and practical application of home-based medical care support systems are progressing. However, it is difficult to bring high-performance computers into home-based medical care settings. Therefore, the image processing method related to the second reference form has the problem of being difficult to apply to home-based medical care settings.

[0024] The image processing apparatus, image processing method, and program according to the embodiment can also resolve these problems.

[0025] (System Configuration) Figure 1 is a diagram showing the configuration of a system according to an embodiment. As shown in Figure 1, the system comprises an image processing device 1 and goggles 2.

[0026] Goggles 2 are worn on the head of subject 3. Goggles 2 are, for example, Frenchell glasses and include an infrared light and an image sensor. In night vision conditions, Goggles 2 illuminates the eyes of subject 3 wearing Goggles 2 with infrared light and images the eyes of the subject 3.

[0027] Furthermore, the goggles 2 include a communication device 201. The communication device 201 transmits video data obtained by imaging (hereinafter simply referred to as "video") to the image processing device 1. The communication device 201 may transmit the video to the image processing device 1 using a wireless communication method, or it may transmit the video to the image processing device 1 using a wired communication method.

[0028] Image processing device 1 is a low-cost, low-spec computer (e.g., a smartphone, tablet, or notebook computer). Image processing device 1 acquires video from goggles 2 and performs image processing to determine the central position of the pupil in each frame of the acquired video.

[0029] (Hardware configuration of the image processing device) Figure 2 shows an example of the hardware configuration of the image processing device shown in Figure 1. As shown in Figure 2, the image processing device 1 comprises a processor 101, RAM (Random Access Memory) 102, storage 103, a communication interface 104, a display 105, and an input device 106. These components are connected to each other via a bus 107 so that they can communicate data with one another.

[0030] The processor 101 consists of a CPU (Central Processing Unit), an MPU (Micro Processing Unit), and the like. The processor 101 performs various processes according to this embodiment by loading programs stored in the storage 103 into the RAM 102 and executing them. The RAM 102 is typically a volatile storage device such as DRAM (Dynamic Random Access Memory) and stores programs read from the storage 103. The storage 103 is typically a non-volatile magnetic storage device such as a hard disk drive. The storage 103 stores the image processing program 110 executed by the processor 101.

[0031] The communication interface (communication IF) 104 exchanges data with an external device (including the goggles 2) using either wireless or wired communication methods.

[0032] The display 105 is, for example, a liquid crystal display. The input device 106 includes, for example, a touchpad, mouse, keyboard, etc. The display 105 and the input device 106 may be an integrated touch panel.

[0033] (Functional configuration of image processing equipment) The functional configuration of the image processing device will be described with reference to Figures 3 to 11. Figure 3 is a diagram showing an example of the functional configuration of the image processing device shown in Figure 1. As shown in Figure 3, the image processing device 1 comprises a storage unit 10, an acquisition unit 11, a detection unit 12, an extraction unit 13, a pupil position determination unit 14, and a providing unit 15. The storage unit 10 is realized by the storage 103 and RAM 102 shown in Figure 2. The acquisition unit 11 is realized by a communication interface 104 and a processor 101 that executes an image processing program 110. The detection unit 12, the extraction unit 13, and the pupil position determination unit 14 are realized by the processor 101 executing the image processing program 110. The providing unit 15 is realized by a display 105, an input device 106, and a processor 101 that executes the image processing program 110.

[0034] The acquisition unit 11 acquires an image of the subject's eyes from the goggles 2. The image shows the brightness value for each pixel. Specifically, the acquisition unit 11 acquires a video 20 of the subject's eyes. Each frame 21 of the video 20 is an image of the subject's eyes. The acquisition unit 11 stores the acquired video 20 in the storage unit 10. The storage unit 10 can store one or more videos 20.

[0035] The detection unit 12 reads the designated video 20 from the storage unit 10 and detects a group of adjacent pixels from each frame 21 of the read video 20, where pixels having a brightness value within the brightness range corresponding to the pupil are adjacent to each other. That is, each pixel in the group of adjacent pixels is adjacent to another pixel in the group of adjacent pixels, located in one of the following directions: up, down, left, right, upper left, upper right, lower left, or lower right. The brightness range is set in advance according to user operation. In other words, the user can set the brightness range by checking the brightness values ​​that the pixel representing the pupil can take.

[0036] The processing of the detection unit 12 will be explained with reference to Figures 4 to 7. Figure 4 is a diagram showing an example of the process of extracting pixels having a brightness value that falls within the brightness range corresponding to the pupil. As shown in Figure 4, the detection unit 12 extracts pixels 42 that have a brightness value that falls within the brightness range, sets the value of the extracted pixels 42 to "1", and generates a binary image 44 by setting the values ​​of the remaining pixels to "0".

[0037] Figure 5 shows an example of labeling processing. As shown in Figure 5, the detection unit 12 performs a known labeling process on the binary image 44 and identifies one or more candidate pixel groups 40 in which pixels having the value "1" are adjacent and consecutive. In the example shown in Figure 5, candidate pixel groups 40a to 40c are identified.

[0038] The detection unit 12 selects a group of candidate pixels 40 from among one or more candidate pixel groups 40 whose area falls within a predetermined area range as the target pixel group 50. The area range is predetermined according to user operation. As an upper limit of the area range, for example, a value greater than the area occupied by the pupil in an image showing the entire pupil is set. As a lower limit of the area range, for example, the area occupied by 30% of the pupil in frame 21 is set. The area is expressed, for example, by the number of pixels.

[0039] Figure 6 shows an example of a target pixel group detected by the detection unit. Figure 6 shows a frame 21 in which the entire pupil is visible. As shown in Figure 6, the pixel group in which the pupil is visible is detected as the target pixel group 50.

[0040] Figure 7 shows another example of a target pixel group detected by the detection unit. Figure 7 shows a frame 21 in which the pupil is captured with its upper half covered by the eyelid. In the example shown in Figure 7, three candidate pixel groups 40d to 40f are identified. Candidate pixel group 40d corresponds to the area in which the lower half of the pupil is captured. Candidate pixel group 40e corresponds to a small area near the outer corner of the eye that is in the shadow of the eyelid. Candidate pixel group 40f corresponds to a small area of ​​the pupil. Of the candidate pixel groups 40d to 40f, only candidate pixel group 40d has an area that falls within a predetermined area range. Therefore, candidate pixel group 40d, which corresponds to the area in which the lower half of the pupil is captured, is determined to be the target pixel group 50.

[0041] The extraction unit 13 extracts pixels from among a plurality of contour pixels that constitute the contour of the target pixel group 50 to be used to identify the center position of the pupil. Contour pixels are pixels included in the target pixel group 50 that are adjacent to the surrounding pixels of the target pixel group 50.

[0042] Figure 8 is a diagram illustrating the processing of the extraction unit. The position of a pixel in frame 21 is represented by coordinates on the X-axis, which is aligned vertically with the subject 3, and the Y-axis, which is aligned horizontally with the subject 3. The coordinates of the top-left pixel in frame 21 are (0,0). The positive direction of the X-axis corresponds to the downward direction of the subject 3, and the positive direction of the Y-axis corresponds to the leftward direction of the subject 3. If the number of pixels in the X-axis direction of frame 21 is M and the number of pixels in the Y-axis direction is N, the coordinates of a pixel in frame 21 are represented as (m, n). m is an integer from 0 to M-1, and n is an integer from 0 to N-1.

[0043] The extraction unit 13 generates a binary image 52 in which the values ​​of pixels included in the target pixel group 50 are "1" and the values ​​of pixels not included in the target pixel group 50 are "0". Using the binary image 52, the extraction unit 13 identifies a plurality of contour pixels 60 that constitute the contour of the target pixel group 50. Specifically, in the binary image 52, the extraction unit 13 scans the Y coordinate from 0 to N-1 along each line X=k (where k is an integer from 0 to M-1), and identifies the pixels that have the value "1" when the value changes from "0" to "1", and the pixels that have the value "1" when the value changes from "1" to "0", as contour pixels 60.

[0044] Next, the extraction unit 13 extracts from the plurality of contour pixels 60 the first pixel 61 with the minimum Y-axis coordinate value, the second pixel 62 with the maximum Y-axis coordinate value, one or more third pixels 63 located below the first pixel 61, and one or more fourth pixels 64 located below the second pixel 62.

[0045] Specifically, the extraction unit 13 identifies the X coordinate p of the first pixel 61. Next, the extraction unit 13 selects target lines parallel to the Y axis in the order X=p+1 to M-1. The extraction unit 13 extracts contour pixels 60 on the target lines as third pixels 63 if their distance from the first pixel 61 or a third pixel 63 already extracted is less than or equal to a predetermined threshold (for example, a distance of 3 pixels). The extraction unit 13 repeats the target line selection process until a predetermined number (for example, 50) of third pixels 63 have been extracted.

[0046] Similarly, the extraction unit 13 identifies the X coordinate q of the second pixel 62. Next, the extraction unit 13 selects target lines parallel to the Y axis in the order X=q+1 to M-1. The extraction unit 13 extracts contour pixels 60 on the target lines as fourth pixels 64 if their distance from the second pixel 62 or the already extracted fourth pixel 64 is less than or equal to a predetermined threshold (for example, a distance of 3 pixels). The extraction unit 13 repeats the target line selection process until a predetermined number (for example, 50) of fourth pixels 64 have been extracted.

[0047] FIG. 9 is a diagram for explaining the processing of the pupil positioning unit. As shown in FIG. 9, the pupil positioning unit 14 obtains an approximate circle 70 of the first pixel 61, the second pixel 62, one or more third pixels 63, and one or more fourth pixels 64 extracted by the extraction unit 13, and determines the center 71 of the approximate circle 70 as the center position of the pupil. The pupil positioning unit 14 may calculate the coordinates of the center 71 of the approximate circle 70 using the least squares method.

[0048] When the total number of pixels of the first pixel 61, the second pixel 62, the third pixel 63, and the fourth pixel 64 is P, and the coordinates of the first pixel 61, the second pixel 62, the third pixel 63, and the fourth pixel 64 are (x i , y i )(where i is an integer from 1 to P), the coordinates (a, b) of the center of the approximate circle are calculated as follows.

[0049] When the first pixel 61, the second pixel 62, the third pixel 63, and the fourth pixel 64 are points on the circumference of a circle centered at the coordinates (a, b), the equation of the circle is represented by Equation (1). r is the radius. (x i - a) 2 + (y i - b) 2 - r 2 = 0 ··· Equation (1)

[0050] By transforming Equation (2) obtained by squaring both sides of Equation (1), Equation (3) is obtained. {(x <o000008>- a) 2 + (y i - b) 2 - r 2} 2 = 0 ··· Equation (2) {x i 2 + y i 2 + Ax i + By i + C} 2 = 0 ··· Equation (3)

[0051] A, B, and C in Equation (3) are as follows, respectively. A = -2a It should be noted that there seems to be a small error in the original text where "<o000008>" should probably be " i ". This has been left as is in the translation to maintain consistency with the original. B = -2b C=a 2 +b 2 -r 2

[0052] By summing equation (3) over i=1 to P and taking partial derivatives with respect to A, B, and C, we obtain equations (4) to (6). Equations (4) to (6) can be transformed into equations (7) to (9), respectively.

[0053]

number

[0054]

number

[0055]

number

[0056]

number

[0057] Transforming equation (11), S=M -1 N...Formula (12) M can be obtained. -1 This is the inverse matrix of matrix M.

[0058] The pupil position determination unit 14 calculates A, B, and C using equation (12). Then, the pupil position determination unit 14 a = -A / 2 b = -B / 2 According to this, calculate the coordinates (a,b) of the center 71 of the approximate circle 70.

[0059] The pupil position determination unit 14 generates trajectory data 16 indicating coordinates (a,b) determined from each frame 21 of the video 20, and stores the trajectory data 16 in the storage unit 10.

[0060] The provisioning unit 15 generates a user interface screen and provides the generated user interface screen. Specifically, the provisioning unit 15 displays the user interface screen on the display 105.

[0061] Figure 10 shows an example of a user interface screen provided by the provider. As shown in Figure 10, the user interface screen 80 includes input fields 81, 82a, 82b, a load button 83, and an area 84.

[0062] The input field 81 is used to set the brightness range. The user enters the upper limit of the brightness range in the input field 81. The supply unit 15 sets the brightness range to a range less than or equal to the value entered in the input field 81.

[0063] Input fields 82a and 82b are used to set the area range. The user enters the upper limit of the area range in input field 82a and the lower limit of the area range in input field 82b. The supply unit 15 sets the area range to be greater than or equal to the value entered in input field 82b and less than or equal to the value entered in input field 82a.

[0064] The load button 83 is used to specify the video 20. When the load button 83 is pressed, the supply unit 15 displays a window on the display 105 that shows a list of video 20 stored in the storage unit 10. The supply unit 15 designates the video 20 selected from the list as the analysis target and starts processing for the analysis target in the detection unit 12, extraction unit 13, and pupil position determination unit 14.

[0065] In region 84, the video 20 to be analyzed is played back. Furthermore, the providing unit 15 overlays the first pixel 61, second pixel 62, third pixel 63, and fourth pixel 64 extracted from the frame 21, along with the approximate circle 70 and the center 71 of the approximate circle 70, onto the frame 21 displayed in region 84.

[0066] The providing unit 15 may display a trajectory line 72 indicating the trajectory of the center 71 in the area 84. Furthermore, the providing unit 15 may display text 73 indicating the area of ​​the target pixel group 50 detected from the frame 21 in the area 84.

[0067] Figure 11 shows another example of a user interface screen provided by the provider unit. As shown in Figure 11, the user interface screen 90 includes a graph 91 showing the trajectory of the pupil's center position. The provider unit 15 creates the graph 91 based on the trajectory data 16 stored in the storage unit 10.

[0068] Medical professionals can determine the movement of the pupil's central position in subject 3 by examining area 84 shown in Figure 10 or graph 91 shown in Figure 11. As a result, medical professionals can diagnose and differentiate dizziness disorders.

[0069] (Processing flow of an image processing device) Figure 12 is a flowchart showing the processing flow of the image processing device. As shown in Figure 12, the processor 101 acquires an image of the subject's eyes from the goggles 2 (step S1).

[0070] Next, the processor 101 detects a group of target pixels 50 from each frame (image) of the acquired video in which pixels having a brightness value included in the brightness range corresponding to the pupil are adjacent to each other and are consecutive (step S2).

[0071] Next, the processor 101 extracts from among the multiple contour pixels 60 that constitute the contour of the target pixel group 50 the first pixel 61 whose Y-axis coordinate value is smallest along the left-right direction of the subject 3, the second pixel 62 whose Y-axis coordinate value is largest, one or more third pixels 63 located below the first pixel 61, and one or more fourth pixels 64 located below the second pixel 62 (step S3).

[0072] Next, the processor 101 calculates an approximate circle 70 of the first pixel 61, the second pixel 62, one or more third pixels 63, and one or more fourth pixels 64, and determines the center of the approximate circle 70 as the center position of the pupil (step S4).

[0073] Finally, the processor 101 provides the analysis results, including the trajectory of the pupil's center position (step S5). Specifically, the processor 101 displays the analysis results on the display 105.

[0074] (advantage) As described above, the image processing apparatus 1 according to this embodiment comprises an acquisition unit 11, a detection unit 12, an extraction unit 13, and a pupil position determination unit 14. The acquisition unit 11 acquires video (including multiple frames (images)) showing the eyes of the subject 3. The detection unit 12 detects a group of target pixels 50 from each frame, in which pixels having a brightness value included in the brightness range corresponding to the pupil are consecutive. The extraction unit 13 extracts from a plurality of contour pixels 60 constituting the contour of the group of target pixels 50 a first pixel 61 whose Y-axis coordinate value is smallest along the left-right direction of the subject 3, a second pixel 62 whose Y-axis coordinate value is largest, one or more third pixels 63 located below the first pixel 61, and one or more fourth pixels 64 located below the second pixel 62. The pupil position determination unit 14 determines an approximate circle 70 of the first pixel 61, the second pixel 62, one or more third pixels 63, and one or more fourth pixels 64, and determines the center 71 of the approximate circle 70 as the center position of the pupil.

[0075] According to the above configuration, the target pixel group 50 has a brightness value that falls within the brightness range corresponding to the pupil. Therefore, there is a high probability that the pixels included in the target pixel group 50 are pixels in the region where the pupil is visible.

[0076] In the human eye, the movement of the upper eyelid is greater than that of the lower eyelid. Therefore, even if part of the pupil is covered by the eyelid, the lower part of the pupil can usually be seen. With the above configuration, the first pixel 61 with the smallest Y-axis coordinate value, the second pixel 62 with the largest Y-axis coordinate value, one or more third pixels 63 located below the first pixel 61, and one or more fourth pixels 64 located below the second pixel 62 are extracted. Therefore, as shown in Figure 9, pixels that appear near the boundary between the pupil and the iris are more easily extracted as the first pixel 61, second pixel 62, third pixel 63, and fourth pixel 64. As a result, the center 71 of the approximate circle 70 of the first pixel 61, second pixel 62, third pixel 63, and fourth pixel 64 is determined as the center position of the pupil, thereby allowing the center position of the pupil to be accurately identified.

[0077] Thus, according to this embodiment, even if a portion of the pupil is covered by the eyelid, the central position of the pupil can be accurately determined.

[0078] Furthermore, the central position of the pupil in the target frame is determined using only that target frame. Therefore, the correction process using preceding and succeeding frames, which is performed in the image processing method related to the first reference embodiment, is unnecessary. As a result, the central position of the pupil can be determined in a short time.

[0079] Furthermore, according to the above configuration, since AI is not used as in the image processing method relating to the second reference embodiment, a low-cost, low-spec computer (e.g., a smartphone or tablet) can be used as the image processing device 1. Therefore, the image processing device 1 is easily applicable to home healthcare settings. For example, a healthcare professional can visit a patient's home with goggles 2 and a smartphone acting as the image processing device 1, and diagnose and differentiate dizziness disorders while checking the smartphone screen.

[0080] The detection unit 12 identifies one or more candidate pixel groups 40 from the frame, which are consecutive pixels having a brightness value that falls within a brightness range. The detection unit 12 then determines the candidate pixel group 40 from among the one or more candidate pixel groups 40 whose area falls within a predetermined area range as the target pixel group 50.

[0081] According to the above configuration, the area range that the pupil can capture is predetermined. For example, even if the brightness value of the area near the outer corner of the eye is included in the brightness range due to the shadow of the eyelid, this prevents that area from being determined as part of the target pixel group 50. This improves the accuracy of identifying the center position of the pupil.

[0082] The pupil position determination unit 14 calculates the center of the approximate circle using the least squares method. This allows the pupil's center position to be determined using a relatively simple calculation method.

[0083] The extraction unit 13 extracts a predetermined number of third pixels 63 by selecting pixels below the first pixel 61 among the multiple contour pixels 60 whose distance from either the first pixel 61 or the already extracted third pixel 63 is less than or equal to a predetermined threshold. Similarly, the extraction unit 13 extracts a predetermined number of fourth pixels 64 by selecting pixels below the second pixel 62 among the multiple contour pixels 60 whose distance from either the second pixel 62 or the already extracted fourth pixel 64 is less than or equal to a predetermined threshold.

[0084] The boundary between the pupil and the iris is circular or arc-shaped. Therefore, pixels that capture this boundary exist consecutively. On the other hand, depending on the frame 21, due to image distortion or other reasons, pixels that capture a part different from the boundary between the pupil and the iris (hereinafter referred to as "singular pixels") may be extracted as contour pixels 60. However, with the above configuration, pixels whose distance from either the first pixel 61 or the already extracted third pixel 63 is less than or equal to a threshold are extracted as the third pixel 63. Therefore, it is prevented that singular pixels that do not exist consecutively with the first pixel 61 or the already extracted third pixel 63 are extracted as the third pixel 63. Similarly, it is prevented that singular pixels that do not exist consecutively with the second pixel 62 or the already extracted fourth pixel 64 are extracted as the fourth pixel 64. As a result, the accuracy of identifying the center position of the pupil is improved.

[0085] As described above, the images used to determine the central position of the pupil are each frame 21 of the video 20 showing the pupil of the subject 3. The image processing device 1 further includes a providing unit 15 that provides information indicating the trajectory of the central position determined by the pupil position determination unit 14.

[0086] According to the above configuration, for example, a healthcare professional can diagnose and differentiate dizziness disorders by reviewing the information provided by the information provision unit 15.

[0087] (modified version) In the above description, the image processing device 1 is assumed to acquire video 20 from the goggles 2. However, the image processing device 1 may also be equipped with an infrared camera, and video 20 may be acquired from the infrared camera. In this case, the image processing device 1 is fixed in a position where it can image the eyes of the subject 3. For example, the image processing device 1 is attached to a goggle-shaped jig. With the subject 3 wearing the jig, imaging with the infrared camera should be started.

[0088] In the above description, the providing unit 15 displays information indicating the trajectory of the central position on the display 105. However, the providing unit 15 may also display information indicating the trajectory of the central position on an external display device. Alternatively, the providing unit 15 may provide information indicating the trajectory of the central position to an external computer (including a cloud server). Medical professionals can access the external computer to confirm the trajectory of the central position of the subject's 3 pupil and perform the diagnosis and differential diagnosis of dizziness disorders.

[0089] While embodiments of the present invention have been described, the embodiments disclosed herein should be considered in all respects to be illustrative and not restrictive. The scope of the present invention is defined by the claims, and all modifications within the meaning and scope equivalent to the claims are intended to be included. [Explanation of Symbols]

[0090] 1 Image processing device, 2 Goggles, 3 Subject, 10 Memory unit, 11 Acquisition unit, 12 Detection unit, 13 Extraction unit, 14 Pupil position determination unit, 15 Provision unit, 16 Trajectory data, 20 Video, 21 Frame, 40, 40a~40f Candidate pixel group, 42 Pixel, 44, 52 Binary image, 50 Target pixel group, 60 Contour pixel, 61 First pixel, 62 Second pixel, 63 Third pixel, 64 Fourth pixel, 70 Approximate circle, 71 Center, 72 Trajectory line, 73 Text, 80, 90 User interface screen, 81, 82a, 82b Input field, 83 Load button, 84 Area, 91 Graph, 101 Processor, 102 RAM, 103 Storage, 104 Communication interface, 105 Display, 106 Input device, 107 Bus, 110 image processing programs, 150 pixel groups, 171 center position, 180,182 parts, 201 communication devices.

Claims

1. An image processing device, An acquisition unit that acquires an image of the subject's eyes, A detection unit detects a group of consecutive pixels from the aforementioned image, in which pixels having a brightness value included in the brightness range corresponding to the pupil. An extraction unit extracts, from among a plurality of contour pixels that constitute the contour of the target pixel group, a first pixel whose coordinate value along the axis in the left-right direction of the subject is smallest, a second pixel whose coordinate value along the axis is largest, one or more third pixels located below the first pixel, and one or more fourth pixels located below the second pixel. An image processing apparatus comprising: a pupil position determination unit that determines an approximate circle for the first pixel, the second pixel, the one or more third pixels, and the one or more fourth pixels, and determines the center of the approximate circle as the center position of the pupil.

2. The detection unit is From the aforementioned image, one or more candidate pixel groups consisting of consecutive pixels having brightness values ​​included in the brightness range are identified. The image processing apparatus according to claim 1, wherein, among the one or more candidate pixel groups, a group of candidate pixels whose area falls within a predetermined area range is determined to be the target pixel group.

3. The image processing apparatus according to claim 1, wherein the pupil position determination unit calculates the center of the approximate circle using the least squares method.

4. The extraction unit is From among the plurality of contour pixels, pixels located below the first pixel whose distance from either the first pixel or the already extracted third pixel is less than or equal to a predetermined threshold are extracted as the third pixel, thereby extracting a predetermined number of the third pixels. The image processing apparatus according to claim 1, wherein a predetermined number of fourth pixels are extracted by selecting pixels from among the plurality of contour pixels that are lower than the second pixel, and whose distance from either the second pixel or the already extracted fourth pixel is less than or equal to the predetermined threshold, thereby extracting a predetermined number of fourth pixels.

5. The aforementioned images are each frame of the video showing the pupil of the subject, The aforementioned image processing device is The image processing apparatus according to any one of claims 1 to 4, further comprising a providing unit that provides information indicating the trajectory of the central position determined by the pupil position determination unit.

6. An image processing method, The steps include obtaining an image of the subject's eyes, From the aforementioned image, the step of detecting a group of target pixels in which pixels having a brightness value included in the brightness range corresponding to the pupil are consecutive, From among a plurality of contour pixels that constitute the contour of the target pixel group, the steps include: extracting a first pixel whose coordinate value along the axis in the left-right direction of the subject is smallest, a second pixel whose coordinate value along the axis is largeest, one or more third pixels located below the first pixel, and one or more fourth pixels located below the second pixel; An image processing method comprising the steps of calculating an approximate circle from the first pixel, the second pixel, the one or more third pixels, and the one or more fourth pixels, and determining the center of the approximate circle as the center position of the pupil.

7. A program that causes a computer to perform an image processing method, The aforementioned image processing method is The steps include obtaining an image of the subject's eyes, From the aforementioned image, the step of detecting a group of target pixels in which pixels having a brightness value included in the brightness range corresponding to the pupil are consecutive, From among a plurality of contour pixels that constitute the contour of the target pixel group, the steps include: extracting a first pixel whose coordinate value along the axis in the left-right direction of the subject is smallest, a second pixel whose coordinate value along the axis is largeest, one or more third pixels located below the first pixel, and one or more fourth pixels located below the second pixel; A program that includes the step of calculating an approximate circle from the first pixel, the second pixel, the one or more third pixels, and the one or more fourth pixels, and determining the center of the approximate circle as the center position of the pupil.

Images