Biological information acquisition device and biological information detection method

Abstract

A biological information acquisition device disclosed herein includes: an imaging unit having a pixel array unit in which pixels for receiving reflected light from an imaging target area including an eyeball area of a user wearing the device are two-dimensionally arrayed; an imaging optical system which is disposed between the imaging unit and the imaging target area and guides the reflected light to the pixel array unit; and a biological information detection unit for detecting a plurality of pieces of biological information about the user wearing the device, the pieces of biological information being detected using a captured image that can be obtained by being captured by the imaging unit. The imaging optical system is designed such that image distortion is greater in an outer edge part than a central part of the pixel array unit. The biological information detection unit executes a first detection process for obtaining a first piece of biological information among the plurality of pieces of biological information, and a second detection process for obtaining a second piece of biological information among the plurality of pieces of biological information.

Description

Biometric information acquisition device and biological information detection method

[0001] The present technology relates to a technical field of a biometric information acquisition device and a biometric information detection method that can acquire various types of information about a user wearing the device.

[0002] 2. Description of the Related Art In the case of information processing devices that are worn by a user, such as head-mounted displays or smart glasses, there are known techniques for obtaining various types of biometric information about the user wearing the devices (for example, see Patent Document 1 listed below).

[0003] Special Publication No. 2021-517689

[0004] Incidentally, when acquiring biometric information about the user, a light source and a camera may be used. However, since the light source and the camera can shorten the operating time of a battery-powered device, efficient operation is required.

[0005] The present technology has been made in consideration of such problems, and aims to efficiently detect multiple pieces of biological information about the wearing user.

[0006] The biometric information acquisition device according to the present technology includes an imaging unit having a pixel array unit in which pixels that receive reflected light from an imaging target area including the eyeball region of a user wearing the device are two-dimensionally arranged, an imaging optical system disposed between the imaging unit and the imaging target area and directing the reflected light to the pixel array unit, and a biometric information detection unit that detects multiple pieces of biometric information about the user wearing the device using an image captured by the imaging unit. The imaging optical system is designed to have greater image distortion at the outer edge than at the center of the pixel array unit, and the biometric information detection unit performs a first detection process to obtain first biometric information from the multiple pieces of biometric information and a second detection process to obtain second biometric information from the multiple pieces of biometric information. The imaging optical system is intentionally designed to have greater image distortion at the outer edge. Therefore, the captured image obtained using the imaging optical system has a high resolution in the center and a compressed image that includes many subjects captured at a wide angle in the outer edge other than the center.

[0007] FIG. 1 is a schematic diagram showing a state in which a user is wearing an HMD. FIG. 2 is a block diagram showing the functional configuration of an HMD. FIG. 3 is a diagram showing an example of the configuration of a camera device. FIG. 4 is a diagram showing the lens configuration of the imaging optical system of the camera device. FIG. 5 is a diagram showing longitudinal spherical aberration diagrams, astigmatism diagrams, and distortion diagrams in a numerical example in which specific numerical values ​​are applied. FIG. 6 is a diagram showing an example of an image without distortion. FIG. 7 is a diagram showing an example of a RAW image. FIG. 8 is a diagram showing an example of a captured image. FIG. 9 is a block diagram showing an example of the functional configuration of an arithmetic processing unit. FIG. 10 is a diagram showing an example of an image without distortion in a state in which only some of the light sources are turned on. FIG. 11 is a diagram showing an example of a RAW image in a state in which only some of the light sources are turned on. FIG. 12 is a diagram showing an example of the relationship between the processing timing of a detection process and captured frames. FIG. 13 is a diagram showing another example of the relationship between the processing timing of a detection process and captured frames. FIG. 14 is a flowchart showing an example of processing executed by the arithmetic processing unit.

[0008] Hereinafter, with reference to the accompanying drawings, embodiments of the present technology will be described in the following order: <1. Configuration of biometric information acquisition device> <2. Configuration example of camera device> <3. Functional configuration of arithmetic processing unit> <4. Processing timing> <5. Processing example> <6. Other> <7. Summary> <8. The present technology>

[0009] <1. Configuration of the Biometric Information Acquisition Device> A description will be given of a configuration example of the biometric information acquisition device 1. In the present embodiment, an HMD (Head Mount Display) 1A is given as an example of the biometric information acquisition device 1. However, the embodiment of the biometric information acquisition device 1 is not limited to this, and various other embodiments are possible, such as smart glasses and a headset.

[0010] An example of a state in which a user wears the HMD 1A is shown in Fig. 1. In the following description, a user wearing the HMD 1A will be referred to as a "wearing user U."

[0011] The HMD 1A is a device worn on the head of the wearing user U. The HMD 1A is a device that allows the wearing user U to visually recognize various display images and is capable of detecting various types of biometric information about the wearing user U.

[0012] The HMD 1A may be provided with an operator (not shown). For example, the user U can operate a menu or the like by operating an operator (not shown) provided on the HMD 1A.

[0013] The functional configuration of the HMD 1A is shown in Fig. 2. The HMD 1A includes a display unit 2, a light source 3, a camera device 4, and a processing unit 5.

[0014] The display unit 2 displays, as a display image, an image to be viewed by the wearing user U. Various configurations of the display unit 2 are conceivable, and for example, the display unit 2 may be a monitor device that makes the outside of the HMD 1A invisible, or a monitor device that makes the real world outside the HMD 1A visible and makes it possible to view a state in which a display image is superimposed on the real world.

[0015] The light source 3 irradiates IR (Infrared) light onto a predetermined range centered on the eyeball of the wearing user U, which is the area to be imaged by the camera device 4 (referred to as the "image-capturing area Aim" in the following explanation).

[0016] The imaging target area Aim in this embodiment is a predetermined area centered on the eyeball of the user U wearing the device.

[0017] The light source 3 includes not only a light emitter that emits IR light, but also a driver that drives the light emitter.

[0018] The camera device 4 generates a RAW image by receiving IR light emitted from the light source 3 and reflected from the imaging target area Aim, and performs predetermined signal processing to obtain a captured image Gr. The captured image Gr is supplied from the camera device 4 to the calculation processing unit 5.

[0019] The arithmetic processing unit 5 detects biometric information about the user U wearing the device by performing predetermined image processing on the captured image Gr supplied from the camera device 4.

[0020] The calculation processing unit 5 also controls each unit to detect biological information about the wearing user U. This will be described in detail later.

[0021] 3 shows an example of the configuration of the camera device 4. The camera device 4 includes an imaging optical system 6, an imaging unit 7, and a signal processing unit 8.

[0022] The imaging optical system 6 guides reflected light from an imaging target area Aim to a pixel array unit 9 included in the imaging unit 7. The imaging optical system 6 is configured with at least two lenses. A specific configuration example of the imaging optical system 6 is shown in FIG.

[0023] The imaging optical system 6 is configured by arranging, in order from the object side to the image side, a first lens unit L1, a second lens unit L2, and a third lens unit L3.

[0024] The first lens unit L1, the second lens unit L2, and the third lens unit L3 may each include a plurality of lenses, but in the description of Fig. 4, each lens unit will be replaced with a single equivalent lens.

[0025] The first lens unit L1 is a negative meniscus lens with a convex surface facing the object side, the second lens unit L2 is a positive lens with a convex surface facing the image side, and the third lens unit L3 is a flat lens with no lens power.

[0026] In addition, when the third lens unit L3 is configured to have a plurality of lenses, for example, the third lens unit L3 may have a lens that is a relatively thin flat plate and a lens that is a relatively thick flat plate.

[0027] The relatively thin, flat lenses in the third lens unit L3 are formed of, for example, a glass material to facilitate the addition of a multilayer coating to improve transmittance. Furthermore, the glass lenses are formed of a thin, flat plate, ensuring ease of dicing (cutting). On the other hand, the relatively thick, flat lenses in the third lens unit L3 are formed of a resin material or the like to ensure ease of processing during dicing.

[0028] Numerical examples in which specific numerical values ​​are applied to the imaging optical system 6 will be described below with reference to the drawings and tables.

[0029] The meanings of the symbols used in the following tables and explanations are as follows:

[0030] "r" is the paraxial radius of curvature of the i-th surface, "d" is the on-axis surface spacing between the i-th surface and the (i+1)-th surface (the thickness at the center of the lens or the air spacing), "nd" is the refractive index at the d-line (λ=587.6 nm) of the lens etc. starting from the i-th surface, and "νd" is the Abbe number at the d-line of the lens etc. starting from the i-th surface.

[0031] Aspherical surfaces are marked with an asterisk (*) to the right of the surface number, and aperture stops are marked with the word "stop" to the right of the surface number.

[0032] "κ" is the conic constant, and "A4", "A6", "A8", "A10", "A12", "A14", and "A16" represent the 4th-order, 6th-order, 8th-order, 10th-order, 12th-order, 14th-order, and 16th-order aspheric coefficients, respectively.

[0033] In the table below showing aspherical coefficients, etc., "E-n" is an exponential notation with base 10, that is, "10 to the minus n-th power." For example, "0.12345E-05" represents "0.12345 × (10 to the minus fifth power)."

[0034] Some lenses used in the imaging optical system 6 have aspherical lens surfaces. The aspherical shape is defined by the following formula 1, where "x" is the distance from the vertex of the lens surface in the optical axis direction (sag amount), "y" is the height in the direction perpendicular to the optical axis direction (image height), "c" is the paraxial curvature at the vertex of the lens (the reciprocal of the radius of curvature), "κ" is the conic constant, and "A4", "A6", ... are fourth-order, sixth-order, ... aspherical coefficients, respectively.

[0035]

[0036] Table 1 shows lens data according to a numerical example in which specific values ​​are applied to the imaging optical system 6.

[0037]

[0038] Table 2 shows the focal length f, F-number, full angle of view 2ω, image height Y, total lens length L, and object distance in the numerical example.

[0039]

[0040] The values ​​of Dp / Da in the numerical examples are shown in Table 3. Note that "Dp" is the thickness of the third lens unit L3 in the optical axis direction, and "Da" is the total lens length L, as shown in FIG.

[0041]

[0042] Table 4 shows the fourth-, sixth-, eighth-, tenth-, twelfth-, fourteenth-, and sixteenth-order aspherical coefficients A4, A6, A8, A10, A12, A14, and A16 of the aspherical surfaces in the numerical examples, together with the conic constant κ.

[0043]

[0044] FIG. 5 shows longitudinal spherical aberration, astigmatism, and distortion of the imaging optical system 6. In FIG.

[0045] In the astigmatism diagram in FIG. 5, the solid line indicates the value on the sagittal image plane, and the broken line indicates the value on the meridional image plane.

[0046] The wavelength of the light in FIG. 5 is 856 nm.

[0047] In the imaging optical system 6 to which the above-described numerical example is applied, the following formula 2 is satisfied.

[0048]

[0049] "H10i" indicates the ideal image height at 100% of the image height, and "H10a" indicates the actual image height at 100% of the image height.

[0050] It is more preferable that the following formula 3 be satisfied.

[0051]

[0052] For example, in this embodiment, the left sides of Equations 2 and 3 are set to −39.7, thereby obtaining a wide-angle image of the subject as a captured image.

[0053] Moreover, in the imaging optical system 6 to which the numerical example is applied, the following formula 4 is satisfied.

[0054]

[0055] It is more preferable that the following formula 5 be satisfied.

[0056]

[0057] For example, in this embodiment, the left sides of Equation 4 and Equation 5 are set to −8.05, so that a wide-angle image is included to some extent not only in the corners but also above and below the center of the captured image.

[0058] Furthermore, in the imaging optical system 6 to which the numerical example is applied, the following formula 6 is satisfied.

[0059]

[0060] It is more preferable that the following formula 7 be satisfied.

[0061]

[0062] For example, the value of Dp / Da is set to 0.56 in this embodiment.

[0063] Returning to the explanation of Fig. 3, the imaging unit 7 is configured to include a pixel array unit 9 in which pixels G are two-dimensionally arranged and receive light reflected from the imaging target area Aim and incident via the imaging optical system 6. In addition to the pixel array unit 9, the imaging unit 7 also includes a readout circuit and the like (not shown).

[0064] In the readout circuit of the imaging unit 7, for example, correlated double sampling (CDS) processing, automatic gain control (AGC) processing, etc. are performed on the electrical signal obtained by photoelectric conversion, and further, analog to digital (A / D) conversion processing is performed, whereby an image (RAW image) is output as digital data from the imaging unit 7.

[0065] The signal processing unit 8 includes a lens distortion correction unit F81 that performs lens distortion correction on the RAW image output from the pixel array unit 9.

[0066] The lens distortion correction unit F81 is capable of correcting distortion not only for the entire RAW image, but also for part of the RAW image.

[0067] The distortion correction for a portion of the RAW image by the lens distortion correction unit F81 may be achieved by limiting the RAW image output from the pixel array unit 9 of the imaging unit 7 to only the central portion of the effective pixel area of ​​the pixel array unit 9.

[0068] Alternatively, the distortion correction for a part of the RAW image by the lens distortion correction unit F81 may be achieved by selectively correcting a partial area of ​​the RAW image output from the pixel array unit 9 of the imaging unit 7.

[0069] The image in which the lens distortion has been corrected by the lens distortion correction unit F81 is output as a captured image Gr to the calculation processing unit 5 at the subsequent stage.

[0070] 3. Functional Configuration of the Arithmetic Processing Unit A description will be given of an example of a captured image Gr obtained by the above-described configuration of the imaging optical system 6. First, an image without lens distortion is shown as a distortion-free image G1 in FIG.

[0071] The area indicated by the dashed line on the distortion-free image G1 is the imaging target area Aim that is the target of imaging by the pixel array unit 9. That is, the RAW image output from the pixel array unit 9 is a distorted image G2 that includes lens distortion, as shown in FIG.

[0072] 7, in the RAW image obtained by employing the imaging optical system 6 of this embodiment, the eyes of the user U wearing the lens are captured in a large image in the center, and the areas around the forehead and ears that are far from the eyes are also captured in the outer edge of the RAW image. That is, in the distorted image G2, the eyes are captured in high resolution in the center, and areas at a distance from the eyes are captured in low resolution in the outer edge.

[0073] The captured image Gr obtained after the lens distortion correction unit F81 processes the distorted image G2 is an image obtained by cutting out the imaging target area Aim from the distortion-free image G1, as shown in FIG.

[0074] The calculation processing unit 5 of the HMD 1A executes a program to realize a function of obtaining a desired distortion-free image G1 and a function of detecting biometric information from the distortion-free image G1.

[0075] An example of the functional configuration of the arithmetic processing unit 5 is shown in FIG.

[0076] The calculation processing unit 5 functions as a display control unit F51, a region of interest setting unit F52, a light emission control unit F53, a camera control unit F54, and a biological information detection unit F55.

[0077] The display control unit F51 performs processing to display a display image on the display unit 2 included in the HMD 1A.

[0078] The region of interest setting unit F52 performs processing to obtain a desired captured image Gr. Specifically, F52 causes the light emission control unit F53 to perform processing to emit light from all or some of the multiple light sources 3 included in the HMD 1A in order to obtain the desired captured image Gr.

[0079] In addition, the region of interest setting unit F52 causes the camera control unit F54 to perform a process of specifying a region to be read out from the pixel array unit 9 in order to obtain a desired captured image Gr in which only pixel data of the ROI (Region Of Interest) is valid data for the camera device 4, and a process of specifying a target region for distortion correction processing by the lens distortion correction unit F81.

[0080] For example, the RAW image shown in FIG. 7 is an image in which the entire effective pixel area of ​​the pixel array unit 9 is set as the ROI.

[0081] On the other hand, when the biological information to be acquired is iris information or gaze information, it is sufficient to obtain an image of the eye area of ​​the wearing user U. Therefore, the region of interest setting unit F52 sets an ROI in the center of an elliptical region in the captured image Gr.

[0082] The region of interest setting unit F52 sets the readout circuit via the camera control unit F54 to read out only the pixels G corresponding to the ROI. The region of interest setting unit F52 then sets the signal processing unit 8 via the camera control unit F54 to perform distortion correction on only the data output from the central portion of the pixel array unit 9.

[0083] 10 shows the range of the wearer U illuminated by the light source 3 under the control of the light emission control unit F53 when the ROI is set in the center of the captured image Gr by the region of interest setting unit F52. Also, FIG. 11 shows the RAW image read out under the control of the camera control unit F54 when the ROI is set in the center of the captured image Gr by the region of interest setting unit F52.

[0084] Note that Fig. 11 is a RAW image, similar to Fig. 7, and is designated as distorted image G2. In Fig. 11, the image size is shown to be the same as Fig. 7 for comparison, but the actual RAW image shown in Fig. 11 may be missing pixel data in the blackened areas other than the center. In other words, the RAW image shown in Fig. 11 may be an image with a smaller image size than the RAW image shown in Fig. 7, since it is composed of areas other than the blackened areas.

[0085] In the following description, when only the central portion of the effective pixel region is set as an ROI as in Figure 11, the ROI will be referred to as a "first region of interest Ar1." Also, when the entire effective pixel region is set as an ROI as in Figure 7, the ROI will be referred to as a "second region of interest Ar2." Note that when the entire effective pixel region is to be processed, setting an ROI is not essential. That is, the ROI may be set only for the first region of interest Ar1.

[0086] The biometric information detection unit F55 detects multiple types of biometric information about the wearing user U based on the captured image Gr supplied from the camera device 4.

[0087] Specifically, when the biometric information detection unit F55 receives the captured image Gr in which the first region of interest Ar1 is set as the ROI, the biometric information detection unit F55 detects, for example, gaze information and iris information as biometric information related to the eyes of the wearing user U. This detection process is referred to as a “first detection process.”

[0088] In the first detection process, the biometric information detection unit F55 performs a detection process on the first region of interest Ar1 in which the ROI is set. The biometric information of the wearing user U detected in the first detection process is referred to as “first biometric information.”

[0089] That is, in the first detection process, first biological information is detected by performing a process on the first region of interest Ar1.

[0090] The first region of interest Ar1 is a region in which a high-resolution image is captured, and is suitable for detecting iris information or the like as first biometric information.

[0091] Furthermore, when the biometric information detection unit F55 receives the captured image Gr in which the second region of interest Ar2 is set as the ROI, it detects, for example, facial expression information as biometric information about the eyes and surrounding areas of the wearing user U. This detection process is referred to as a “second detection process.”

[0092] In the second detection process, the biometric information detection unit F55 performs a detection process on the second region of interest Ar2 in which the ROI is set. The biometric information of the wearing user U detected in the second detection process is referred to as “second biometric information.”

[0093] That is, in the second detection process, the second region of interest Ar2 is subjected to processing to detect second biological information.

[0094] The second region of interest Ar2 is a region where a wide range image is captured at low resolution, and is suitable for detecting facial expression information as second biometric information, which does not require high resolution.

[0095] Note that these pieces of biological information are merely examples, and vital sensing information, such as heart rate information, using subcutaneous reflection of the wearing user U based on the first region of interest Ar1 and the second region of interest Ar2 may also be detected as biological information. In this case, it is preferable to irradiate from the light source 3 with a wavelength of approximately 700 nm to 900 nm, which is a wavelength at which hemoglobin circulating in the blood of the wearing user U has a high light absorption rate. This can increase the fluctuation of reflected light due to pulsation.

[0096] Note that existing configurations can be used to detect various types of biometric information, such as gaze information, iris information, and emotional information, from the captured image Gr. For example, the biometric information detection unit F55 can detect desired biometric information by procedural image processing or image processing using AI.

[0097] 4. Processing Timing Biometric information needs to be acquired at appropriate intervals depending on the type. Furthermore, considering the processing load on each part of the HMD 1A, the intervals cannot be shortened indiscriminately.

[0098] Here, the optimization of the timing for acquiring various types of biological information will be described.

[0099] For example, it is not necessary to detect second biometric information such as facial expression information for each imaging frame of the imaging unit 7. On the other hand, it is desirable to obtain first biometric information such as gaze information as frequently as possible.

[0100] An example of the relationship between the processing timing of each detection process and the captured frames is shown in Fig. 12. The captured frame numbers in the figure are assigned in ascending order, starting with "1" for a frame captured at a specific timing. The frame rate for capturing images is 120 fps (frames per second).

[0101] In addition, the ROI in the figure is shown as "narrow" when the ROI is set in the first region of interest Ar1, and as "wide" when the ROI is set in the second region of interest Ar2. That is, the ROI in the figure indicates the size of the ROI.

[0102] Furthermore, in the drawing, the first detection process and the second detection process are executed by adding a check mark to the imaging frames in which they are executed. That is, the first detection process is executed at 120 fps, and the second detection process is executed at 30 fps.

[0103] If the load of the second detection process is large, the first detection process and the second detection process may be executed in the manner shown in Fig. 13. That is, the second detection process is executed at 30 fps, and the first detection process is executed only for imaging frames in which the second detection process is not executed.

[0104] This makes it possible to complete the processing within the captured frame even if an algorithm with a large processing load is executed in the second detection processing.

[0105] 5. Processing Example An example of processing executed by the arithmetic processing unit 5 is shown in FIG.

[0106] In step S101, the calculation processing unit 5 determines whether or not the frame in which the image is captured is the frame in which the second detection process is to be executed.

[0107] For example, in the case of the image frames with frame numbers "1," "5," and "9" in FIG. 12, the result in step S101 is "Yes," while the result in other image frames such as frame numbers "2," "3," and "4" is "No."

[0108] If it is determined that the captured frame is not one for which the second detection process is to be executed (step S101: No determination), the calculation processing unit 5 executes the processes of steps S102, S103 and S104, and then proceeds to step S108.

[0109] Specifically, the arithmetic processing unit 5 sets an ROI in the first region of interest Ar1 in step S102. In step S103, the arithmetic processing unit 5 selects the light source 3 to emit light so that the illumination area of ​​the light source 3 corresponds to the region on the subject corresponding to the first region of interest Ar1. In step S104, the arithmetic processing unit 5 performs imaging settings to read out the region corresponding to the first region of interest Ar1, and instructs the signal processing unit 8 to perform lens distortion correction on the pixel data output from the first region of interest Ar1.

[0110] On the other hand, if it is determined that the captured frame is one for which the second detection process is to be executed (step S101: Yes determination), the calculation processing unit 5 executes the processes of steps S105, S106 and S107, and proceeds to step S108.

[0111] Specifically, the arithmetic processing unit 5 sets an ROI in the second region of interest Ar2 in step S105. In step S106, the arithmetic processing unit 5 selects all light sources 3 so that the illumination area of ​​the light sources 3 corresponds to the region on the subject corresponding to the second region of interest Ar2. In step S107, the arithmetic processing unit 5 performs imaging settings so as to read out the region corresponding to the second region of interest Ar2, specifically so as to read out the entire effective pixel area, and instructs the signal processing unit 8 to perform lens distortion correction on the pixel data output from the second region of interest Ar2.

[0112] In step S108, the processing unit 5 issues a lighting instruction to the light source 3 selected in step S103 or step S106, so that only the specific light source 3 emits light.

[0113] In step S109, the calculation processing unit 5 issues an imaging instruction to the imaging unit 7. At this time, the imaging unit 7 reads out pixel data output from pixels G in the range specified in step S104 or step S107. In addition, the lens distortion correction unit F81 in the signal processing unit 8 performs distortion correction processing on the pixel data output from pixels G in the specified range.

[0114] In step S110, the processing unit 5 issues an instruction to turn off the light source 3 that was turned on in step S108.

[0115] In step S111, the processing unit 5 receives the captured image Gr.

[0116] In step S112, the processing unit 5 first executes a first detection process, thereby detecting iris information, line-of-sight information, and the like for the captured image frame.

[0117] Next, in step S113, the calculation processing unit 5 determines whether or not the captured frame is a frame for executing the second detection process. This determination process is the same as the process in step S101.

[0118] If it is determined that the captured image frame is one for which the second detection process should be executed (step S113: Yes determination), the calculation processing unit 5 executes the second detection process in step S114, thereby detecting facial expression information and the like for the captured image frame.

[0119] On the other hand, if it is determined that the captured frame is not one for which the second detection process is to be executed (step S113: N determination), the calculation processing unit 5 avoids the process of step S114.

[0120] As described above, the processing unit 5 executes the series of processes shown in FIG. 14 for each captured frame, so that the first detection process is executed at 120 fps and the second detection process is executed at 30 fps.

[0121] The instruction to turn off the light source 3 in step S110 is not essential. For example, the processing unit 5 can turn off the light source 3 without issuing an instruction by instructing the timing to start and end light emission in step S108.

[0122] <6. Others> When individual differences in the imaging optical systems 6 are taken into consideration, the distortion of the imaging optical systems 6 may differ for each imaging optical system 6 produced.

[0123] Therefore, it is desirable that the correction formula used by the lens distortion correction unit F81 to correct distortion in the imaging optical system 6 is calculated by actually capturing an image of a chart using the imaging optical system 6 and the imaging unit 7, determining the intersection points of the chart in the obtained RAW image, and calculating a distortion approximation formula.

[0124] 7. Summary The biometric information acquisition device 1 of the present technology, including the HMD 1A described above, includes an imaging unit 7 having a pixel array unit 9 in which pixels G are two-dimensionally arranged to receive reflected light from an imaging target area Aim including the eyeball region of the wearing user U; an imaging optical system 6 disposed between the imaging unit 7 and the imaging target area Aim and configured to guide the reflected light to the pixel array unit 9; and a biometric information detection unit F55 that detects multiple pieces of biometric information about the wearing user U using a captured image Gr obtained by imaging by the imaging unit 7. The imaging optical system 6 is designed to have greater image distortion at the outer edge than at the center of the pixel array unit 9, and the biometric information detection unit F55 performs a first detection process to obtain first biometric information from the multiple pieces of biometric information and a second detection process to obtain second biometric information from the multiple pieces of biometric information. The imaging optical system 6 is intentionally designed to have greater image distortion at the outer edge. Therefore, the captured image Gr obtained using the imaging optical system 6 has a high resolution in the center and a compressed image including many subjects captured at a wide angle in the outer edge other than the center. Therefore, for example, when the pupil of the user U wearing the device is captured in the center of the captured image Gr, the pupil is captured in high resolution, making it possible to detect biometric information related to the pupil with high accuracy. Furthermore, the outer edge of the captured image Gr captures a wide area of ​​subjects around the eyes, such as the eyebrows and cheeks, making it possible to detect biometric information such as facial expressions. In particular, the wide area around the eyes is captured in the outer edge of the captured image Gr, making it possible to estimate facial expressions with high accuracy. That is, with this configuration, biometric information that can be estimated with high accuracy using high-resolution images and biometric information that can be estimated with high accuracy using wide-angle images can be obtained from a single captured image Gr. This reduces the number of captures required to obtain multiple pieces of biometric information, and eliminates the need for separate image capture units 7 and imaging optical systems 6 for each type of biometric information. This reduces the number of components, simplifies the device, reduces costs, and reduces processing load. Furthermore, by using a single image capture unit 7, the biometric information acquisition device 1 can be operated for a long time when powered by a battery.

[0125] As described with reference to Fig. 5 etc., in the biometric information acquisition device 1 including the HMD 1A, it is desirable that, for 100% of the image height of the imaging optical system 6, when the ideal image height is H10i and the actual image height is H10a, the following conditional expression (1) is satisfied: (1) (H10a-H10i) / H10i*100<-20 This makes it possible to include many subjects captured at a wide angle in the outer edge of the captured image Gr.

[0126] 5 and other figures, in the biometric information acquisition device 1 including the HMD 1A, it is desirable to set the right side of the above conditional expression (1) to "-30" rather than "-20." This allows the captured image Gr to include more subjects captured at a wider angle in areas other than the central portion.

[0127] As described with reference to FIG. 5 and other figures, in the biometric information acquisition device 1 including the HMD 1A, it is desirable that for 50% of the image height of the imaging optical system 6, when the ideal image height is H5i and the actual image height is H5a, the following conditional expression (2) be satisfied: (2) (H5a-H5i) / H5i*100<-5 This causes a certain degree of image distortion at the boundary between the center and outer edge of the captured image Gr. Therefore, when an image is captured so that the pupils of the user U wearing the image are at the center of the captured image Gr, the area around the eyebrows of the user U wearing the image is also captured at a relatively wide angle, making it possible to estimate facial expressions and emotions with high accuracy.

[0128] As described with reference to FIG. 4 and other figures, the imaging optical system 6 in the biometric information acquisition device 1 including the HMD 1A includes a first lens unit L1 and a second lens unit L2 arranged in this order from the object side to the image side. The first lens unit L1 may be a lens unit with a convex surface facing the object side and having negative refractive power, and the second lens unit L2 may be a lens unit with a convex surface facing the image side and having positive refractive power. Specifically, a negative meniscus lens and a convex lens are arranged in this order from the object side to the image side. Using two lenses can reduce the number of parts, cost, weight, and thickness. When the biometric information acquisition device 1 is an HMD 1A or is mounted on an HMD 1A, reducing the weight of the imaging optical system 6 can reduce the burden on the user U wearing it. Furthermore, reducing the thickness of the imaging optical system 6 increases the flexibility in the placement location of the imaging optical system 6, improving design ease.

[0129] As described with reference to Figure 4 etc., the imaging optical system 6 in the biometric information acquisition device 1 including the HMD 1A has a third lens unit L3 that is arranged closest to the image side and has no lens power, and it is desirable that the third lens unit L3 satisfy the following conditional expression (3) where Dp is the thickness in the optical axis direction and Da is the overall thickness in the optical axis direction of the imaging optical system 6: (3) 0.2 < Dp / Da < 0.7 The third lens unit L3 is, for example, flat glass. "Having no lens power" refers to the case where design errors are not taken into consideration, and means that the lens power as a design value is "0" which neither converges nor diffuses.

[0130] As described with reference to FIGS. 9 and 14 , the biometric information detection unit F55 in the biometric information acquisition device 1 including the HMD 1A may perform a first detection process targeting a first region (first region of interest Ar1) on the pixel array unit 9, and a second detection process targeting a second region (second region of interest Ar2, or effective pixel region) on the pixel array unit 9 that is wider than the first region. For example, the first region may be a central region of the pixel array unit 9. The second region may be the entire effective pixel region including the central and outer edges of the pixel array unit 9. By changing the detection target region depending on the content of the detection process, the biometric information detection unit F55 can achieve efficient detection with reduced computational complexity depending on the situation. Note that the second region may be a region other than the central region of the effective pixel region of the pixel array unit 9.

[0131] As described with reference to FIG. 3 and other figures, the biometric information acquisition device 1 including the HMD 1A may be provided with a lens distortion correction unit F81 that corrects image distortion caused by the imaging optical system 6. The outer edge is considered to be an area where image distortion is large. By correcting distortion in this area in particular, it becomes possible to use existing algorithms, etc., as detection processing for images without image distortion. In other words, it is possible to avoid the need to create a new algorithm for detecting biometric information from images with large image distortion, thereby reducing development man-hours and costs.

[0132] As described with reference to FIG. 3 and other figures, the lens distortion correction unit F81 in the biometric information acquisition device 1 including the HMD 1A may perform distortion correction only on a region corresponding to a first region (first region of interest Ar1) when detecting first biometric information. Detection of the first biometric information is achieved, for example, by image processing of a region corresponding to the first region, which is the center of the pixel array unit 9, i.e., the center of the captured image Gr. By performing distortion correction only on the first region when detecting the first biometric information, the amount of calculation required for correction can be reduced, thereby reducing the processing load on the biometric information acquisition device 1. Furthermore, the reduced amount of calculation also makes it possible to support a high frame rate.

[0133] 11 and the like, the biometric information acquisition device 1 including the HMD 1A includes a region-of-interest setting unit F52 that sets a first region as a first region of interest Ar1 in detecting first biometric information, and the biometric information detection unit F55 may perform processing that targets only the first region of interest Ar1 in the first detection process. By limiting the region targeted for the first detection process to the central part of the captured image Gr, the amount of calculation required for the first detection process can be reduced, and the processing load on the biometric information acquisition device 1 can be reduced.

[0134] 11 and the like, the region of interest setting unit F52 in the biometric information acquisition device 1 including the HMD 1A may set the imaging unit 7 to output only pixel data related to the first region of interest Ar1 when detecting the first biometric information. By outputting a RAW image from the imaging unit 7 containing only data related to the first region of interest Ar1, the amount of data in the RAW image can be reduced, and the processing load related to sending and receiving the RAW image and the amount of calculation required for distortion correction performed on the RAW image can be reduced.

[0135] 12 and 13 , the biometric information detection unit F55 in the biometric information acquisition device 1 including the HMD 1A may detect the first biometric information more frequently than the second biometric information. For example, the detection frequency of the second biometric information detected from the entire captured image Gr is reduced. That is, the frequency of the second detection process, which has a relatively large processing target area and a heavy processing load, is reduced. Therefore, even when the second detection process is performed for each predetermined captured frame, it is possible to keep up with improvements in the capture frame rate.

[0136] 12 and the like, in the biometric information acquisition device 1 including the HMD 1A, the first biometric information may be detected for each imaging frame of the imaging unit 7, and the second biometric information may be detected for each of multiple imaging frames. That is, by executing the first detection process, which has a relatively low processing load, for each imaging frame, the first biometric information can be detected frequently. For example, if the first biometric information is gaze information that can change relatively frequently, detecting the gaze information frequently makes it less likely to miss subtle changes in the subject of interest of the wearing user U, and allows subsequent processing to realize an appropriate response according to the state of the wearing user U.

[0137] 9 and 10 , the biometric information acquisition device 1 including the HMD 1A may include a light source 3 that irradiates light onto an imaging target area Aim, and a light emission control unit F53 that controls the light emission of the light source 3. By providing the light source 3 that illuminates the imaging target subject, a suitable captured image Gr can be obtained even in a dark situation where no light is shining on the subject, and appropriate detection processing can be realized.

[0138] As described with reference to FIGS. 9 and 10 , in the biometric information acquisition device 1 including the HMD 1A, the first biometric information is information obtained by a detection process targeting a first region on the pixel array unit 9, and the second biometric information is information obtained by a detection process targeting a second region on the pixel array unit 9 that is larger than the first region. A plurality of light sources 3 are provided, and the light-emission control unit F53 may cause one of the light sources 3 to emit light onto a region on the subject corresponding to the first region during detection of the first biometric information, and another of the light sources 3 to emit light onto a region on the subject corresponding to the second region during detection of the second biometric information. This allows control to cause only some of the light sources 3 to emit light during the first detection process. This reduces power consumption related to the light emission of the light sources 3. In particular, when the biometric information acquisition device 1 is battery-powered, the operating time of the biometric information acquisition device 1 can be extended.

[0139] 2 and the like, in the biometric information acquisition device 1 including the HMD 1A, the light source 3 may be IR light. This prevents the wearing user U from feeling dazzled by the light emitted from the light source 3. Therefore, it is possible to detect biometric information about the wearing user U without affecting the wearing user U due to the light emitted from the light source 3.

[0140] 2 and 9 , the biometric information acquisition device 1 including the HMD 1A may include a display unit 2 that displays a display image to be viewed by the wearing user U, and a display control unit F51 that controls the display of the display image on the display unit 2. For example, the wearing user U views the display image displayed on the display unit 2. Then, it is possible to perform processing such as detecting the influence that the wearing user U has received from viewing the display image through a detection process and changing the display image according to the result. That is, it is possible to display an appropriate image according to the personality of the wearing user U, such as displaying a display image that is more frightening to the wearing user U, or changing the display image to a milder image so as not to cause further fear.

[0141] 1 and the like, the biometric information acquisition device 1 may be configured as a head-mounted display (HMD 1A). That is, by providing the HMD 1A with the above-described configuration, it becomes possible to obtain various functions and effects in the HMD 1A as well.

[0142] As described with reference to FIG. 9 and the like, in the biometric information acquisition device 1 including the HMD 1A, the first biometric information may be iris information or gaze information, and the second biometric information may be facial expression information. The first biometric information is information detected by performing image processing on a narrower area than the second biometric information. Iris information and gaze information are information that can be detected using only an area in which the eyes of the wearing user U are captured, and are suitable as the first biometric information. Meanwhile, the second biometric information is information detected by performing image processing on a wider area than the first biometric information. Facial expression information is information that can be appropriately detected by performing image processing on an area in which not only the eyes of the wearing user U but also the eyebrows, cheeks, and space between the eyebrows are captured, and is suitable as the second biometric information.

[0143] The biometric information detection method of the present technology is a process of detecting multiple pieces of biometric information about the wearing user U using an imaged image Gr obtained by imaging with an imaging unit 7 having a pixel array unit 9 in which pixels G that receive reflected light from an imaging target area Aim including the eyeball area of ​​the wearing user U are two-dimensionally arranged, and a computer device executes a first detection process to obtain first biometric information from the multiple pieces of biometric information and a second detection process to obtain second biometric information from the multiple pieces of biometric information, and the imaging optical system 6 that is positioned between the imaging unit 7 and the imaging target area Aim and guides the reflected light to the pixel array unit 9 is designed so that image distortion is greater at the outer edge than at the center of the pixel array unit 9.

[0144] The program in the present technology is a program that causes an arithmetic processing unit included in the biometric information acquisition device 1 as the HMD 1A to execute the various processes described above and illustrated in Fig. 14 etc. Such a program enables the biometric information acquisition device 1 as the HMD 1A to realize the various processes for obtaining the above-described information about the wearing user U.

[0145] These programs can be pre-recorded on a hard disk drive (HDD) or a ROM in a microcomputer having a CPU, which serves as a built-in recording medium in a computer or other device. Alternatively, the programs can be temporarily or permanently stored (recorded) on removable recording media such as a flexible disk, a CD-ROM (Compact Disk Read Only Memory), a Magneto Optical (MO) disk, a Digital Versatile Disc (DVD), a Blu-ray Disc (registered trademark), a magnetic disk, a semiconductor memory, or a memory card. Such removable recording media can be provided as so-called packaged software. Furthermore, such programs can be installed on a personal computer or the like from a removable recording medium, or can be downloaded from a download site via a network such as a local area network (LAN) or the Internet.

[0146] The effects described in this specification are merely examples and are not limiting, and other effects may also be present.

[0147] Furthermore, the above-described examples may be combined in any manner, and even when various combinations are used, the above-described various operational effects can be obtained.

[0148] <8. Present Technology> The present technology may also have the following configuration: <1> A biometric information acquisition device comprising: an imaging unit having a pixel array unit in which pixels that receive reflected light from an imaging target area including an eyeball area of ​​a user wearing the biometric information acquisition device are two-dimensionally arranged; an imaging optical system that is arranged between the imaging unit and the imaging target area and guides the reflected light to the pixel array unit; and a biometric information detection unit that detects multiple pieces of biometric information about the user wearing the biometric information acquisition device using an image acquired by imaging by the imaging unit, wherein the imaging optical system is designed to have greater image distortion at an outer edge of the pixel array unit than at a center, and the biometric information detection unit performs a first detection process to obtain first biometric information from the multiple pieces of biometric information, and a second detection process to obtain second biometric information from the multiple pieces of biometric information. <2> The biometric information acquisition device according to <1> above, wherein, for 100% of an image height for the imaging optical system, when an ideal image height is H10i and an actual image height is H10a, the following conditional expression (1) is satisfied: (1) (H10a-H10i) / H10i*100<-20 <3> The biometric information acquisition device according to any of <2> to <3> above, wherein for 50% of image heights in the imaging optical system, when an ideal image height is H5i and an actual image height is H5a, the following conditional expression (2) is satisfied: (2) (H5a-H5i) / H5i*100<-5 <4> The biometric information acquisition device according to any of <2> to <3> above, wherein the imaging optical system has a first lens unit and a second lens unit arranged in this order from the object side to the image side, wherein the first lens unit is a lens unit having a convex surface facing the object side and having negative refractive power, and the second lens unit is a lens unit having a convex surface facing the image side and having positive refractive power. <5> The biometric information acquisition device according to <4>, wherein the imaging optical system includes a third lens unit that is disposed closest to the image side and has no lens power, and the third lens unit satisfies the following conditional expression (3) when a thickness of the third lens unit in the optical axis direction is Dp and a thickness of the entire imaging optical system in the optical axis direction is Da:(3) 0.2<Dp / Da<0.7 <6> The biometric information acquisition device according to any one of <1> to <5>, wherein the biometric information detection unit performs processing on a first region on the pixel array unit in the first detection processing, and performs processing on a second region on the pixel array unit that is wider than the first region in the second detection processing. <7> The biometric information acquisition device according to <6>, comprising a lens distortion correction unit that corrects distortion of an image caused by the imaging optical system. <8> The biometric information acquisition device according to <7>, wherein the lens distortion correction unit performs distortion correction only on a region corresponding to the first region in detecting the first biometric information. <9> The biometric information acquisition device according to any one of <6> to <8>, comprising a region-of-interest setting unit that sets the first region as a first region of interest in detecting the first biometric information, and the biometric information detection unit performs processing on only the first region of interest in the first detection processing. <10> The biometric information acquisition device according to <9> above, wherein the region of interest setting unit sets the imaging unit to output only pixel data relating to the first region of interest when detecting the first biometric information. <11> The biometric information acquisition device according to any of <6> to <10> above, wherein the biometric information detection unit detects the first biometric information more frequently than the second biometric information. <12> The biometric information acquisition device according to <11> above, wherein the first biometric information is detected for each imaging frame of the imaging unit, and the second biometric information is detected for each of a plurality of the imaging frames. <13> The biometric information acquisition device according to any of <1> to <12> above, comprising: a light source that irradiates light onto the imaging target region; and an emission control unit that controls emission of light from the light source.<14> The biometric information acquisition device according to <13>, wherein the first biometric information is information obtained by a detection process targeting a first region on the pixel array unit, and the second biometric information is information obtained by a detection process targeting a second region on the pixel array unit that is larger than the first region, and wherein a plurality of the light sources are provided, and the light emission control unit, in detecting the first biometric information, causes one of the plurality of light sources to emit light onto a region on the subject corresponding to the first region, and in detecting the second biometric information, causes one of the plurality of light sources to emit light onto a region on the subject corresponding to the second region. <15> The biometric information acquisition device according to any of <13> to <14>, wherein the light source emits IR light. <16> The biometric information acquisition device according to any of <1> to <15>, comprising: a display unit on which a display image to be visually recognized by the wearing user is displayed; and a display control unit that controls display of the display image on the display unit. <17> The biometric information acquisition device according to <16>, wherein the biometric information acquisition device is a head-mounted display. <18> The biometric information acquisition device according to any one of <1> to <17>, wherein the first biometric information is iris information or gaze information, and the second biometric information is facial expression information. <19> A biometric information detection method, wherein a computer device executes a first detection process for obtaining first biometric information from the plurality of pieces of biometric information and a second detection process for obtaining second biometric information from the plurality of pieces of biometric information, as a process for detecting a plurality of pieces of biometric information about the user wearing the device using a captured image obtained by imaging with an imaging unit having a pixel array unit in which pixels that receive reflected light from an imaging target area including an eyeball region of the user wearing the device are two-dimensionally arranged, and an imaging optical system that is arranged between the imaging unit and the imaging target area and guides the reflected light to the pixel array unit is designed so that image distortion is greater at the outer edge than at the center of the pixel array unit.

[0149] REFERENCE SIGNS LIST 1 Biometric information acquisition device 1A HMD 2 Display unit 3 Light source 6 Imaging optical system 7 Imaging unit 9 Pixel array unit Aim Imaging target area Ar1 First region of interest (first region) Ar2 Second region of interest (second region) F51 Display control unit F52 Region of interest setting unit F53 Light emission control unit F55 Biometric information detection unit F81 Lens distortion correction unit G Pixel Gr Captured image L1 First lens unit L2 Second lens unit L3 Third lens unit U Wearer

Claims

1. A biometric information acquisition device comprising: an imaging unit having a pixel array unit in which pixels that receive reflected light from an imaging target area including the eyeball area of ​​the user wearing the device are arranged two-dimensionally; an imaging optical system that is arranged between the imaging unit and the imaging target area and directs the reflected light to the pixel array unit; and a biometric information detection unit that detects multiple pieces of biometric information about the user wearing the device using an image obtained by imaging by the imaging unit, wherein the imaging optical system is designed to have greater image distortion at the outer edge of the pixel array unit than at the center, and the biometric information detection unit performs a first detection process to obtain first biometric information from the multiple pieces of biometric information, and a second detection process to obtain second biometric information from the multiple pieces of biometric information.

2. The biometric information acquisition device according to claim 1, wherein, for 100% of the image height of the imaging optical system, when the ideal image height is H10i and the actual image height is H10a, the following conditional expression (1) is satisfied: (1) (H10a-H10i) / H10i*100<-20 3. The biometric information acquisition device according to claim 2, wherein, for 50% of the image heights of the imaging optical system, when the ideal image height is H5i and the actual image height is H5a, the following conditional expression (2) is satisfied: (2) (H5a-H5i) / H5i*100<-5 4. The biometric information acquisition device according to claim 2, wherein the imaging optical system has a first lens unit and a second lens unit arranged in this order from the object side to the image side, the first lens unit being a lens unit with a convex surface facing the object side and having negative refractive power, and the second lens unit being a lens unit with a convex surface facing the image side and having positive refractive power.

5. The biometric information acquisition device according to claim 4, wherein the imaging optical system has a third lens unit disposed closest to the image side and having no lens power, and the third lens unit satisfies the following conditional expression (3) when the thickness in the optical axis direction is Dp and the thickness in the entire imaging optical system in the optical axis direction is Da: (3) 0.2<Dp / Da<0.7 6. The biometric information acquisition device according to claim 1, wherein the biometric information detection unit performs processing on a first region on the pixel array unit in the first detection processing, and performs processing on a second region on the pixel array unit that is wider than the first region in the second detection processing.

7. The biometric information acquisition device according to claim 6, further comprising a lens distortion correction unit that corrects image distortion caused by the imaging optical system.

8. The biometric information acquisition device according to claim 7, wherein the lens distortion correction unit performs distortion correction only on an area corresponding to the first area in detecting the first biometric information.

9. The biometric information acquisition device according to claim 6, further comprising a region of interest setting unit that sets the first region as a first region of interest in detecting the first biometric information, wherein the biometric information detection unit performs processing targeting only the first region of interest in the first detection processing.

10. The biometric information acquisition device according to claim 9, wherein the region of interest setting unit sets the imaging unit to output only pixel data relating to the first region of interest when detecting the first biometric information.

11. The biometric information acquisition device according to claim 6, wherein the biometric information detection unit detects the first biometric information more frequently than the second biometric information.

12. The biometric information acquisition device according to claim 11, wherein the first biometric information is detected for each imaging frame of the imaging unit, and the second biometric information is detected for each of a plurality of the imaging frames.

13. The biometric information acquisition device according to claim 1, comprising: a light source that irradiates the imaging target area with light; and a light emission control unit that controls the light emission of the light source.

14. The biometric information acquisition device of claim 13, wherein the first biometric information is information obtained by a detection process targeting a first region on the pixel array unit, and the second biometric information is information obtained by a detection process targeting a second region on the pixel array unit that is wider than the first region, and wherein a plurality of light sources are provided, and the light emission control unit, in detecting the first biometric information, causes one of the plurality of light sources to emit light to an area on the subject corresponding to the first region, and in detecting the second biometric information, causes one of the plurality of light sources to emit light to an area on the subject corresponding to the second region.

15. The biometric information acquisition device according to claim 13, wherein the light source is IR light.

16. The biometric information acquisition device according to claim 1, comprising: a display unit that displays a display image to be viewed by the wearer; and a display control unit that controls the display of the display image on the display unit.

17. The biometric information acquisition device according to claim 16, which is a head-mounted display.

18. The biometric information acquisition device according to claim 1, wherein the first biometric information is iris information or gaze information, and the second biometric information is facial expression information.

19. A biometric information detection method, in which a computer device executes a first detection process for obtaining first biometric information from the plurality of pieces of biometric information and a second detection process for obtaining second biometric information from the plurality of pieces of biometric information, as a process for detecting multiple pieces of biometric information about the user using an image obtained by imaging with an imaging unit having a pixel array unit in which pixels that receive reflected light from an imaging target area including the eyeball area of ​​the user, and an imaging optical system that is arranged between the imaging unit and the imaging target area and directs the reflected light to the pixel array unit is designed so that image distortion is greater at the outer edge of the pixel array unit than at the center.

Images