Living body state detection device and living body state detection method

Abstract

A living state detection device (1) for generating living state information of a person (9) comprises: a light source for emitting light of a first wavelength and a second wavelength; an imaging element (3) for receiving reflected light of the emitted light; a control unit (4) for controlling the light source; a calculation unit (5) for reading a first image (51) and a second image (52) taken from the imaging element (3) and performing calculations; and a state estimation unit (6) for generating living state information of a person (9), wherein the calculation unit (5) generates a distance image (39) based on the first image (51) and the second image (52).

Description

Technical Field

[0001] The present invention relates to a living body state detection device and a living body state detection method, and in particular to a living body state detection device and a living body state detection method for detecting the living body state of an object in a non-contact manner. Background Art

[0002] There is a need to determine whether a worker is in a suitable working state by detecting the worker's living state in a work environment. In order to avoid interfering with the worker's work, a non-contact detector is desirable (see, for example, Patent Document 1).

[0003] The technology disclosed in Patent Document 1 is one of the technologies for detecting living body status by non-contact detection. The pulse wave detection device disclosed in this disclosure can reduce the influence of body movement, thereby enabling optical pulse wave detection with high detection accuracy.

[0004] Prior art literature

[0005] Patent Literature

[0006] Patent Document 1: Japanese Patent Application Laid-Open No. 2017-140202 Summary of the Invention

[0007] Problems to be solved by the invention

[0008] However, the pulse wave detection device described in Patent Document 1 has a problem in that it is limited to detecting pulse waves based on images, and therefore lacks information for comprehensive judgment of the living body's condition.

[0009] Therefore, in order to solve the above-mentioned problems, the present invention aims to provide a living body state detection device and a living body state detection method that can generate more living body information than before by using images.

[0010] Means used to solve the problem

[0011] In order to achieve the above-mentioned purpose, one embodiment of the living state detection device involved in the present invention is a living state detection device that detects the living state of a person, and the living state detection device comprises: a first light source that emits light of a first wavelength; a second light source that emits light of a second wavelength different from the first wavelength; an imaging element composed of a plurality of elements that receive reflected light emitted from the first light source and the second light source and reflected from the person; a control unit that controls the first light source and the second light source so that the first light source and the second light source alternately emit light; and a computing unit. , reading out a first image and a second image from the imaging element, generating a third image by performing an operation on the readout first image and the second image, wherein the first image is obtained by receiving reflected light from the first light source, and the second image is obtained by receiving reflected light from the second light source; and a state estimation unit that generates and outputs living body information showing the living body state of the person based on the third image generated by the operation unit, and the operation unit further generates a distance image based on at least one of the first image and the second image.

[0012] In order to achieve the above-mentioned purpose, one embodiment of the living state detection method of the present invention is a living state detection method for detecting the living state of a person, the living state detection method comprising: a control step of controlling a first light source emitting light of a first wavelength and a second light source emitting light of a second wavelength different from the first wavelength so that the first light source and the second light source alternately emit light; a calculation step of reading a first image and a second image from an imaging element composed of a plurality of elements, and calculating the first image and the second image by calculating the second image. a calculation step to generate a third image, wherein the plurality of elements receive reflected light emitted from the first light source and the second light source and reflected on the person, the first image is obtained by receiving reflected light from the first light source, and the second image is obtained by receiving reflected light from the second light source; and a state estimation step to generate living body information showing the living body state of the person based on the third image generated in the calculation step, and in the calculation step, further generate a distance image based on at least one of the first image and the second image.

[0013] Furthermore, the present invention can be implemented not only as a living body status detection device and a living body status detection method, but also as a program that causes a computer to execute the steps of the living body status detection method, as a computer-readable recording medium such as a CD-ROM storing the program, or as information, data, or signals representing the program. Furthermore, these programs, information, data, and signals can be distributed via a communication network such as the Internet.

[0014] Effects of the Invention

[0015] The living body state detection device and the living body state detection method according to the present invention can generate more living body information than before using images. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a functional block diagram of a living body state detection device in an embodiment.

[0017] Figure 2 This is a diagram for explaining a projected image formed on an imaging element through a lens included in the living body state detection device according to the embodiment.

[0018] Figure 3 This is a diagram for explaining the arrangement relationship between the light source unit and the imaging element included in the living body state detection device in the embodiment.

[0019] Figure 4 It is a diagram for explaining the arrangement relationship between the light source unit and the imaging element included in the living body state detection device in another embodiment.

[0020] Figure 5 This is an enlarged view of a unit element constituting the imaging element included in the living body state detection device according to the embodiment.

[0021] Figure 6 This is a flowchart showing the overall operation of the living body state detection device in the embodiment.

[0022] Figure 7 This is a diagram for explaining determination of a detection area and pulse detection performed by the living body state detection device in this embodiment.

[0023] Figure 8 This is a timing chart of the control of the light source unit and the imaging element included in the living body state detection device in the embodiment.

[0024] Figure 9 It is a diagram for explaining image processing performed by the living body state detection device in the embodiment.

[0025] Figure 10This is a diagram for explaining image processing performed by a computing unit included in the living body state detection device in the embodiment.

[0026] Figure 11 This is a flowchart showing the overall operation of the state estimation unit included in the living body state detection device in the embodiment.

[0027] Figure 12 This is a diagram for explaining an example of an alarm display of a living body state detection device installed in a vehicle according to an embodiment. DETAILED DESCRIPTION

[0028] An embodiment of the present invention is described in detail below using the accompanying drawings. In addition, the embodiments to be described below are all preferred specific examples of the present invention. The numerical values, shapes, materials, constituent elements, configuration positions of constituent elements, connection methods, order of operation, etc. shown in the following embodiments are examples, and their purpose is not to limit the present invention. Moreover, among the constituent elements of the following embodiments, the constituent elements that are not recorded in the independent technical solutions showing the highest concept of the present invention are described as arbitrary constituent elements constituting a more preferred method.

[0029] First, use Figure 1 The constituent elements in the embodiment will be described.

[0030] Figure 1 1 is a functional block diagram of the living body state detection device 1 in the embodiment. Figure 1 In addition to the living state detection device 1, a person 9 serving as the subject of living state detection is shown. Furthermore, regarding the outgoing light from the light source unit 2 and the reflected light reflected by the person 9, the outgoing light 21a and the reflected light 21b from the first light source 21 are shown as dashed lines oriented in the direction of travel, and the outgoing light 22a and the reflected light 22b from the second light source 22 are shown as dashed lines oriented in the direction of travel. Furthermore, the background light 10a irradiated from the background light source 10 to the imaging element 3 of the living state detection device 1 is shown as a dashed line oriented in the direction of travel.

[0031] This living body state detection device 1 is a device that detects the living body state of a person 9 and is used, for example, to monitor the health of a person 9 in various environments, such as a driver, a factory operator, a company employee, or a student. This living body state detection device 1 includes a light source unit 2, an imaging element 3, a control unit 4, a computing unit 5, a state estimation unit 6, a storage unit 7, and a display device 8.

[0032] The light source unit 2 is composed of a first light source 21 that emits light 21a having a first wavelength, and a second light source 22 that emits light 22a having a second wavelength different from the first wavelength. The first light source 21 and the second light source 22 are each implemented by an independent light source (e.g., a semiconductor laser, etc.), and both emit predetermined monochromatic light.

[0033] Furthermore, the imaging element 3 is composed of a plurality of elements that receive reflected light emitted from the first light source 21 and the second light source 22 and reflected by the person 9. Specifically, the imaging element 3 is an image sensor such as a CMOS (Complementary Metal Oxide Semiconductor) or a CCD (Charge Coupled Device) having a photoelectric conversion element implemented by a photodiode, etc., and converts the received light into an electrical signal corresponding to the amount of light. In this embodiment, the imaging element 3 is composed of a first element 31 corresponding to the reflected light 21b (i.e., for receiving the reflected light 21b) and a second element 32 corresponding to the reflected light 22b (i.e., for receiving the reflected light 22b).

[0034] Outgoing light 21a of the first wavelength emitted from the first light source 21 is reflected by the person 9 and other objects in the imaging space, becoming reflected light 21b of the first wavelength, which then enters at least the first element 31. Similarly, outgoing light 22a of the second wavelength emitted from the second light source 22 is reflected by the person 9 and other objects in the imaging space, becoming reflected light 22b of the second wavelength, which then enters at least the second element 32. The first element 31 and the second element 32 perform exposure in conjunction with the emission of the corresponding light sources. They receive background light 10a emitted from a background light source 10 such as external light such as sunlight, along with the reflected light 21b and 22b, generate electric charge through photoelectric conversion, and output these as electrical signals.

[0035] Furthermore, the control unit 4 is a processing unit that controls the first light source 21 and the second light source 22 so that they alternately emit light. The control unit 4 also controls the imaging element 3 so that the first element 31 and the second element 32 perform exposure in conjunction with the emission of light from the corresponding light sources. More specifically, the control unit 4 is implemented by a ROM and RAM storing a control program, and a CPU executing the control program. Furthermore, the living body state detection device 1 includes an input unit for receiving instructions from the user, and the control unit 4 can receive user instructions via the input unit.

[0036] The computing unit 5 is a processing unit that reads out the first image obtained by receiving the reflected light 21b of the light from the first light source 21 and the second image obtained by receiving the reflected light 22b from the imaging element 3, and generates a third image by performing calculations on the readout first and second images. Specifically, the computing unit 5 is implemented by a ROM and RAM storing a calculation program, and a CPU executing the calculation program. This ROM, RAM, CPU, and other hardware can be shared with the hardware implementing the control unit 4.

[0037] The calculation unit 5 is also a processing unit that generates a luminance image and a distance image based on the first image, the second image, or both. The luminance image is an image composed of pixel values ​​indicating brightness (i.e., luminance). The distance image is an image composed of pixel values ​​indicating distance.

[0038] The state estimation unit 6 is a processing unit that generates biological information indicating the biological state of the person 9 based on the third image generated by the calculation unit 5. Specifically, the state estimation unit 6 is implemented by a ROM storing an estimation program, a RAM, and a CPU that executes the estimation program. This ROM, RAM, CPU, and other hardware can be shared with the hardware that implements the control unit 4.

[0039] The state estimation unit 6 further includes a detection region determination unit 61 that determines a detection region from the third image. The detection region is a region used for generating biological information. The state estimation unit 6 generates biological information using the detection region determined by the detection region determination unit 61 .

[0040] Furthermore, the storage unit 7 is a storage device that contains a face detection library for face detection required for generating living body information, and stores various setting values, such as a distance reference value and distance correction value used in generating distance images, and a threshold value used for comparing living body information. Furthermore, the storage unit 7 may include a volatile storage area for temporarily storing generated image information during processing by the state estimation unit 6.

[0041] Furthermore, the display device 8 is a device that notifies the person 9, or the administrator who manages the person 9, or both, of the estimated state of the person 9 by the state estimation unit 6, and is comprised of, for example, a liquid crystal display. Furthermore, if the notification is simple, such as simply indicating whether the person 9 is in good or bad condition, a simple device such as a warning light may be used in place of the display device 8. Furthermore, although not shown, in addition to the display device 8, a sound device that emits a buzzer or provides voice guidance to notify the person 9 of the estimation result output by the state estimation unit 6, or a vibration generating device that vibrates to provide notification, may also be provided. Furthermore, these devices may be provided in addition to the display device 8.

[0042] Furthermore, all of the aforementioned components may be incorporated into a single housing. Alternatively, the light source unit 2 and the imaging element 3 may be incorporated into the single housing, and the remaining components may be implemented on a computer connected to the housing by wire or wireless connection. Furthermore, the display device 8 may be separately connected to the state estimation unit 6 by wire or wireless connection, so that it can be positioned in a location that facilitates visual confirmation of the person to be notified.

[0043] Although not shown, the living body state detection device 1 includes a lens for focusing reflected light 21 b and 22 b emitted from the first light source 21 and the second light source 22 and reflected by the person 9 onto the imaging element 3 .

[0044] Here, regarding the characteristics of this lens, Figure 2 To explain.

[0045] Figure 2 This is a diagram for explaining a projection image formed on the imaging element 3 through a lens included in the living body state detection device 1 according to the embodiment.

[0046] Figure 2 (a) shows a projection image 3a formed on the imaging element 3 when a general spherical or aspherical lens is used, and Figure 2 (b) shows a projection image 3b formed on the imaging element 3 when an optically adjusted local magnification lens (ie, a free-form surface lens) is used. As an example, a person 9 existing in the imaging space is reflected in the projection images 3a and 3b.

[0047] In projected image 3a, facial area A1, as viewed directly within the capture space and reproduced on the surface of imaging element 3, and the area outside facial area A1 are imaged at the same magnification. Facial area A1 is encircled by a dotted-line frame. Furthermore, in projected image 3b, when the image formed on imaging element 3 corresponds to the upper body of person 9, a portion of the lens is optically adjusted to magnify the portion corresponding to facial area A1z. Therefore, in projected image 3b, facial area A1z and the area outside facial area A1z have different magnifications (i.e., different resolutions).

[0048] The living body state detection device 1 in this embodiment may include a lens as shown in the projection image 3 a , and may also include a lens as shown in the projection image 3 b when assuming an application that places emphasis on the face.

[0049] Next, use Figures 3 to 5 , the configuration of the light source unit 2 and the imaging element 3 will be described in detail.

[0050] Figure 31 and 2 are diagrams for explaining the arrangement relationship between the light source unit 2 and the imaging element 3 included in the living body state detection device 1 in this embodiment.

[0051] Figure 3 The figure above is a plan view of the surface of the living body state detection device 1 including the light source unit 2 and the imaging element 3. Figure 3 The lower figure is an enlarged view of the imaging element 3. The unit elements constituting the imaging element 3 are arranged in a grid pattern. The corresponding corners of the upper and lower figures are connected by a dotted line. For simplicity of explanation, the lower figure shows only a portion of the grid surface formed by the unit elements.

[0052] In the figure above, the first light source 21 and the second light source 22, which constitute the light source unit 2, are arranged at either end of the imaging element 3 across from the living body state detection device 1. The first and second light sources 21, 22, which constitute the light source unit 2, are arranged on the same plane as the imaging element 3. To minimize the difference in optical distance between the first and second light sources 21, 22, and the distance between the emitted light from the light source to the object and the reflected light from the object to the imaging element 3, the first and second light sources 21, 22 are arranged on a circle c centered on the imaging element 3. Furthermore, to minimize the difference in the emission angles relative to the reflected object, the first and second light sources 21, 22 are preferably arranged adjacent to each other within the permitted installation range. Furthermore, by arranging the pair of first and second light sources 21, 22 on the circle c centered on the imaging element 3, additional light sources can be added to increase the amount of emitted light.

[0053] Regarding the imaging element 3, the first elements 31 and the second elements 32 are arranged adjacent to each other in at least one of the vertical, horizontal, and leftward directions when the imaging element 3 is viewed in a plan view, so that the arrangement of the first elements 31 and the second elements 32 corresponding to the respective light sources is not biased. In the example shown in this embodiment, when the imaging element 3 is viewed in a plan view, the first elements 31 and the second elements 32 are arranged alternately in a horizontal stripe pattern in the horizontal rows and vertical columns in which the first elements 31 are arranged.

[0054] Here, using Figure 4 Another embodiment of the arrangement relationship between the light source unit 2 and the imaging element 3 will be described.

[0055] Figure 4 This is a diagram for explaining another form of the arrangement relationship between the light source unit 2b and the imaging element 3b included in the living body state detection device 1 in the embodiment.

[0056] Figure 4The figure above is a plan view of the surface including the light source unit 2b and the imaging element 3b in another embodiment of the living body state detection device 1. Figure 4 The lower figure is an enlarged view of imaging element 3b. The unit elements that make up imaging element 3b are arranged in a grid pattern. Corresponding corners in the upper and lower figures are connected by dotted lines. For simplicity, the lower figure shows only a portion of the grid surface formed by the unit elements.

[0057] Relative to Figure 3 In terms of the light source portion 2b, the light sources are concentrated on the right side of the figure, and there is no light source on the left side. Figure 3 Compared with the arrangement of the light sources in the embodiment of the present invention, although the light sources are arranged closer to the imaging element 3, since the pair of light sources (the first light source 21x and the second light source 22x) are arranged on the circumference cx centered on the imaging element 3b, the living state can be detected only with this pair of light sources. Figure 3 In the embodiment, although the light sources are replaced, since the pair of light sources (the first light source 21y and the second light source 22y) are arranged on the circumference cy centered on the imaging element 3b, the living body state can be detected only with this pair of light sources. Figure 3 Compared with the configuration of the light source in the embodiment, it is configured at a position far away from the camera element 3. Although the light sources are concentrated at the bottom in the figure, since these pairs of light sources (the first light source 21z and the second light source 22z) are configured on the circle cz centered on the camera element 3b, only configuring this pair of light sources can detect the living state.

[0058] Image pickup element 3b and Figure 3 Compared to the imaging element 3, the first elements 31 and second elements 32 are arranged alternately in both the vertical and horizontal directions, forming a houndstooth pattern. In this embodiment, the first elements 31 and second elements 32 are arranged adjacent to each other in all directions, vertically, horizontally, and horizontally. Furthermore, in this case, it is necessary to control the light received by each unit element individually, so the use of a CMOS image sensor, a solid-state imaging element, is considered.

[0059] Here, further use Figure 5 , the first element 31 and the second element 32 are described in detail.

[0060] Figure 5 It is an enlarged view of a unit element constituting the imaging element 3 included in the living body state detection device 1 in the embodiment.

[0061] Figure 5 The figure below shows the Figure 3 The same imaging element 3 as in the lower figure is further enlarged in the upper figure to show the 24 unit elements of the imaging element 3 in the lower figure, which are 4 rows from the top and 6 columns from the left. In addition, in this embodiment, the exposure of the imaging element 3 can be divided into multiple timings, and the elements of the imaging element 3 that accumulate the charge generated in each exposure timing are particularly shown here. And, in Figure 5 In the figure below, the unit elements corresponding to the figure above are circled in squares, and the corresponding corners are connected by dotted lines.

[0062] The unit element of the first element 31 further receives the reflected light 21b in addition to the background light 10a. Figure 5 The first element 31 is composed of four elements (not shown) which are a first light-receiving element that generates electric charges by photoelectric conversion, an element 311 that accumulates electric charges generated at the first exposure timing, an element 312 that accumulates electric charges generated at the second exposure timing, and an element 313 that accumulates electric charges generated at the third exposure timing. The unit elements of the second element 32 are similarly composed of a second light-receiving element, an element 321, an element 322, and an element 323. The elements 311 to 313 of the first element 31 accumulate electric charges sequentially generated by the emission of the corresponding first light source 21, and output electric signals corresponding to the amount of light received by the first element 31 at their respective exposure timings. Furthermore, the elements 321 to 323 of the second element 32 are similarly composed of electric charges sequentially generated by the emission of the corresponding second light source 22, and output electric signals corresponding to the amount of light received by the second element 32 at their respective exposure timings. Next, the operation of the living body state detection device 1 in this embodiment constructed as described above is described using Figures 6 to 12 Provide explanation.

[0063] First, use Figure 6 as well as Figure 7 Next, the overall operation of the living body state detection apparatus 1 in this embodiment and the pulse detection using images in the state estimation unit 6 will be described.

[0064] Figure 6 1 is a flowchart showing the overall operation of the living body state detection device 1 in the embodiment (ie, the living body state detection method). Figure 7 This is a diagram explaining determination of a detection area and pulse detection performed by the living body state detection device 1 in the embodiment.

[0065] First, the user of the living body state detection device 1 sets up the living body state detection device 1 at a position where the person 9 to be detected enters the imaging space of the living body state detection device 1 and starts operating it.

[0066] The life state detection device 1 then begins generating life information indicating the life state of the person 9 based on pre-set settings or settings set by the user at the start of operation. First, the control unit 4 of the life state detection device 1 controls the alternating emission of outgoing light 21a from the first light source 21 and outgoing light 22a from the second light source 22 (control step S101). The outgoing light 21a and 22a are reflected by objects within the imaging space, including the person 9, and received as reflected light 21b and 22b by the first element 31 and the second element 32 (S102). The received light includes background light 10a emitted from the background light source 10. The calculation unit 5 calculates the first and second images read from the first element 31 and the second element 32 to generate a third image 37, a luminance image 38, and a distance image 39 (calculation step S103). The detection region specifying unit 61 of the state estimation unit 6 specifies a detection region 62 for determining a region in the third image 37 used for generating biological information based on the generated luminance image 38 and distance image 39 (state estimation step S104 ).

[0067] More specifically, if Figure 7 As shown, the detection area determination unit 61 of the state estimation unit 6 obtains the luminance image 38 and the distance image 39 generated by the calculation unit 5. Here, the luminance image 38 and the distance image 39, as shown in the image examples 38a and 39a of the luminance image 38 and the distance image 39, respectively, are images showing the luminance and distance of the space in which the face of person 9 is captured. The detection area determination unit 61 extracts the outline of the pixels located in the face of person 9 from the obtained luminance image 38 and generates a circumscribed rectangle, thereby detecting a face detection area 38b. In the image example 38a, the face and the detected face detection area 38b are enclosed by a dotted box with the corners emphasized.

[0068] Next, the detection area determination unit 61 selects pixels from the distance image 39 corresponding to the face portion previously detected in the luminance image 38. Among the pixels corresponding to the face detection area 38b selected in the distance image 39, pixels located at approximately the same distance from the center pixel of the distance image 39 (i.e., an equal distance range) are selected as the detection area example 62b. In other words, the area excluding the background of the face in the distance image 39 is selected as the detection area example 62b. Because the distance between the pixels in the distance image 39 and the pixels corresponding to the face detected in the luminance image 38 reflects the distance to the face, the distance between these pixels is approximately equal. In the image example 62a of the detection area 62, the face detection area 38b is superimposed on the image example 39a. Furthermore, the range of approximately equal distance information within the face detection area 38b is selected as the detection area example 62b.

[0069] exist Figure 6 In the process, after the detection area 62 is determined, the state estimation unit 6 processes the third image 37 output from the operation unit 5 according to the determined detection area 62, and generates a processed third image 63 having only information of the portion corresponding to the detection area 62 (S105).

[0070] Figure 7 Image example 37a of the third image 37 shown is also an image captured in the same frame as image examples 38a and 39a, and includes the face of person 9. The state estimation unit 6 overlays the detection area 62 previously obtained in the same frame on the third image 37 and deletes image information at locations where there is no overlap from the third image 37. The state estimation unit 6 temporarily stores the processed third image 63 obtained through this process in the storage unit 7. Furthermore, in image example 63a of the processed third image 63, detection area example 62b is overlaid on image example 37a.

[0071] After generating the processed third image 63, the state estimation unit 6 generates biological information such as the pulse, respiration, and posture of the person 9 based on the processed third image 63, the brightness image 38, and the distance image 39 accumulated in the multiple frames of the storage unit 7 (S106).

[0072] Regarding the generation process of the living body information, the third image 63 after the processing is used as an example. Figure 7 Let's explain in detail.

[0073] The detection area determination unit 61 applies the above-described series of processing to all third images 37 of the frames sequentially obtained by the calculation unit 5, and the processed third images 63 obtained for each frame are accumulated in the storage unit 7. The accumulated series of processed third images 63 show the changes in the processed third images 63 with respect to time. In other words, these continuously accumulated processed third images 63 are moving images 63b of the third images 37 showing the changes in the face over time.

[0074] Here, vascular dilation, or blood flow, is related to the amount of hemoglobin. Using light of a wavelength that is highly reflective to hemoglobin yields reflected light corresponding to the amount of hemoglobin per unit time of irradiation. Furthermore, wavelengths that are highly reflective to hemoglobin are those that are less easily absorbed by body tissue and in which at least a portion of the incident light is reflected by hemoglobin. Utilizing this reflected light allows for non-interferometric observation of vascular dilation. Furthermore, by plotting the temporal changes in vascular dilation, pulse estimation is possible. In other words, pulse estimation using images is possible without interference.

[0075] The first obtained third image, moving image 63b, is a moving image in which reflected light of light of an arbitrary wavelength irradiated on the face is arranged in a temporal pattern. By using the arbitrary wavelength as a wavelength with high reflectivity to hemoglobin, the pulse of person 9 can be estimated. Accordingly, the state estimation unit 6 analyzes the third image, moving image 63b, estimates the pulse of person 9, generates an estimated pulse graph 63c, and generates an estimated pulse value as biological information.

[0076] The same is true for the luminance image 38 and the distance image 39 . The state estimation unit 6 can generate various biological information by analyzing the moving images of the luminance image and the moving images of the distance image.

[0077] exist Figure 6 After generating the biometric information, the state estimation unit 6 determines whether the generated biometric information contains information indicating a value that deviates from a preset reference value (S107). If not (No in S107), the state estimation unit 6 causes the display device 8 to display a list or a portion of the currently obtained biometric information (S108). Furthermore, if information indicating a value that deviates from a preset reference value is present (Yes in S107), the state estimation unit 6 causes the display device 8 to display an alarm, notifying the user of the presence of an abnormal value (S109). Thereafter, the state estimation unit 6 further causes the display device 8 to display a list or a portion of the biometric information, including the information indicating the abnormal value (S108).

[0078] Then, Figure 6 as well as Figure 7 Each operation of the overall operation of the living body state detection device 1 shown will be described in detail.

[0079] First, use Figure 8 The light emission of the light source unit 2 and the light reception processing of the imaging element 3 are described in detail. Figure 6 Step S101 and step S102 in the flowchart.

[0080] Figure 8 1 is a timing chart showing control of the light source unit 2 and the imaging element 3 included in the living body state detection device 1 according to the embodiment.

[0081] exist Figure 8 In the figure, the horizontal axis represents the passage of time, and the vertical axis represents the emission of the light source or the opening (ON) and closing (OFF) of the exposure of the imaging element 3. Here, the upper side of the vertical axis represents the ON control (open control), and the lower side of the vertical axis represents the OFF control (closed control). In addition, since each element receives light at two or three different exposure timings, the light is received by the imaging element 3 at two or three different exposure timings. Figure 5 The following description will be made using the elements 311 and 321, the elements 312 and 322, and the elements 313 and 323. When receiving light at two different exposure timings, the first element 31 and the second element 32 may not include the elements 312 and 322.

[0082] Figure 8 (a) is a timing diagram showing that, within one frame period, the emission of the first light source 21 (first timing diagram p101) and the light reception of the first element 31 (second timing diagram p102), the emission of the second light source 22 (third timing diagram p103) and the light reception of the second element 32 (fourth timing diagram p104), and the light reception of the background light 10a in the first element 31 and the second element 32 (fifth timing diagram p105) are controlled once. Here, since light reception is performed at two different timings, the control is performed using Figure 5 Components 311 and 321, and components 313 and 323 are described.

[0083] First, in the first timing chart p101 corresponding to the emission of the first light source 21 , the emission of the first light source 21 is started, and the emission is stopped after the time of the first pulse width w1 has elapsed.

[0084] In the second time diagram p102 corresponding to the light reception of the first element 31, charge accumulation in the element 311 starts simultaneously with the start of emission of the first light source 21, and stops after the second pulse width w2 has elapsed, thereby obtaining an electrical signal corresponding to the first image.

[0085] Furthermore, in the third timing diagram p103 corresponding to emission of the second light source 22, emission of the second light source 22 starts after the first interval w3 has elapsed since charge accumulation in the element 311 stops, and stops after the first pulse width w1 has elapsed.

[0086] In the fourth time diagram p104 corresponding to the light reception of the second element 32, the charge accumulation of the element 321 starts simultaneously with the start of emission of the second light source 22, and stops after the second pulse width w2 has elapsed, thereby obtaining an electrical signal corresponding to the second image.

[0087] Moreover, in the fifth time diagram p105 corresponding to the reception of the background light 10a in the first element 31 and the second element 32, charge accumulation in the elements 313 and 323 starts after the second interval w4 has passed since the charge accumulation in the element 321 stopped, and the charge accumulation stops after the second pulse width w2 has passed, thereby obtaining an electrical signal corresponding to the fourth image.

[0088] The above operations are combined as a set of controls and executed once per frame period. In this time chart, in addition to the first and second images captured at two wavelengths, a fourth image can be obtained. The calculation unit 5 generates a first image and a second image that are differential images after background light removal.

[0089] and, Figure 8 (b) shows the Figure 8 (a) is a diagram of a group of controls divided into multiple times. Figure 8 In (b), immediately after the emission of the first light source 21 (6th timing diagram p201) and the reception of light by the first element 31 (7th timing diagram p202), the reception of the background light 10a by the first element 31 is performed (8th timing diagram p203). Immediately after the emission of the second light source 22 (9th timing diagram p204) and the reception of light by the second element 32 (10th timing diagram p205), the reception of the background light 10a by the second element 32 is performed (11th timing diagram p206). Here, since light is received at two different timings, the light is received by the second element 32. Figure 5 Components 311 and 321, and components 313 and 323 are described.

[0090] First, in the sixth time chart p201 corresponding to the emission of the first light source 21 , the emission of the first light source 21 is started, and after the time of the third pulse width w5 has elapsed, the emission is stopped.

[0091] In the seventh time pattern p202 corresponding to light reception by the first element 31 , charge accumulation in the element 311 starts simultaneously with the start of emission from the first light source 21 , and stops after the fourth pulse width w6 has elapsed.

[0092] Next, in the eighth time diagram p203 corresponding to the reception of the background light 10a in the first element 31, charge accumulation in the element 313 starts after the third interval w7 has passed since the charge accumulation in the element 311 stopped, and stops after the third pulse width w5 has passed.

[0093] Furthermore, in the 9th time diagram p204 corresponding to the emission of the second light source 22, the emission of the second light source 22 starts after the 4th interval w8 has passed since the charge accumulation of the element 313 stops, and stops after the 3rd pulse width w5 has passed.

[0094] In the tenth time diagram p205 corresponding to light reception by the second element 32, charge accumulation in the element 321 starts simultaneously with the start of emission of the second light source 22, and stops after the fourth pulse width w6 has elapsed.

[0095] Next, in the 11th time diagram p206 corresponding to the reception of the background light 10a in the second element 32, charge accumulation in the element 323 starts after the third interval w7 has passed since the charge accumulation in the element 321 stopped, and stops after the third pulse width w5 has passed.

[0096] The above operations are controlled as a set, and multiple sets of emission and reception are repeated within a single frame period to accumulate the charge stored in each element. Similarly, in this time chart, in addition to the first and second images captured at two wavelengths, a fourth image can be obtained within a single frame period. The calculation unit 5 generates a differential first image and a differential second image, eliminating background light. In this time chart, because emission and reception are repeated multiple times, imaging can be performed while accurately understanding the changes occurring within a single frame period. Furthermore, because the difference in charge accumulation timing between elements 311 and 313 is the same as the difference in charge accumulation timing between elements 321 and 323, the difference in imaging between the two light sources can be reduced.

[0097] And, in Figure 8 In (c), except Figure 8 In addition to (b), in order to be able to shoot in TOF (Time of Flight) mode, the light received by the first element 31 and the second element 32 is divided into two timings on the time axis. Here, since light is received at three different timings, Figure 5 The components 311 and 321, components 312 and 322, and components 313 and 323 are described.

[0098] First, in the twelfth time chart p301 corresponding to the emission of the first light source 21 , the emission of the first light source 21 is started, and after the time of the third pulse width w5 has elapsed, the emission is stopped.

[0099] In the thirteenth time diagram p302 corresponding to charge accumulation in the element 311 , charge accumulation in the element 311 starts simultaneously with the start of emission from the first light source 21 , and stops after the third pulse width w5 has elapsed.

[0100] Furthermore, in the fourteenth time diagram p303 corresponding to charge accumulation in the element 312 , charge accumulation in the element 311 is stopped and charge accumulation in the element 312 is started. After the time of the third pulse width w5 has elapsed, charge accumulation is stopped.

[0101] Next, in the 15th time diagram p304 corresponding to the reception of the background light 10a in the first element 31, charge accumulation in the element 313 starts after the fifth interval w9 has passed since the charge accumulation in the element 312 stopped, and stops after the third pulse width w5 has passed.

[0102] Furthermore, in the 16th time diagram p305 corresponding to the emission of the second light source 22, the emission of the second light source 22 starts after the sixth interval w10 has passed since the charge accumulation of the element 313 stops, and stops after the third pulse width w5 has passed.

[0103] In the seventeenth time diagram p306 corresponding to light reception by the element 321 , charge accumulation in the element 321 starts simultaneously with the start of emission from the second light source 22 , and stops after the third pulse width w5 has elapsed.

[0104] Furthermore, in the 18th time graph p307 corresponding to charge accumulation in the element 322 , charge accumulation in the element 321 is stopped and charge accumulation in the element 322 is started. After the time of the third pulse width w5 has elapsed, charge accumulation is stopped.

[0105] Next, in the 19th time diagram p308 corresponding to the reception of the background light 10a in the second element 32, charge accumulation in the element 323 starts after the fifth interval w9 has passed since the charge accumulation in the element 322 stopped, and stops after the third pulse width w5 has passed.

[0106] The above operations are controlled as a set, and multiple sets of emission and reception are repeated within a single frame period, accumulating the charge stored in each element. In this time chart, the time of flight analysis is used to calculate the distance between the imaging element 3 and objects within the captured space.

[0107] In addition, the description in this embodiment is equivalent to Figure 8 By emitting and receiving light (c), the computing unit 5 can obtain a distance image using the TOF method.

[0108] Here, a method of calculating distance in the TOF method will be described using the following formula (1).

[0109] [Formula 1]

[0110]

[0111] Formula (1) is a calculation formula for calculating the distance from an object existing in the imaging space to the imaging element 3 at the first wavelength. W is the distance from the object in the imaging space to the imaging element 3, c is the speed of light, and T W represents the irradiation pulse width of the light of the first wavelength, t1 represents the amount of light received at the first exposure timing, t2 represents the amount of light received at the second exposure timing, and t0 represents the amount of light received at the third exposure timing. More specifically, 3.00×10 8 , in T W The calculation is performed by substituting the third pulse width w5 into , substituting the charge amount of the element 311 at t1, substituting the charge amount of the element 312 at t2, and substituting the charge amount of the element 313 at t0.

[0112] Furthermore, the distance from the object existing in the imaging space at the second wavelength to the imaging element 3 is also calculated using the same formula. More specifically, 3.00×10 8 , in T W The calculation is performed by substituting the third pulse width w5 into , substituting the charge amount of the element 321 at t1, substituting the charge amount of the element 322 at t2, and substituting the charge amount of the element 323 at t0.

[0113] Next, use Figure 9 The process of generating biological information from the electrical signals accumulated in the imaging element 3 as described above will be described. Figure 6 This corresponds to steps S103 to S108 in the flowchart.

[0114] Figure 9 It is a diagram for explaining image processing performed by the living body state detection device 1 in the embodiment.

[0115] exist Figure 9 The following series of image processing procedures are shown in FIG. 5 , that is, the operation unit 5 reads and processes the image from the imaging element 3, and the image processed by the operation unit 5 is further processed by the state estimation unit 6 to estimate the state of the living body. Figure 9 2 , it is also shown that, in the image processing of the state estimation unit 6 , the image is temporarily stored in the storage unit 7 , and the estimation result is output to the display device 8 .

[0116] First, the computing unit 5 reads image information obtained from the imaging element 3 and performs processing to calculate differences and / or synthesize the image information, thereby generating three types of image information. Specifically, the computing unit 5 reads from the first element 31 and the second element 32 a first image 51 obtained when a light source with a first wavelength is emitted, and a second image 52 obtained when a light source with a second wavelength is emitted. Furthermore, the computing unit 5 reads from the imaging element 3 fourth images 53a and 53b, which were captured only in an environment without background light 10a from the light source, from the first element 31 and the second element 32. The computing unit 5 calculates the difference between the first image 51 and the fourth image 53a read from the imaging element 3, thereby removing the background light 10a and outputting the differential first image 51a. Similarly, the computing unit 5 calculates the difference between the second image 52 and the fourth image 53b read from the imaging element 3, thereby removing the background light 10a and outputting the differential second image 52a.

[0117] Here, the differential first image 51a includes the first luminance image 33 and the first distance image 35. Furthermore, the differential second image 52a includes the second luminance image 34 and the second distance image 36. The computing unit 5 calculates the difference between the first luminance image 33 and the second luminance image 34 to generate a third image 37 for use in generating biological information. Furthermore, the computing unit 5 synthesizes the first luminance image 33 and the second luminance image 34 to generate a luminance image 38 for use in determining the detection area 62. Furthermore, the computing unit 5 synthesizes the first distance image 35 and the second distance image 36 to generate a distance image 39 for use in determining the detection area 62.

[0118] The calculation unit 5 further stores the generated luminance image 38 and distance image 39 in the storage unit 7. As described above, the calculation unit 5 generates a luminance image moving image 71 and a distance image moving image 72 by storing and accumulating the sequentially acquired frames in the storage unit 7.

[0119] Next, the detection region specifying unit 61 of the state estimation unit 6 specifies a detection region 62 used when detecting biological information, using the luminance image 38 and the distance image 39 generated by the calculation unit 5 .

[0120] The state estimation unit 6 generates a processed third image 63, which includes image information of only the portion corresponding to the detection area 62, from the third image 37 generated by the calculation unit 5, using the detection area 62, and stores the processed third image 63 in the storage unit 7. The state estimation unit 6 applies the above-described processing to the sequentially obtained frames to generate a moving image 63b of the third image.

[0121] The state estimation unit 6 generates biological information of the person 9 using the moving image 71 of the luminance image, the moving image 72 of the distance image, and the moving image 63b of the third image stored in the storage unit 7. The state estimation unit 6 displays the biological information generated in this manner on the display device 8.

[0122] Next, use Figure 10 The image processing performed by the computing unit 5 in this embodiment will be described in more detail. Figure 6 This corresponds to step S103 in the flowchart.

[0123] Figure 10 It is a diagram for explaining image processing performed by the calculation unit 5 included in the living body state detection device 1 in the embodiment.

[0124] exist Figure 10 shows the imaging element 3 and image information. This image information is formed by arranging pixels corresponding to the unit elements of the imaging element 3 in a manner corresponding to the array of the unit elements. For ease of explanation, the right and bottom ends of the imaging element 3 and each image information are omitted. The imaging element 3 is composed of unit elements arranged in N rows in the vertical direction and M columns in the horizontal direction (N and M are integers greater than or equal to 2). First elements 31 are arranged in odd-numbered rows in the vertical direction, and second elements 32 are arranged in even-numbered rows, alternating in this manner.

[0125] Here, the first image 51 captured by the first element 31 includes brightness information and distance information. The brightness information here refers to the amount of light received by each pixel constituting the first image 51, and the distance information here indicates the distance between the object in the shooting space and the imaging element 3 obtained by the TOF method. The same is true for the second image 52 captured by the second element 32. Figure 10In the first image 51, the image showing brightness information is shown as the first brightness image 33, the image showing distance information is shown as the first distance image 35, and the image showing brightness information is shown as the second brightness image 34, the image showing distance information is shown as the second distance image 36. Figure 10 3 , a third image 37 , a brightness image 38 , and a distance image 39 obtained by the processing of the calculation unit 5 are also shown.

[0126] When imaging element 3 completes its image capture, computing unit 5 reads out electrical signals representing image information stored in elements 311 and 321, and elements 312 and 322 of imaging element 3. At this point, processing is performed to extract the amount of light received by first element 31, arranged in odd rows, to produce first image 51, while processing is performed to extract the amount of light received by second element 32, arranged in even rows, to produce second image 52. In other words, two images containing N / 2 rows and M columns of pixels, each with half the vertical information volume, are generated from the unit elements originally arranged in N rows and M columns of imaging element 3. Furthermore, computing unit 5 reads out fourth images 53a and 53b, captured under an environment illuminated only by background light 10a, from elements 313 and 323. The operation unit 5 first removes the fourth images 53a and 53b read from the corresponding elements from the first image 51 and the second image 52, respectively, to generate images after removing the amount of light received by the background light 10a, that is, to generate a differential first image 51a and a differential second image 52a.

[0127] Here, in addition to removing the fourth images 53a and 53b from the first image 51 and the second image 52, the calculation unit 5 also calculates the brightness information and distance information included in each pixel separately, thereby obtaining the first brightness image 33 and the first distance image 35, and the second brightness image 34 and the second distance image 36.

[0128] Specifically, the calculation unit 5 adds the charge amounts of elements 311 and 321 and the charge amounts of elements 312 and 322, and subtracts twice the charge amounts of elements 313 and 323, thereby obtaining pixel values. By arranging the obtained pixel values ​​in a corresponding array on the imaging element, a first luminance image 33 and a second luminance image 34 are obtained. Furthermore, the calculation unit 5 calculates the charge amounts of elements 311 and 321, the charge amounts of elements 312 and 322, and the charge amounts of elements 313 and 323 using the above-mentioned equation (1), thereby obtaining a first distance image 35 and a second distance image 36.

[0129] Through the processing so far, the calculation unit 5 obtains the above-mentioned four pieces of image information (first luminance image 33 , second luminance image 34 , first distance image 35 , and second distance image 36 ) from the electrical signals accumulated in the imaging element 3 during one frame period.

[0130] The calculation unit 5 further calculates the difference between the first luminance image 33 and the second luminance image 34 to obtain a third image 37. In the calculation of the third image 37, since the difference is calculated for each pixel at the same position in the first luminance image 33 and the second luminance image 34, the value represented by each pixel changes, while the number of pixels in the images before and after the processing does not change, thereby obtaining image information having N / 2 rows and M columns of pixels.

[0131] In the present embodiment, the calculation unit 5 generates the third image 37 by the difference between the first luminance image 33 and the second luminance image 34 , but the third image 37 may be generated by the ratio of the two images.

[0132] The calculation unit 5 then synthesizes the first luminance image 33 and the second luminance image 34 to obtain a luminance image 38. Specifically, the calculation unit 5 inserts the first row of the second luminance image 35 below the first row of the first luminance image 33, and the second row of the second luminance image 34 below the second row of the first luminance image 33. This process is repeated until the N / 2th row, recreating the original arrangement of the corresponding unit elements on the imaging element 3 and synthesizing them. Although the value of each pixel in the image obtained by this process does not change, the number of pixels in the image changes before and after the process, resulting in image information having N rows and M columns of pixels.

[0133] Here, the first luminance image 33 and the second luminance image 34 are images obtained by emitting light of different wavelengths. Due to the difference in reflectivity of the objects in the shooting space between the two wavelengths and the difference in sensitivity of the first element 31 and the second element 32, there may be a difference in the brightness (i.e., grayscale) of the first luminance image 33 and the second luminance image 34 obtained. Therefore, in the synthesis of the above-mentioned luminance image 38, the process of correcting the difference in light intensity is also performed simultaneously. Specifically, a pixel that serves as a reference in the obtained image is determined, and for this pixel that serves as a reference, the ratio of the amount of light received by the image with less light received to the image with more light received is calculated. With respect to the light source that emits light in the shooting of the image with more light received, the brightness is adjusted using the ratio of the amount of light received, so that the amount of light received by the pixel that serves as the reference in the shooting of the next frame is consistent, for synthesis. Alternatively, all pixels in the image with the larger amount of received light may be corrected using the ratio of the amount of received light based on image information, thereby creating a state in which the amount of received light of the pixels is uniform for synthesis.

[0134] The calculation unit 5 then synthesizes the first range image 35 and the second range image 36 to obtain a range image 39. Specifically, the calculation unit 5 inserts the first row of the second range image 36 below the first row of the first range image 35, and the second row of the second range image 36 below the second row of the first range image 35. This process is repeated until the N / 2th row, thereby reproducing the original array of corresponding unit elements on the imaging element 3 and synthesizing them. Although the value of each pixel in the image obtained by this process does not change, the number of pixels in the image changes before and after the process, resulting in image information having N rows and M columns of pixels.

[0135] The numbers of pixels in the vertical direction of the third image 37, luminance image 38, and distance image 39 obtained in the above manner are 2 / N rows, N rows, and N rows, respectively. Therefore, when these images are compared simultaneously, the number of pixels in the vertical direction of the third image 37 is doubled. Alternatively, the number of pixels in the vertical direction of the luminance image 38 and the distance image 39 may be reduced to 1 / 2 times.

[0136] Further, using Figure 11 The processing of the state estimation unit 6 in this embodiment will be described. Figure 6 The steps S104 to S109 in the flowchart of are corresponded. Figure 12 Next, the operation of the living body state detection device 1 when drowsiness information is detected will be described as an example of a warning display.

[0137] Figure 11This is a flowchart for explaining the overall operation of the state estimation unit 6 included in the living body state detection device 1 in the embodiment.

[0138] First, the state estimation unit 6 is as shown in Figure 6 As described in the description of FIG, the detection region determination unit 61 obtains the brightness image 38 and the distance image 39 generated by the calculation unit 5. The detection region determination unit 61 of the state estimation unit 6 performs image processing using the distance image 39 and the brightness image 38 to determine the detection region 62 for limiting the area for pulse detection based on the image information (S201).

[0139] Next, the state estimation unit 6 obtains the third image 37 generated by the calculation unit 5 and processes the third image 37 based on the detection area 62 (S202). More specifically, the state estimation unit 6 deletes image information outside the range of the detection area 62 from the third image 37, and generates a processed third image 63 that uses only the information within the detection area 62.

[0140] The state estimation unit 6 further stores the processed third image 63 in the storage unit 7 and accumulates the processed third image 63 over a plurality of frames through the same process. The accumulated processed third image 63 becomes a moving image 63b of the third image, which shows how image information of the face of person 9 changes over time within the detection area 62. The state estimation unit 6 analyzes the moving image 63b of the third image to estimate the pulse of person 9, which is biological information (S203), and generates an estimated pulse diagram 63c.

[0141] Furthermore, the state estimation unit 6 generates posture information, action information, and information related to movement as living body information of the person 9 by analyzing the distance image 39 stored in the storage unit 7, that is, the motion image 72 of the distance image, which is the change of the distance image 39 over time (S204).

[0142] Furthermore, the state estimation unit 6 generates line of sight information, blink information, face orientation information, and breathing information (S205) as living body information of the person 9 by analyzing the motion image 71 of the brightness image 38 stored in the storage unit 7, which is the change of the brightness image 38 over time.

[0143] Based on the biological information thus obtained, the state estimation unit 6 determines whether the information includes information indicating drowsiness of the person 9 (S206). For example, the state estimation unit 6 comprehensively determines the characteristic pulse that indicates drowsiness, prolonged holding of the same posture, increased blinking, and downward tilt of the face, thereby detecting drowsiness of the person 9. When the state estimation unit detects drowsiness of the person 9, it displays an alert indicating that drowsiness has been detected on the display device 8 (S211).

[0144] Here, Figure 12 This is a diagram for explaining an example of an alarm display of the living body state detection device 1 in the embodiment installed in a vehicle.

[0145] exist Figure 12 , the steering wheel 100, the instrument panel 101, and the living body state detection device 1 are within the field of view of the person 9, and the living body state detection device 1 has started operating.

[0146] The living state detection device 1 emits light of different wavelengths from the light source unit 2 toward the driver, i.e., the person 9, who is the detection target of the living body information. The emitted light 21a, 22a is reflected by the person 9 and reaches the imaging element 3 as reflected light 21b, 22b via the lens 3c. Figure 12 In the example shown, since information on the driver's drowsiness is detected from the captured image, the state estimation unit 6 displays an alarm on the display device 8 to inform that drowsiness has been detected.

[0147] exist Figure 11 In the flowchart, when no drowsiness is detected (No in S206) or after displaying an alarm indicating drowsiness has been detected (S211), the state estimation unit 6 determines whether the acquired biological information includes information indicating that the person 9 is distracted (S207). For example, the state estimation unit 6 comprehensively determines whether the person 9 is distracted by the same action for a long time, the detection of a pulse characteristic of distraction, and a change in the number of eye movements compared to usual times, thereby detecting that the person 9 is distracted. If the state estimation unit detects that the person 9 is distracted, it displays an alarm indicating that distraction has been detected on the display device 8 (S212).

[0148] If inattention is not detected (No in S207), or after displaying an alert indicating inattention has been detected (S212), the state estimation unit 6 determines whether the acquired biological information includes information indicating anxiety in person 9 (S208). For example, the state estimation unit 6 comprehensively determines information such as the pulse rate, a change in the speed of eye movement compared to normal, an increase in the respiratory rate, and other behaviors characteristic of anxiety, thereby detecting anxiety in person 9. If the state estimation unit detects anxiety in person 9, it displays an alert indicating that anxiety has been detected on the display device 8 (S213).

[0149] If no agitation is detected ("No" in S208), or if an alarm indicating agitation is detected is displayed (S213), the state estimation unit 6 determines whether the obtained vital information contains any abnormal values ​​(S209). For example, in addition to abnormal values ​​such as pulse, respiratory rate, and blood pressure, the state estimation unit may also detect one or more abnormal values, such as prolonged eye closure or prolonged lying posture. If the state estimation unit detects an abnormal value in the vital information, it displays an alarm indicating the detection of the abnormal value on the display device 8 (S214).

[0150] When no abnormal value is detected in the biological information (No in S209 ), or after displaying an alarm indicating abnormal value detection ( S214 ), the state estimation unit 6 displays all or part of the biological information currently obtained on the display device 8 ( S210 ).

[0151] Furthermore, when determining an alarm display using this vital information, a comparison with a pre-set threshold value may be used. Alternatively, past vital information of person 9 may be stored as past data in storage unit 7, and the determination may be made by comparing the maximum, minimum, and average values ​​of the past data. Furthermore, in this case, individual identification of person 9 may be performed based on facial features of luminance image 38 (e.g., facial contour, the size of facial organs such as the eyes, nose, and mouth, the relative positions of at least two facial organs, iris color, wrinkles, etc.), and past data may be retrieved based on the identified information. Alternatively, if past data on person 9 is not available, the information used for individual identification of person 9 and vital information may be newly recorded in storage unit 7.

[0152] As described above, the living state detection device 1 in this embodiment includes: a first light source 21 that emits light of a first wavelength; a second light source 22 that emits light of a second wavelength different from the first wavelength; an imaging element 3 composed of a plurality of elements that receives reflected light 21b and 22b emitted from the first light source 21 and the second light source 22 and reflected by the person 9; a control unit 4 that controls the first light source 21 and the second light source 22 so that the first light source 21 and the second light source 22 alternately emit light; and a computing unit 5 that reads a first image 51 from the imaging element 3. and a second image 52, and generates a third image 37 by performing an operation on the read first image 51 and the second image 52, wherein the first image 51 is an image obtained by receiving the reflected light 21b of the light from the first light source 21, and the second image 52 is an image obtained by receiving the reflected light 22b of the light from the second light source 22; and a state estimation unit 6, which generates living body information of the person 9 based on the third image 37 generated in the operation unit 5, and the operation unit 5 generates a distance image 39 based on at least one of the first image 51 and the second image 52.

[0153] With this configuration, the life state detection device 1 of this embodiment can not only estimate the pulse as life information but also obtain information on posture, behavior, and motion based on the distance image 39. More specifically, by analyzing the temporal changes in the position of the person 9 serving as the detection target within the imaging space, as shown in the moving image 72 of the distance image obtained by storing the distance image 39 in the storage unit 7, it is possible to estimate the posture, behavior, and motion. This allows for a life state detection device that utilizes images while generating more life information than ever before.

[0154] Furthermore, the computing unit 5 may generate the distance image 39 by synthesizing the first image 51 and the second image 52. Thus, the first image 51 and the second image 52 obtained for pulse estimation can be used to generate the distance image 39, eliminating the need for re-shooting for generating the distance image 39.

[0155] Alternatively, the computing unit 5 may read the first image 51 and the second image 52 from the imaging element 3, and synthesize the read first image 51 and the read second image 52 so as to reproduce the arrangement positions of the first element 31 corresponding to the first image 51 and the second element 32 corresponding to the second image 52 in the imaging element 3, thereby generating the distance image 39. In this manner, the distance image 39 can be generated as a single image that matches the arrangement of the unit elements on the imaging element 3 based on the elements corresponding to the two images that are divided into two parts for pulse estimation. Therefore, the detection area 62 can be accurately determined using the high-resolution distance image 39.

[0156] Furthermore, the calculation unit 5 may use a Time of Flight method to generate the distance image 39. This allows the distance image 39 to be generated with only one imaging element 3, thereby miniaturizing the overall configuration.

[0157] Furthermore, the state estimation unit 6 may include a detection area determination unit 61 that determines a detection area 62, which is an area in the third image 37 used for generating living body information. The living body information is generated using the detection area 62 determined by the detection area determination unit 61. In this case, for example, if it is determined that skin is obscured from the image sensor 3 by glasses, a mask, a hat, etc., the detection area determination unit 61 may exclude the obscured area and determine the detection area 62. Alternatively, the detection area 62 may be determined by excluding areas such as the eyes or the mouth that are determined to have a different brightness from the skin. This allows more appropriate pixels to be selected from the image for generating living body information, thereby improving the accuracy of the generated living body information.

[0158] Furthermore, the detection area determination unit 61 may determine the detection area 62 using the distance image 39. This allows the part of the person 9 to be determined as the detection area 62 to be selected based on the distance, thereby enabling efficient determination of the detection area.

[0159] Furthermore, the computing unit 5 may further generate a luminance image 38 based on at least one of the first image 51 and the second image 52, and the detection area determination unit 61 may determine the detection area 62 using the distance image 39 and the luminance image 38. Thus, information on gaze, blinking, facial orientation, and breathing can be obtained based on the luminance image 38. Furthermore, since the determination of the detection area 62 can be based not only on distance but also on luminance, the accuracy of the generation of the detection area 62 can be improved.

[0160] Furthermore, the calculation unit 5 may generate the luminance image 38 by synthesizing the first image 51 and the second image 52. In this way, the first image 51 and the second image 52 obtained for pulse estimation can be reused to generate the luminance image 38, eliminating the need for re-shooting to generate the luminance image 38.

[0161] Alternatively, the computing unit 5 may read the first image 51 and the second image 52 from the imaging element 3, and synthesize the read first image 51 and the read second image 52 so as to reproduce the arrangement positions of the first element 31 corresponding to the first image 51 and the second element 32 corresponding to the second image 52 in the imaging element 3, thereby generating a luminance image 38. In this manner, since the luminance image 38 can be generated as a single image that matches the arrangement of the unit elements on the imaging element 3 based on the elements corresponding to the two images divided into two parts for pulse estimation, the detection area 62 can be accurately determined using the high-resolution luminance image 38.

[0162] Furthermore, the computing unit 5 may combine the first image 51 and the second image 52 after adjusting the brightness of at least one of the first image 51 and the second image 52. This allows correction of the difference in reflectivity of objects within the imaging space between the first image 51 and the second image 52, and also allows correction of the difference in received light amount caused by the difference in sensitivity between the first element 31 and the second element 32.

[0163] Furthermore, the first image 51 and the second image 52 used for synthesis may be images read by the computing unit 5 after the control unit 4 adjusts the light intensity of at least one of the first light source 21 and the second light source 22. This can reduce the power required for the emitted light 21a and 22a from the light sources while correcting the difference in the light intensity received by the first element 31 and the second element 32.

[0164] Furthermore, the control unit 4 may further control the imaging element 3 so as to cause the imaging element 3 to capture images when no light is emitted from the first light source 21 and the second light source 22. The calculation unit 5 may further read out fourth images 53a and 53b captured when no light is emitted from the first light source 21 and the second light source 22 from the imaging element 3, subtract the readout fourth images 53a and 53b from the first image 51 and the second image 52, respectively, to obtain a differential first image 51a and a differential second image 52a, and perform calculations on the differential first image 51a and the differential second image 52a to generate the third image 37. In this manner, the influence of the background light 10a can be eliminated, and an image obtained by capturing only the reflected light 21b and 22b from the light source unit 2 can be generated.

[0165] Furthermore, the computing unit 5 may generate the third image 37 by calculating the difference or ratio between the first image 51 and the second image 52, and the state estimation unit 6 may generate biological information related to the pulse of the person 9 based on the third image 37. This allows comparison of image information captured at two different wavelengths, thereby reducing the effects of body movement and changes in external light, and accurately obtaining information reflecting the blood flow rate at the time of capture. Consequently, more accurate biological information related to the pulse can be generated, as well as biological information related to respiration and blood pressure.

[0166] Furthermore, the first and second wavelengths can be near-infrared wavelengths. This allows the selection of wavelengths with a low concentration of light, such as sunlight, as background light 10a, thereby reducing the impact of background light source 10. Furthermore, near-infrared light, a wavelength with high hemoglobin reflectivity, can be selected for use. Furthermore, since near-infrared light is not visible light, it does not illuminate objects or people within the imaging space, effectively preventing them from working and enabling the detection of the worker's vital signs.

[0167] Furthermore, the state estimation unit 6 may further estimate the state of the person 9 using the biological information and the distance image 39. This allows the distance between various parts of the body of the person 9 to be flexibly used to evaluate body movements, postures, and the like.

[0168] Furthermore, a lens may be provided for focusing reflected light 21b and 22b emitted from the first light source 21 and the second light source 22 and reflected by the person onto the imaging element 3. The lens may have characteristics such that, when the first image 51 and the second image 52 correspond to the upper body of the person 9, the magnification of the reflected light 21b and 22b projected onto the area of ​​the imaging element 3 corresponding to the face of the person 9 is greater than the magnification of the reflected light 21b and 22b projected onto the area of ​​the imaging element 3 corresponding to the face of the person 9 outside the face of the person 9. This further improves the accuracy of detecting subtle facial changes in biological information obtained by observing these changes. More specifically, this can be effectively applied to evaluations that require detecting subtle facial movements, such as gaze and blinking.

[0169] Furthermore, the living state detection method in this embodiment includes: a control step S101, controlling a first light source that emits light of a first wavelength and a second light source that emits light of a second wavelength different from the first wavelength so that the first light source and the second light source alternately emit light; an operation step S103, reading a first image and a second image from an imaging element composed of a plurality of elements, and generating a third image by operating on the read first image and second image, wherein the plurality of elements receive reflected light emitted from the first light source and the second light source and reflected by the person, the first image being obtained by receiving reflected light from the first light source, and the second image being obtained by receiving reflected light from the second light source; and a state estimation step S104, generating living body information of the person based on the third image generated in the operation step S103, wherein in the operation step S103, a distance image is further generated based on at least one of the first image and the second image.

[0170] With this configuration, the living body status detection method of this embodiment can not only estimate the pulse as living body information, but also obtain information on posture, behavior, and motion from the distance image 39. More specifically, by analyzing the temporal changes in the position of the person 9, the detection target, within the imaging space, as shown in the moving image 72 of the distance image obtained by storing the distance image 39 in the storage unit 7, it is possible to estimate the posture, behavior, and motion. This allows the construction of a living body status detection method that utilizes images while generating more living body information than ever before.

[0171] The above description of the living body state detection device 1 and the living body state detection method of the present invention is based on the embodiments. However, the present invention is not limited to the embodiments. Within the scope of the present invention, various modifications that can be imagined by those skilled in the art are applied to the present embodiment, as well as other embodiments obtained by arbitrarily combining some of the components of the embodiment and its modifications are all included within the scope of the present invention.

[0172] For example, in this embodiment, the first range image 35 and the second range image 36 are synthesized to generate the range image 39. However, it is also possible to use only one of the first range image 35 or the second range image 36. In this case, the first range image 35 and the second range image 36 can be compared to determine which one was captured with less noise, and the one with less noise can be used. Alternatively, it is possible to predetermine which image to use and always use the predetermined image.

[0173] Furthermore, although in this embodiment the first image 51 and the second image 52 are synthesized by reproducing the configuration positions on the imaging element 3 to generate the distance image 39 and the brightness image 38, for example, the synthesis may be performed by obtaining the average value of the positions where the first image 51 and the second image 52 overlap, thereby generating the distance image 39 and the brightness image 38.

[0174] Furthermore, although the distance image 39 is generated using the TOF method in this embodiment, the distance image 39 may be generated using, for example, a stereo ranging method or a structured light method.

[0175] Furthermore, in the present embodiment, the detection area 62 is determined using the distance image 39 or the distance image 39 and the luminance image 38 . However, for example, the detection area 62 may be determined using only the luminance image 38 .

[0176] Furthermore, in this embodiment, although the luminance image 38 is generated by synthesizing the first luminance image 33 and the second luminance image 34, for example, only one of the first luminance image 33 or the second luminance image 34 may be used. In this case, the first luminance image 33 or the second luminance image 34 can be compared to determine which one was captured with less noise, and the one with less noise can be used. Alternatively, the image to be used can be determined in advance, and the determined image can be used constantly.

[0177] Furthermore, while the configuration in this embodiment is such that the first element 31 is exposed when light is emitted from the first light source 21, and the second element 32 is exposed when light is emitted from the second light source 22, it is also possible to expose each element simultaneously, each equipped with a filter that transmits only light of the corresponding wavelength. In this case, the first and second light sources 21 and 22 emit light simultaneously. Since the reflected light 21b from the first light source 21 can pass through the filter of the first element 31 but not the filter of the second element 32, only the amount of light of the first wavelength is recorded in the first element 31. Furthermore, since the reflected light 22b from the second light source 22 can pass through the filter of the second element 32 but not the filter of the first element 31, only the amount of light of the second wavelength is recorded in the second element 32.

[0178] Furthermore, the main body of the apparatus, system, or method disclosed herein is provided with a computer. The computer executes a program to realize the functions of the main body of the apparatus, system, or method disclosed herein. The computer is provided with a processor that operates according to the program as the main hardware component. As long as it can execute the program to realize the function, the processor can be any. The processor is composed of an electronic circuit including one or more semiconductor integrated circuits (ICs) or LSIs (large-scale integration). Although referred to herein as IC or LSI, it may also be referred to as system LSI, VLSI (very large scale integration), or ULSI (ultra large scale integration) depending on the degree of integration. In addition, a field programmable gate array (FPGA) that can be programmed after the LSI is manufactured may be used, or a reconfigurable logic device in which, for example, the connection relationship inside the LSI can be reconfigured or the circuit area inside the LSI can be set may be used. Multiple electronic circuits may be integrated on one chip or may be provided on multiple chips. Multiple chips may be installed in one device or may be provided by multiple devices. The program is recorded on a non-temporary recording medium such as a computer-readable ROM, optical disc, or hard disk. The program may be stored in advance on a recording medium, or may be provided to the recording medium via a wide area communication network including the Internet.

[0179] Explanation of symbols

[0180] 1. Living state detection device

[0181] 3 Camera components

[0182] 4 Control Unit

[0183] 5. Operation unit

[0184] 6 State Estimation Unit

[0185] 9 people

[0186] 37 Image 3

[0187] 38 Luminance Image

[0188] 39 Distance Image

[0189] 51 Image 1

[0190] 52 Image 2

[0191] 53a, 53b Image 4

[0192] 61 Detection area determination unit

[0193] 62 detection areas

[0194] 63b Moving image of the third image

[0195] 71 Luminance Image Motion Image

[0196] 72 Motion images of distance images

Claims

1. A living body state detection device for detecting the living body state of a person. The living state detection device comprises: A first light source emits light of a first wavelength; a second light source emitting light of a second wavelength different from the first wavelength; An imaging element, composed of a plurality of elements, which receives reflected light emitted from the first light source and the second light source and reflected from the person; a control unit that controls the first light source and the second light source so that the first light source and the second light source emit light alternately; a calculation unit that reads out a first image and a second image from the imaging element, generates a third image by calculating the read out first image and second image, and further generates a distance image based on at least one of the first image and the second image, wherein the first image is obtained by receiving reflected light from the light from the first light source and the second image is obtained by receiving reflected light from the light from the second light source; as well as The state estimation unit generates and outputs biological information indicating the biological state of the person based on the third image and the distance image generated by the calculation unit.

2. The living state detection device according to claim 1, The calculation unit generates the distance image by synthesizing the first image and the second image.

3. The living state detection device according to claim 2, The operation unit, reading the first image and the second image from the imaging element, The read first image and the second image are synthesized so that the arrangement positions of the first element corresponding to the first image and the second element corresponding to the second image in the imaging element can be reproduced, thereby generating the distance image.

4. The living state detection device according to any one of claims 1 to 3, The computing unit generates the distance image using a Time of Flight method.

5. The living state detection device according to any one of claims 1 to 3, The state estimation unit includes a detection region determination unit that determines a detection region, and generates the living body information by using the detection region determined by the detection region determination unit, wherein the detection region is a region in the third image for generating the living body information.

6. The living state detection device according to claim 5, The detection area determination unit determines the detection area using the distance image.

7. The living state detection device according to claim 6, The computing unit further generates a brightness image based on at least one of the first image and the second image. The detection area determination unit determines the detection area using the distance image and the brightness image.

8. The living state detection device according to claim 7, The calculation unit generates the luminance image by synthesizing the first image and the second image.

9. The living state detection device according to claim 8, The operation unit, reading the first image and the second image from the imaging element, The luminance image is generated by synthesizing the read first image and the second image so that the arrangement positions of the first element corresponding to the first image and the second element corresponding to the second image in the imaging element can be reproduced.

10. The living state detection device according to claim 8 or 9, The calculation unit combines the first image and the second image after adjusting the brightness of at least one of the first image and the second image.

11. The living state detection device according to claim 8 or 9, The first image and the second image used for the synthesis are images read out by the calculation unit after the control unit adjusts the light intensity of at least one of the first light source and the second light source.

12. The living body state detection device according to any one of claims 1 to 3, The control unit further controls the imaging element so that the imaging element performs imaging when no light is emitted from the first light source and the second light source. The operation unit further reads out the fourth image captured when the first light source and the second light source do not emit light from the imaging element, subtracts the read fourth image from the first image and the second image respectively, thereby obtaining a differential first image and a differential second image, and generates the third image by performing operations on the differential first image and the differential second image.

13. The living body state detection device according to any one of claims 1 to 3, The calculation unit generates the third image by calculating a difference or a ratio between the first image and the second image. The state estimation unit generates biological information related to the pulse of the person based on the third image.

14. The living state detection device according to any one of claims 1 to 3, The first wavelength and the second wavelength are wavelengths of near-infrared light.

15. The living body state detection device according to any one of claims 1 to 3, The state estimation unit further estimates the state of the person using the living body information and the distance image.

16. The living body state detection device according to any one of claims 1 to 3, The living body state detection device further includes a lens for focusing reflected light emitted from the first light source and the second light source on the person, onto the imaging element. The characteristic of the lens is that, when the first image and the second image are images corresponding to the upper body of the person, the magnification of the reflected light projected on the area of ​​the imaging element corresponding to the face of the person is greater than the magnification of the reflected light projected on the area of ​​the imaging element corresponding to other than the face of the person.

17. A method for detecting the living state of a person, The living state detection method comprises: A control step of controlling a first light source that emits light of a first wavelength and a second light source that emits light of a second wavelength different from the first wavelength so that the first light source and the second light source alternately emit light; a calculation step of reading out a first image and a second image from an imaging element composed of a plurality of elements, generating a third image by calculating the read out first image and second image, and further generating a distance image based on at least one of the first image and the second image, wherein the plurality of elements receive reflected light of the light emitted from the first light source and the second light source and reflected from the person, the first image being obtained by receiving the reflected light of the light from the first light source, and the second image being obtained by receiving the reflected light of the light from the second light source; as well as The state estimation step generates biological information indicating the biological state of the person based on the third image and the distance image generated in the calculation step.

Images