Image pickup system, image pickup method, and computer program product
By designing a non-contact camera device, using pulse light sources and image sensors to separate the surface reflection and internal scattering components in the internal light signals of the organism, the burden and noise interference problems caused by close contact of the photo detector are solved, and stable and high-precision Internal determination of organisms.
Patent Information
- Application Number
- CN202411048183.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2016-12-15
- Filing Date
- 2017-09-26
- Publication Date
- 2024-11-19
AI Technical Summary
In the existing internal measurement methods of biological organisms, the photodetector is closely connected to the measurement site, which causes a high psychological and physical burden on the subject, and is difficult to use for a long time, and there is noise interference caused by the surface reflective components, which affects the measurement accuracy.
A non-contact imaging device is designed to detect light signals inside organisms using pulse light sources and image sensors, adjust the luminous timing and shutter timing of pulse light through control circuits, separate surface reflection components and internal scattering components, reduce noise and improve Determination accuracy.
It is possible to stably measure internal information of the biological body without contacting the object, reduce noise interference, reduce measurement errors, and avoid error factors caused by non-contact measurement, thereby improving measurement accuracy and reliability.
Smart Images

Figure CN118984426A_ABST
Abstract
Description
[0001] This application is a divisional of an invention patent application with an application date of September 26, 2017, application number 201710880052.6, and invention name “Camera device and camera method”. Technical Field
[0002] The present application relates to an imaging device and an imaging method for measuring the interior of an object. Background Art
[0003] In the field of biological measurement, there is a method of obtaining internal information of an object by irradiating light to the object and based on the information of the light that has penetrated the interior of the object. In this method, there is a case where the surface reflection component, which is a reflection component from the surface of the object, becomes noise. As a method of removing the noise caused by these surface reflection components and obtaining only the desired internal information, there is a method disclosed in Japanese Patent Publication No. 11-164826 in the field of biological measurement. Japanese Patent Publication No. 11-164826 discloses a method of performing measurement by bringing a light source and a light detector into close contact with a measurement site in a state where they are separated by a certain distance. Summary of the invention
[0004] A camera system according to a technical solution of the present application comprises: a light source for emitting pulsed light for irradiating a biological body; a light detector for detecting light returning from the biological body in response to the irradiation of the pulsed light and outputting a signal corresponding to the cerebral blood flow of the biological body; and a calculation unit for determining whether the temporal change of the cerebral blood flow of the biological body is within a certain range based on the signal from the light detector. BRIEF DESCRIPTION OF THE DRAWINGS
[0005] Figure 1A This is a schematic diagram showing the imaging device of Embodiment 1 and a state in which the imaging device captures an object.
[0006] Figure 1B This is a diagram showing an example of the structure of an image sensor.
[0007] Figure 1C This is a flowchart schematically showing the operation of the control circuit.
[0008] Figure 2 The diagram shows the waveform of the surface reflection component, the waveform of the internal scattering component, the waveform of the sum of the surface reflection component and the internal scattering component, and the timing of OPEN and CLOSE of the electronic shutter.
[0009] Figure 3 This is a flowchart showing the operation of the imaging device according to the first embodiment before the actual measurement.
[0010] Figure 4A This is a diagram showing an example of determination by the measurement environment determination unit.
[0011] Figure 4B This is a diagram showing an example of determination by the measurement environment determination unit.
[0012] Figure 4C This is a diagram showing an example of determination by the measurement environment determination unit.
[0013] Figure 4D This is a diagram showing an example of determination by the measurement environment determination unit.
[0014] Figure 5A This is a diagram showing an example of a display that displays a captured image obtained by an imaging device and a detection area of an object.
[0015] Figure 5B This is a diagram showing an example of a display that displays a captured image obtained by an imaging device and a detection area of an object.
[0016] Figure 5C This is a diagram showing the detection area after the size and position have been adjusted.
[0017] Figure 5D This is a diagram showing the detection area maximized by the area maximization function.
[0018] Figure 5E It is a diagram showing a plurality of detection areas on a captured image.
[0019] Fig. 6A This is a diagram showing an example of an error message output to a display when the measurement environment determination unit determines that the detection area is incorrect.
[0020] Figure 6B A diagram showing auxiliary lines displayed on a display to facilitate adjustment of a detection area for an object.
[0021] Figure 6C This is a diagram showing an adjustment stage for adjusting the detection area by adjusting the orientation and position of the imaging device.
[0022] Fig.6D It is a diagram showing a fixing jig for fixing an object.
[0023] Fig. 7A This is a diagram showing a situation where light amount adjustment is required.
[0024] Figure 7B This is a diagram showing a situation where light amount adjustment is required.
[0025] Figure 7CThis is a diagram showing the relationship between a plurality of light emission pulses in one frame, a light signal on the sensor, a plurality of shutter timings, and a charge accumulation timing.
[0026] Fig. 8A FIG. 1 is a diagram showing an example of determination in a signal stability determination unit.
[0027] Figure 8B FIG. 1 is a diagram showing an example of determination in a signal stability determination unit.
[0028] Fig. 9 This is a diagram showing an example of an error message output to a display when the signal stability determination unit determines that the signal is unstable.
[0029] Fig. 10A It is a schematic diagram showing the imaging device of Embodiment 2 and the state of an object photographed by the imaging device.
[0030] Fig. 10B This is a flowchart showing the operation of the imaging device according to the second embodiment during the actual measurement.
[0031] Fig.11A 2 is a diagram showing an example of determination in the abnormal value determination unit.
[0032] Fig. 11B 2 is a diagram showing an example of determination in the abnormal value determination unit.
[0033] Fig. 12A This is a diagram showing an example of an error message output to a display when the abnormal value determination unit determines that an abnormal value has occurred.
[0034] Fig. 12B This is a diagram showing an example of an error message output by a display when the abnormal value determination unit determines that an abnormal value has occurred.
[0035] Description of symbols
[0036] 100, 800 camera device
[0037] 101 Object
[0038] 102 Light Source
[0039] 104 Photoelectric conversion unit
[0040] 106 Charge storage unit
[0041] 110 Image Sensor
[0042] 120 Control circuit
[0043] 130 Light emission amount adjustment unit
[0044] 140 Measurement Environment Judgment Department
[0045] 150 Signal stability determination unit
[0046] 400 Detection Area
[0047] 500 Display
[0048] 510 Capture image
[0049] 520 Position adjustment icon
[0050] 530 Auxiliary Lines
[0051] 540 Adjustment Table
[0052] 550 Fixing fixture
[0053] 710 Floodlight
[0054] 810 Abnormal Value Judgment Unit DETAILED DESCRIPTION
[0055] However, in the method described in Japanese Patent Application Laid-Open No. 11-164826, since the photodetector is brought into close contact with the measurement site, the psychological and physical burden on the subject is large, so it takes time to wear the device and it is difficult to use for a long time.
[0056] This application includes, for example, the technical solutions described in the following items.
[0057] [Project 1]
[0058] The imaging device according to item 1 of the present application is used for measuring the interior of an object, and comprises: a light source for emitting pulse light for irradiating the object; a light detector for detecting light returned from the object in response to the irradiation of the pulse light; and a computing unit. The computing unit determines the temporal stability of the amount of light returned from the object.
[0059] [Project 2]
[0060] In the imaging device described in Item 1, the light detector may be an image sensor that converts the light returned from the object into signal charges and accumulates the light; and the calculation unit may determine the temporal stability of the amount of light returned from the object by determining the temporal stability of the amount of accumulation of the signal charges in the image sensor.
[0061] [Item 3]
[0062] In the imaging device described in item 1 or 2, the computing unit may further determine whether the environment of the object is suitable for measuring the inside of the object, and adjust the light intensity of the pulse light.
[0063] [Item 4]
[0064] In the imaging device described in Item 3, the calculation unit may adjust the light intensity of the pulse light by adjusting the number of times the pulse light emits light per unit time.
[0065] [Item 5]
[0066] In the imaging device described in Item 2, the image sensor may acquire a first image of the object based on the signal charge; and the calculation unit may further determine a position of a region in the first image for measuring the interior of the object.
[0067] [Item 6]
[0068] In the imaging device described in Item 5, the object may be a living body; the region may be within a specific part of the living body; and the calculation unit may further adjust the size of the region so that the region is maximized within the specific part.
[0069] [Item 7]
[0070] The imaging device according to item 5 or 6 may further include a display, wherein the display displays the first image and the second image representing the area in an overlapping manner.
[0071] [Item 8]
[0072] In the imaging device described in Item 7, the display may further display auxiliary lines for determining the position of the region superimposed on the first image and the second image.
[0073] [Item 9]
[0074] In the imaging device described in any one of items 1 to 8, the computing unit may further determine abnormality in measurement of the interior of the object.
[0075] [Item 10]
[0076] In the imaging device described in Item 2, the image sensor may accumulate the signal charge corresponding to a component of the light returning from the object that is scattered inside the object.
[0077] [Item 11]
[0078] In the imaging device described in any one of items 1 to 10, the object may be a living body; and the computing unit may generate information indicating changes in blood flow of the living body based on the signal from the photodetector.
[0079] [Item 12]
[0080] The imaging method of Item 12 of the present application is used for measuring the interior of an object, comprising: a step of irradiating the object with pulsed light, detecting light returned from the object in response to the irradiation with the pulsed light by a light detector; and a step of determining the temporal stability of the amount of light returned from the object.
[0081] [Item 13]
[0082] In the imaging method described in Item 12, the light detector may be an image sensor that converts the light returned from the object into signal charge and accumulates the signal charge; and in the step of performing the determination, the temporal stability of the amount of light returned from the object is determined by determining the temporal stability of the amount of accumulation of the signal charge in the image sensor.
[0083] [Item 14]
[0084] The imaging method according to item 12 or 13 may further include: a step of determining whether the environment of the object is suitable for measuring the interior of the object; and a step of adjusting the light intensity of the pulse light.
[0085] [Item 15]
[0086] In the imaging method described in any one of items 12 to 14, the object may be a living body, and the method may further include: generating information indicating blood flow changes in the living body based on the signal from the photodetector.
[0087] In the present application, all or part of a circuit, unit, device, component or part, or all or part of a functional block of a block diagram may also be performed by one or more electronic circuits including a semiconductor device, a semiconductor integrated circuit (IC), or an LSI (large scale integration). LSI or IC may be integrated on one chip or may be composed of a combination of multiple chips. For example, functional blocks other than storage elements may be integrated on one chip. It is referred to as LSI or IC here, but the name may change depending on the degree of integration, and may also be referred to as system LSI, VLSI (very large scale integration) or ULSI (ultra large scale integration). Field Programmable Gate Array (FPGA) programmed after the manufacture of LSI, or a reconfigurable logic device capable of reconfiguring the bonding relationship within LSI or setting the circuit division within LSI can also be used for the same purpose.
[0088] Furthermore, all or part of the functions or operations of a circuit, unit, device, component or part can be executed by software processing. In this case, the software is recorded in one or more non-transitory recording media such as ROM, optical disk, hard disk drive, etc. When the software is executed by a processor, the functions determined by the software are executed by the processor and peripheral devices. The system or device may also have one or more non-transitory recording media with software recorded, a processor, and required hardware devices, such as interfaces.
[0089] According to a technical solution of the present application, the internal information of the object can be measured without contact with the object and in a state where the noise caused by the reflection component from the surface of the object is suppressed. In addition, according to a technical solution of the present application, the error factor caused by non-contact measurement can be eliminated and the object can be measured stably.
[0090] The embodiments described below all represent general or specific examples. The numerical values, shapes, materials, constituent elements, configurations of constituent elements and connection forms, steps, and the order of steps shown in the following embodiments are examples and are not intended to limit the present application. In addition, constituent elements of the following embodiments that are not described in the independent claims representing the highest concept are described as arbitrary constituent elements.
[0091] Hereinafter, the embodiments will be described in detail with reference to the drawings.
[0092] (Implementation Method 1)
[0093] [1. Camera device]
[0094] First, use Figures 1A to 3 The structure of the imaging device 100 according to the first embodiment will be described.
[0095] Figure 1A 1 is a schematic diagram showing an imaging device 100 according to the present embodiment. The imaging device 100 includes a light source 102, an image sensor 110 including a photoelectric conversion unit 104 and a charge storage unit 106, a control circuit 120, a light emission amount adjustment unit 130, a measurement environment determination unit 140, and a signal stability determination unit 150. The image sensor 110 corresponds to a photodetector. The light emission amount adjustment unit 130, the measurement environment determination unit 140, and the signal stability determination unit 150 correspond to a computing unit.
[0096] [1-1. Light source 102]
[0097] The light source 102 irradiates light to the object 101. The light irradiated from the light source 102 and reaching the object 101 is a surface reflection component I1, which is a component reflected by the surface of the object 101, and an internal scattering component I2, which is a component reflected once or scattered, or multiple scattered, inside the object 101. The surface reflection component I1 includes three types: a direct reflection component, a diffuse reflection component, and a scattered reflection component. The direct reflection component is a reflection component whose incident angle is equal to the reflection angle. The diffuse reflection component is a component reflected by diffusion due to the concave-convex shape of the surface. The scattered reflection component is a component reflected by scattering from the internal tissue near the surface. In the case where the object 101 is set as a forehead of a person, the scattered reflection component is a component scattered and reflected inside the epidermis. In the following, in this application, it is assumed that the surface reflection component I1 of the object 101 includes these three components for description. In addition, it is assumed that the internal scattering component I2 does not include the component scattered and reflected by the internal tissue near the surface for description.
[0098] The traveling directions of the surface reflection component I1 and the internal scattering component I2 change due to reflection or scattering, and a part of them reaches the image sensor 110. The light source 102 generates pulse light multiple times at a specified time interval or timing. The fall time of the pulse light generated by the light source 102 can also be close to zero, for example, a rectangular wave. Generally, if the extension of the rear end of the internal scattering component I2 of the object 101 is considered to be 4ns, the fall time can be less than 2ns, which is less than half of it, or less than 1ns. The rise time of the pulse light generated by the light source 102 is arbitrary. This is because, in the measurement using the camera device of the present application described later, the falling part of the time axis of the pulse light is used, and the rising part is not used. The light source 102 is, for example, a laser such as LD in which the falling part of the pulse light is close to a right angle with respect to the time axis and the time response characteristic is fast.
[0099] The wavelength of the pulse light emitted from the light source 102 can be set to, for example, about 650 nm or more and about 950 nm or less. This wavelength range is included in the wavelength range from red to near infrared. This wavelength band is a wavelength band in which light easily penetrates into the object 101. In this specification, the term "light" is used not only for visible light but also for infrared light.
[0100] Since the imaging device 100 of the present application measures the object 101 in a non-contact manner, when the object 101 is a person, the impact on the retina is taken into consideration. Therefore, it is also possible to meet the level 1 of the laser safety standard that each country has. In this case, low-illuminance light with a radiation emission limit AEL of less than 1 mW is irradiated to the object 101. However, the light source 102 itself may not meet the level 1. For example, a diffuser or an ND filter may be arranged in front of the light source 102, and the light may be diffused or attenuated to meet the level 1 of the laser safety standard.
[0101] The conventional ultrafast scanning camera described in the Japanese Patent Publication No. 4-189349 is used to detect information (e.g., absorption coefficient and scattering coefficient) at different locations in the depth direction of the biological body. Therefore, in order to measure with the desired spatial resolution, an ultrashort pulse light with a pulse width of femtoseconds or picoseconds is used. On the other hand, the imaging device 100 of the present application is used to detect the internal scattered component I2 from the surface reflected component I1.
[0102] Therefore, the pulse light emitted by the light source 102 does not need to be an ultra-short pulse light, and the pulse width is arbitrary. When irradiating light to the head in order to measure cerebral blood flow, the amount of light of the internal scattered component I2 is several thousand to tens of thousands of times smaller than the amount of light of the surface reflected component I1, which is very small. Furthermore, if the laser safety standard is taken into consideration, the amount of light that can be irradiated is small, and the detection of the internal scattered component I2 becomes difficult. Therefore, by making the light source 102 generate pulse light with a relatively large pulse width, the cumulative amount of the internal scattered component accompanied by the time delay can be increased, and the SN ratio can be improved by increasing the detection light amount.
[0103] The light source 102 emits pulse light with a pulse width of 3 ns or more, for example. Alternatively, the light source 102 may emit pulse light with a pulse width of 5 ns or more, or even more than 10 ns. On the other hand, if the pulse width is too large, unused light increases and becomes a waste, so the light source 102 generates pulse light with a pulse width of 50 ns or less, for example. Alternatively, the light source 102 may emit pulse light with a pulse width of 30 ns or less, or even less than 20 ns.
[0104] In addition, the illumination pattern of the light source 102 may be a uniform intensity distribution in the illumination area. The method disclosed in the Japanese Patent Gazette No. 11-164826 etc. can only perform discrete light illumination because the detector and the light source are separated by 3 cm, thereby reducing the surface reflection component I1 in space. On the other hand, the imaging device D of the present application is a method of separating and reducing the surface reflection component I1 in time, so the internal scattering component I2 can be detected even directly below the illumination point on the object 101. In order to improve the measurement resolution, the object 101 may be illuminated spatially over the entire surface.
[0105] [1-2. Image sensor 110]
[0106] The image sensor 110 receives light emitted from the light source 102 and reflected from the object 101. The image sensor 110 has a plurality of pixels arranged two-dimensionally, and acquires two-dimensional information of the object 101. The image sensor 110 is, for example, a CCD image sensor or a CMOS image sensor.
[0107] The image sensor 110 has an electronic shutter. The electronic shutter is a circuit that controls the shutter width and shutter timing. The shutter width is the duration of one signal accumulation, i.e., the length of the exposure period, during which the received light is converted into an effective electrical signal and accumulated. The shutter timing is the time from the end of one exposure period to the beginning of the next exposure period. In the following, the state in which the electronic shutter is exposing is referred to as "OPEN (open state)", and the state in which the electronic shutter stops exposing is referred to as "CLOSE (closed state)".
[0108] The image sensor 110 can adjust the shutter timing to sub-nanoseconds, for example, 30ps to 1ns, by means of an electronic shutter. In order to correct the influence of the brightness of the subject, conventional TOF cameras for the purpose of distance measurement detect all the light reflected from the subject by the pulse light emitted by the light source 102. Therefore, in conventional TOF cameras, the shutter width needs to be larger than the pulse width of the light. In contrast, since the imaging device 100 of the present embodiment does not need to correct the amount of light of the subject, the shutter width does not need to be larger than the pulse width, for example, about 1 to 30ns. According to the imaging device 100 of the present embodiment, since the shutter width can be reduced, the dark current contained in the detection signal can be reduced.
[0109] When the object 101 is a forehead of a person and information such as cerebral blood flow is detected, the attenuation rate of the light inside is very large, about 1 in 1 million. Therefore, in order to detect the internal scattered component I2, there is a case where the amount of light is insufficient when irradiating with only one pulse. When irradiating at level 1 of the laser safety standard, the amount of light is particularly weak. In this case, the light source 102 emits pulse light multiple times, and the image sensor 110 is also exposed multiple times through the electronic shutter accordingly, so as to accumulate the detection signal and improve the sensitivity.
[0110] Hereinafter, a configuration example of the image sensor 110 will be described.
[0111] The image sensor 110 has a plurality of pixels as light detection units arranged two-dimensionally on an imaging surface. Each pixel has a light receiving element (eg, a photodiode).
[0112] Figure 1B 1 is a diagram showing an example of the structure of the image sensor 110. Figure 1B In FIG. 1 , the area surrounded by the double-dashed line frame corresponds to one pixel 201. The pixel 201 includes one photodiode. Figure 1B2 shows only four pixels arranged in two rows and two columns, but more pixels are actually configured. Pixel 201 includes a photodiode, a source follower transistor 309, a row selection transistor 308, and a reset transistor 310. Each transistor is, for example, a field effect transistor formed on a semiconductor substrate, but is not limited thereto.
[0113] As shown in the figure, one of the input terminal and the output terminal (typically the source) of the source follower transistor 309 is connected to one of the input terminal and the output terminal (typically the drain) of the row selection transistor 308. The gate as the control terminal of the source follower transistor 309 is connected to the photodiode. The signal charge (holes or electrons) generated by the photodiode is accumulated in the floating diffusion layers 204, 205, 206, 207 of the charge accumulation section, which is the charge accumulation node between the photodiode and the source follower transistor 309.
[0114] Although in Figure 1B Although not shown, a switch may be provided between the photodiode and the floating diffusion layer 204, 205, 206, 207. The switch switches the conduction state between the photodiode and the floating diffusion layer 204, 205, 206, 207 according to a control signal from the control circuit 120. Thus, the start and stop of the accumulation of signal charges in the floating diffusion layer 204, 205, 206, 207 are controlled. The electronic shutter of this embodiment has such a mechanism for exposure control.
[0115] By turning on the gate of the row selection transistor 308 by the row selection circuit 302, the signal charge accumulated in the floating diffusion layers 204, 205, 206, and 207 is read out. At this time, the current flowing from the source follower power supply 305 to the source follower transistor 309 and the source follower load 306 is amplified according to the signal potential of the floating diffusion layers 204, 205, 206, and 207. The analog signal formed by the current read out from the vertical signal line 304 is converted into digital signal data by the analog-to-digital (AD) conversion circuit 307 connected to each column. The digital signal data is read out by each column by the column selection circuit 303 and output from the image sensor 110. After reading out one row, the row selection circuit 302 and the column selection circuit 303 read out the next row, and the information of the signal charge of the floating diffusion layers of all rows is read out in the same manner. After reading out all the signal charges, the control circuit 120 turns on the gate of the reset transistor 310 to reset all the floating diffusion layers. Thus, the imaging of one frame is completed. Similarly, by repeatedly performing high-speed imaging of frames, the imaging of a series of frames by the image sensor 110 is completed.
[0116] In the present embodiment, an example of a CMOS type image sensor 110 is described, but the image sensor 110 may be a CCD type, a single photon counting type element, or an amplification type image sensor (EMCCD, ICCD).
[0117] [1-3. Control circuit 120]
[0118] The control circuit 120 adjusts the time difference between the emission timing of the pulse light of the light source 102 and the shutter timing of the image sensor 110. Hereinafter, this time difference may be referred to as "phase" or "phase delay". The "emission timing" of the light source 102 is the time when the pulse light emitted by the light source 102 starts to rise. The control circuit 120 can adjust the phase by changing the emission timing or by changing the shutter timing.
[0119] The control circuit 120 may also be configured to remove an offset component from a signal detected by a light receiving element of the image sensor 110. The offset component is a signal component formed by ambient light and disturbance light such as sunlight and fluorescent light. When the light source 102 is not emitting light, that is, when the drive of the light source 102 is turned off, the image sensor 110 detects a signal to estimate the offset component caused by the ambient light and disturbance light.
[0120] The control circuit 120 may be, for example, an integrated circuit having a processor such as a central processing unit (CPU) or a microcomputer and a memory. The control circuit 120 adjusts the light emission timing and shutter timing, estimates the offset component, removes the offset component, etc., by executing a program recorded in the memory. In addition, the control circuit 120 may also include an operation circuit that performs operation processing such as image processing. Such an operation circuit may be implemented by a combination of a programmable logic device (PLD) such as a digital signal processor (DSP) or a field programmable gate array (FPGA), or a central processing unit (CPU) or an image processing operation processor (GPU) and a computer program. In addition, the control circuit 120 and the operation circuit may be either a combined circuit or separate individual circuits.
[0121] Figure 1C 1 is a flowchart schematically showing the operation performed by the control circuit 120. The details will be described later, but the control circuit 120 generally performs Figure 1CThe action shown. The control circuit 120 first causes the light source 102 to emit pulse light for a specified time (step S101). At this time, the electronic shutter of the image sensor 110 is in a state of stopping exposure. The control circuit 120 stops the electronic shutter from exposing until the period in which a part of the pulse light is reflected from the surface of the object 101 and reaches the image sensor 110 ends. Next, the control circuit 120 starts exposing the electronic shutter at the timing when the other part of the pulse light is scattered inside the object 101 and reaches the image sensor 110 (step S102). After the specified time has passed, the control circuit 120 stops the electronic shutter from exposing (step S103). Next, the control circuit 120 determines whether the number of times the above-mentioned signal accumulation has been performed has reached a specified number (step S104). If the determination is "no", steps S101 to S103 are repeated until the determination is "yes". If it is determined to be “YES” in step S104 , the control circuit 120 causes the image sensor 110 to generate and output a signal representing an image based on the signal charges accumulated in each floating diffusion layer (step S105 ).
[0122] Through the above operation, it is possible to detect the components of light scattered inside the measurement object with high sensitivity. In addition, multiple times of light emission and exposure are not essential, but are performed as needed.
[0123] [1-4. Others]
[0124] The imaging device 100 may also include an imaging optical system that forms a two-dimensional image of the object 101 on the light receiving surface of the image sensor 110. The optical axis of the imaging optical system is substantially orthogonal to the light receiving surface of the image sensor 110. The imaging optical system may also include a zoom lens. If the position of the zoom lens changes, the magnification of the two-dimensional image of the object 101 changes, and the resolution of the two-dimensional image on the image sensor 110 changes. Therefore, even if the distance to the object 101 is far, the area to be measured can be enlarged and observed in detail.
[0125] In addition, the imaging device 100 may also include a bandpass filter between the object 101 and the image sensor 110 that allows only light in the wavelength band or its vicinity emitted from the light source 102 to pass through. This can reduce the influence of interference components such as ambient light. The bandpass filter is composed of a multilayer film filter or an absorption filter. Considering the temperature of the light source 102 and the wavelength shift accompanying the oblique incidence to the filter, the bandwidth of the bandpass filter may also have a width of about 20 to 100 nm.
[0126] In addition, the imaging device 100 may also include polarizing plates between the light source 102 and the object 101, and between the image sensor 110 and the object 101. In this case, the polarization directions of the polarizing plate arranged on the light source 102 side and the polarizing plate arranged on the image sensor side are in a cross-Nicol relationship. Thus, it is possible to prevent the regular reflection component (component with the same incident angle and reflection angle) of the surface reflection component I1 of the object 101 from reaching the image sensor 110. That is, it is possible to reduce the amount of light of the surface reflection component I1 reaching the image sensor 110.
[0127] [2.Action]
[0128] The camera device 100 of the present application distinguishes and detects the internal scattered component I2 from the surface reflected component I1. When the object 101 is set as a human forehead, the signal intensity of the internal scattered component I2 to be detected is very small. This is because, in addition to the very small amount of light that meets the laser safety standard as described above, the scattering and absorption of light caused by the scalp, cerebral myocardial fluid, skull, gray matter, white matter and blood flow are relatively large. Furthermore, the change in signal intensity caused by the change in blood flow or components in the blood flow during brain activity is equivalent to a few tens of times the size, which is very small. Therefore, it is best to avoid mixing the surface reflected component I1, which is thousands to tens of thousands times the signal component to be detected, into the image.
[0129] The operation of the imaging device 100 according to this embodiment will be described below.
[0130] like Figure 1A As shown in FIG. 1 , when the light source 102 irradiates the object 101 with pulse light, a surface reflection component I1 and an internal scattering component I2 are generated. Parts of the surface reflection component I1 and the internal scattering component I2 reach the image sensor 110. Since the internal scattering component I2 passes through the inside of the object 101 before being emitted from the light source 102 and reaching the image sensor 110, the optical path length is longer than that of the surface reflection component I1. Therefore, the time when the internal scattering component I2 reaches the image sensor 110 is later than that of the surface reflection component I1 on average.
[0131] Figure 2 1 is a diagram showing a light signal obtained by a rectangular pulse light emitted from a light source 102 and reflected from an object 101 and reaching an image sensor 110. Figure 2 In FIG. 1 , signal A represents the waveform of the surface reflection component I1. Signal B represents the waveform of the internal scattering component I2. Signal C represents the waveform of the surface reflection component I1 and the internal scattering component I2 added together. Signal D represents the timing of opening (OPEN) and closing (CLOSE) of the electronic shutter. The horizontal axis represents time, and the vertical axis of signals A to C represents the intensity of light, and the vertical axis of signal D represents the open or closed state of the electronic shutter.
[0132] As shown in signal A, the surface reflection component I1 maintains a rectangular shape. On the other hand, as shown in signal B, since the internal scattering component I2 is the sum of light that has passed through various optical path lengths, it shows a characteristic that the falling time at the rear end of the pulse light is longer than that of the surface reflection component I1. In order to increase the proportion of the internal scattering component I2 from signal C and extract it, as shown in signal D, it is sufficient to start the electronic shutter exposure after the rear end of the surface reflection component I1 (when or after the surface reflection component I1 falls). The shutter timing is adjusted by the control circuit 120. As described above, the camera device 100 of the present application only needs to be able to distinguish and detect the internal scattering component I2 from the surface reflection component I1, so the light pulse width and shutter width are arbitrary. Therefore, unlike the previous method of using an ultrafast scanning camera, it can be implemented with a simple structure and can significantly reduce costs.
[0133] according to Figure 2 It can be seen from the signal A that the rear end of the surface reflection component I1 drops vertically. In other words, the time from the beginning of the drop to the end of the drop of the surface reflection component I1 is zero. However, in reality, there are cases where the pulse light irradiated by the light source 102 itself is not completely vertical, or there are fine irregularities on the surface of the object 101, or the rear end of the surface reflection component I1 does not drop vertically due to scattering in the epidermis. In addition, since the object 101 is usually an opaque object in many cases, the surface reflection component I1 has a much larger amount of light than the internal scattering component I2. Therefore, even if the rear end of the surface reflection component I1 slightly exceeds the vertical drop position, the internal scattering component I2 will be buried and become a problem. In addition, due to the time delay of the electron movement during the readout period of the electronic shutter, it is not possible to achieve the following Figure 2 The ideal binary readout situation shown by the signal D. Therefore, the control circuit 120 may also delay the shutter timing of the electronic shutter slightly after the surface reflection component I1 just drops. For example, from the perspective of the accuracy of the electronic shutter, the shutter timing of the electronic shutter may be delayed by more than 1ns after the surface reflection component I1 just drops. In addition, instead of adjusting the shutter timing of the electronic shutter, the control circuit 120 may adjust the light emission timing of the light source 102. The control circuit 120 only needs to adjust the time difference between the shutter timing of the electronic shutter and the light emission timing of the light source 102. In addition, in the case of measuring the blood flow or the change of the components in the blood flow during brain activity in a non-contact manner, if the shutter timing is delayed too much, the minimum internal scattering component I2 will be further reduced, so the shutter timing may also be kept near the rear end of the surface reflection component I1. Since the time delay caused by the scattering of the object 101 is 4ns, the maximum delay amount of the shutter timing is about 4ns.
[0134] The light source 102 may emit pulse light a plurality of times, and multiple exposures may be performed with shutter timings of the same phase for each pulse light, thereby amplifying the detection light amount of the internal scattered component I2.
[0135] Alternatively or in addition to placing a bandpass filter between the object 101 and the image sensor 110, the control circuit 120 may estimate the offset component by performing imaging with the same exposure time without causing the light source 102 to emit light. The estimated offset component is subtracted from the signal detected by the light receiving element of the image sensor 110 to be removed. In this way, the dark current component generated in the image sensor 110 can be removed.
[0136] Figure 3 1 is a flowchart showing the actions of the camera device 100 before the formal measurement of the first embodiment. After the start, the camera device 100 performs confirmation of whether the environment of the object 101 is suitable for measurement by the measurement environment determination unit 140 (step S201). When the result of the confirmation of the measurement environment determines that the environment of the object 101 is not suitable for measurement (No in step S202), an error is output (step S210). When an error is output, after the treatment, the measurement environment confirmation is performed again. If it is determined that it is an environment suitable for measurement (Yes in step S202), the light intensity adjustment unit 130 then performs light intensity adjustment (step S203). Furthermore, after the light intensity adjustment is completed, the stability of the detection signal is measured by the signal stability determination unit 150 (step S204). When it is determined that the detection signal is unstable (No in step S205), an error is output (step S220). When an error is output, after the treatment, the signal stability measurement is performed again. When it is determined that the detection signal is stable (yes in step S205), the formal measurement is started (step S206). By implementing in this order, the measurement of blood flow changes in a living body can be carried out accurately and non-contactly without waste. For example, if the signal stability judgment is implemented before the measurement environment judgment, if the object 101 is not captured by the camera device 100 and other stationary objects are photographed, the signal stability judgment unit 150 will also erroneously judge that the signal is stable and proceed to the next step. In addition, even if the adjustment of the light emission amount adjustment unit 130 is implemented before the measurement environment judgment, when an object other than the object 101 is photographed for the same reason, the light amount is also erroneously adjusted. In addition, when the signal stability measurement is implemented before the light amount adjustment, if the light amount is too low or too high, the SN of the detection data of the camera device 100 decreases or saturates. Therefore, if Figure 3 As shown, in biological measurement using the imaging device 100 of the present application, it is optimal to perform the measurement environment determination, the emission light amount adjustment, and the signal stability determination in this order.
[0137] The following, Figure 3 The details of each function are described in the order of FIG. 4A to FIG. 4D FIG. 1 shows an example of determination by the measurement environment determination unit 140. The measurement environment determination unit 140 has a function of confirming whether the detection area 400 exists at the desired position of the object 101 and whether there are no interference error factors that affect the measurement. For example, when it is desired to observe the change of cerebral blood flow in the frontal lobe based on the change of oxyhemoglobin and deoxyhemoglobin, the forehead is photographed as the object 101. In this case, Figure 4A In this way, if no part other than the forehead is captured in the detection area 400, the measurement environment determination unit 140 determines that the environment is suitable for measurement. Figure 4B In the case where the detection area 400 includes parts other than the forehead such as hair and a headband, or Figure 4C If the detection area 400 is different from the desired measurement location, the measurement environment determination unit 140 determines that the environment is not suitable for measurement and outputs an error. Figure 4D Whether or not there is interference light mixed in can be determined by adding a mode in which pulse light is not emitted from the light source 102 but obtained by shutter, and confirming the pixel value of the offset component equivalent to the interference light. Interference light is light containing near-infrared light from 750 to 850nm close to the wavelength of the irradiated light source. In addition to sunlight, indoor lighting such as incandescent lamps, halogen lamps, and xenon lamps are also major factors. Tiny interference light can be removed by performing a differential operation on the offset component estimated by turning off the illumination of the light source and performing a shutter action obtained by the camera device 100. However, when there are a lot of offset components, the dynamic range of the photodiode is reduced. Therefore, for example, when the offset component satisfies half of the dynamic range, the measurement environment determination unit 140 determines that it is an environment not suitable for measurement.
[0138] like Figure 5A Since the imaging device 100 also has the function of being a camera for photographing the face of the subject, the camera image is displayed on the display 500 so that the subject or the inspector can recognize whether the environment is capable of measurement. At this time, the detection area 400 is displayed overlapping the captured image 510. If there are no obstructions in the captured image 510, the detection area 400 can be enlarged to be consistent with the entire area of the captured image 510. In this way, the pixels of the image sensor of the imaging device 100 can be used without waste, and higher-resolution measurements can be achieved. In addition, if Figure 5B By wirelessly connecting a tablet or smartphone as the display 500, more convenient measurements can be performed anytime and anywhere, such as at home or at a visiting place.
[0139] In such Figure 5C In the case where parts other than the part to be measured are mixed in the initial detection area 400, the user can manually change the detection area 400. The position adjustment icon 520 can be displayed on the captured image 510, and the position and size of the detection area 400 can be changed by dragging or inputting coordinates. In the case where the forehead of the subject is small and the hair or eyebrows are included in the initial detection area 400, the detection area 400 is reduced to match the size of the forehead of the subject. In addition, if the area of the captured image 510 includes feature quantities such as eyes, eyebrows, and nose for measurement, the detection area 400 is automatically set in a specified area of the forehead by pressing the automatic adjustment button using facial recognition calculation. In the case where the forehead is blocked by an obstruction such as hair, or the feature quantity is not detected correctly, an error is returned that the detection area 400 cannot be set. In addition, if the area maximization is set to ON through automatic adjustment, it can be as follows Figure 5D Thus, the forehead exposed part is detected by image processing, and the entire forehead is set as the detection area 400. Thus, by using the GUI for setting the detection area 400, adjustment can be made so as to accurately and easily obtain the two-dimensional distribution of cerebral blood flow to the maximum extent in the entire forehead.
[0140] In addition, you can Figure 5E In this way, multiple detection areas 400 are set. To add detection areas 400, tap on the screen. To delete a detection area 400, tap the detection area 400 you want to delete for a long time. By setting multiple detection areas 400 at specific locations, it is possible to perform a customized evaluation of the brain activity location of interest. Since there is only information on a specific location, it is also possible to reduce the load of data processing and the amount of transmission.
[0141] If the measurement is to be started with the detection area 400 including the parts other than the measurement object such as hair and eyebrows, Fig. 6AAs shown, the error of confirming whether the detection area 400 is correct is prompted by outputting characters, sounds, error sounds, etc. The judgment of whether the part other than the measurement object is included is realized by image processing using the image obtained by the camera device 100. For example, when the contrast is found to be extremely changed locally in the intensity distribution within the detection area 400, it is judged that the part other than the measurement object is mixed. The extreme change in contrast is the case where the pixel values around the pixel of interest change by more than 20%. Regarding the change in contrast, it can be easily detected if an edge detection filter such as Sobel, Laplacian, Canny, etc. is used. In addition, as another method, it can also be pattern matching of the feature amount of the interference factor or judgment based on machine learning. Since the interference factor in the case of detecting the forehead is determined to be hair, eyebrows, etc. to some extent, even in the method using learning, not too much data is required in the prior learning, so it is easy to implement. In addition, it is also possible to make an exception based on size and make a smoothed judgment so that the subtle contrast changes of moles, wrinkles, etc. can be ignored.
[0142] In the output Fig. 6A In the event of an error, the detection area 400 is changed on the screen. In this case, the detection area 400 is adjusted manually or automatically. In addition, if the area of the captured image 510 is extremely offset from the desired position and the detection area 400 cannot be changed by software on the screen, the detection area 400 can be set to the desired position by moving the display 500 while the subject confirms it. In this case, the detection area 400 can be adjusted manually or automatically. Figure 6B Auxiliary lines 530 are displayed on the display 500 so that it is easy to understand the position of the subject relative to the detection area 400. Based on the auxiliary lines 530, the center of the detection area 400 and the center of the forehead of the subject can be smoothly adjusted. When the subject makes adjustments while observing the display 500, the captured image 510 can be displayed as a left-right inverted mirror image to facilitate adjustments. In addition, the examiner can also adjust the detection area 400 by changing the angle and position of the camera 100 while checking the display 500. On the camera 100, as shown in FIG. Figure 6C As shown in FIG. 1 , an adjustment table 540 is installed to adjust the direction of the imaging device 100 in the x, y, z directions and the tilt (swing, pitch, rotation) so that the direction of the imaging device 100 can be adjusted so that the forehead of the subject can be illuminated with light and the camera can be detected. Fig.6D In this way, the subject can fix the chin and head with the fixing clamp 550, and the measurement with further reduced motion effect error can be performed. By the examiner moving the camera device 100 to adjust, the burden on the subject can be reduced compared to the subject's own adjustment, and the psychological noise effect on the cerebral blood flow information to be obtained can also be reduced.
[0143] like Fig. 7A and Figure 7B As shown, the brightness of the captured image 510 detected by the camera device 100 varies depending on the object 101. This is due to the difference in light absorption caused by the skin color of the object 101, that is, melanin. If the object 101 is too bright, the captured image 510 is saturated and cannot be measured. If it is too dark, it will affect the SN of the detected light amount, which is not good. Therefore, the luminous light amount adjustment unit 130 adjusts the light amount of the light source 102 according to the brightness of the object 101. In addition, the surface reflectivity and diffusion rate vary from person to person depending on the sweating state and skin shape of the object 101. As shown in FIG. Figure 7B As shown, even when bloom 710 is seen on the object 101, the emission light amount adjustment unit 130 adjusts the light amount so as not to saturate the light.
[0144] Since the imaging device 100 detects very little light that reaches the brain and is reflected back, it is important to ensure the amount of light to be detected. Therefore, since the SN is not improved in the digital gain adjustment in image processing, the light amount of the light source 102 is increased to ensure sensitivity. However, in order to comply with the safety standard level 1 of the laser, the amount of light that can be irradiated is limited. Therefore, instead of increasing the light amount per pulse of the light source 102, the imaging device 100 of this embodiment increases the light amount per pulse of the light source 102, such as Figure 7C As shown, it has a light intensity adjustment function to adjust the number of times the pulse light is emitted within one frame. Figure 7C In the figure, signal E represents the waveform of the pulse light emitted from the light source 102. Signal C represents the waveform of the surface reflection component I1 and the internal scattering component I2 added together. Signal D represents the timing of opening and closing of the electronic shutter. Signal F represents the timing of charge accumulation in the charge accumulation unit. The horizontal axis represents time, and the vertical axis in signals C and E represents the intensity of light. The vertical axis in signal D represents the open or closed state of the electronic shutter, and the vertical axis in signal F represents the open or closed state of the charge accumulation unit. By changing the number of pulses emitted by the light source 102 within one frame, the amount of light irradiated to the object 101 and the amount of light detected in the imaging device 100 can be adjusted. The light amount adjustment by changing the number of pulses also has better stability of the laser intensity than the method of changing the current value of the laser diode. At this time, the number of shutter times within one frame also increases or decreases in synchronization with the change in the number of light emitting pulses. As shown in FIG. Figure 7C As shown, if the time for calculation and output processing is ensured within one frame, the pulse light can be increased in the remaining period. Therefore, changing the pulse light per one frame means changing the average number of pulse lights emitted per unit time.
[0145] Fig. 8A and Figure 8B1 is a diagram showing the function of the signal stability determination unit 150 of the imaging device 100. The signal stability determination unit 150 checks the stability of the time series data of the detection signal when the subject is in the rest state. The rest state is a state in which the subject does not think about anything. In order to guide the subject to the rest state, the subject is allowed to continuously watch an image without patterns, or to continuously watch an image with only dots or plus signs. At this time, it is ideal to Fig. 8A As shown in FIG. 1 , the cerebral blood flow signal of the subject does not show any increase or decrease but remains constant. However, depending on the condition of the subject, Figure 8B As shown in FIG. 1 , the detection signal is unstable. One of the unstable factors is that the mental state of the subject is not in a state of full concentration. In this case, if Fig. 9 As shown, an output indicating that the signal is unstable is performed on the display 500, and countermeasures such as calming the subject or taking a period of time are taken to reconfirm the signal stability. In addition, if the subject moves or moves his eyebrows during the signal stability evaluation, the detection signal changes. The change in the detection signal caused by body movement can be determined by calculating oxyhemoglobin and deoxyhemoglobin. Since the measurement is performed in a non-contact manner, if body movement occurs, the distance between the camera device 100 and the object 101 changes, the amount of irradiated light on the object 101 changes, and the amount of light incident on the object 101 increases or decreases. Therefore, body movement causes changes in the detection signal, so both oxyhemoglobin and deoxyhemoglobin change greatly in the same positive and negative directions at the same time. Therefore, the changes in oxyhemoglobin and deoxyhemoglobin are observed, and when a signal change unique to body movement is detected, the camera device 100 outputs an erroneous response instructing the subject not to move. In addition, sometimes the detection signal is unstable due to the instability of the light source 102. This is because the laser light intensity decreases monotonically as the temperature changes, and the oxygenated hemoglobin and deoxyhemoglobin signals appear to increase monotonically. Therefore, based on this monotonous change phenomenon, it is possible to determine whether the light source 102 is stable. In this case, the camera device 100 responds by issuing an instruction to wait until the light source 102 is stable, or by performing a calibration correction process for the intensity change of the light source caused by the temperature. Through the stability determination of the signal stability determination unit 150, a more accurate measurement can be performed with reduced / eliminated error factors.
[0146] If there are no problems in the measurement environment confirmation, light intensity adjustment, and detection signal stability confirmation, the actual measurement will then begin.
[0147] (Implementation Method 2)
[0148] In the second embodiment, the imaging device 800 includes an abnormal value determination unit 810 that detects the occurrence of an abnormal value during measurement. Here, in the present embodiment, detailed descriptions of the same contents as those in the first embodiment are omitted. The abnormal value determination unit 810 corresponds to a calculation unit.
[0149] Fig. 10A 1 is a schematic diagram showing an imaging device 800 according to Embodiment 2 and a state in which the imaging device 800 images an object 101. Different from Embodiment 1, an abnormal value determination unit 810 is added. Fig. 10B 1 is a flowchart showing the operation of the camera device 800 in the formal measurement of the second embodiment. In the second embodiment, during the formal measurement (step S902), an abnormal value is determined (step S904). When the abnormal value determination unit 810 determines that an abnormal value has occurred (yes in step S906), it is confirmed whether the environment of the object 101 is suitable for the measurement (step S201). The abnormal value is a value used to confirm whether an irregular value has occurred in the detection signal during the measurement. For example, an obstruction such as hair, interference light, and body movement are factors that cause abnormal values. If an obstruction such as hair is mixed in the measurement, the hair absorbs light, so the detection signal drops extremely, and the cerebral blood flow signal increases in a simulated manner, so it can be determined. Alternatively, it is determined whether a foreign object is mixed in the camera image of the camera device 800 by image recognition. In addition, if interference light is mixed, it is determined based on the situation that the detected offset component increases greatly. Furthermore, when body movement occurs, since the values of oxyhemoglobin (HbO2) and deoxyhemoglobin (Hb) change rapidly at the same time, it can be determined. Fig.11A This is the time series data of cerebral blood flow changes when the abnormal value determination unit 810 determines that there is no abnormal value. For tasks, oxygenated hemoglobin increases more often, while deoxyhemoglobin tends to decrease or increase slightly. On the other hand, Fig. 11B This is an example in which the detection signal changes significantly due to the body movement of the subject during the measurement. In response to body movement, the amount of light irradiated on the forehead increases or decreases, so the apparent values of oxygenated hemoglobin and deoxygenated hemoglobin increase or decrease in the same direction. In the case of a signal value that exceeds the normal blood flow change of a person (about 0.1mM·mm), the abnormal value determination unit 810 displays an error. For example, when HbO2 is above 1mM·mm, an abnormal value error is output. In addition, since the blood flow change does not occur sharply, when the time series waveform changes approximately by 90°, or when there is a blood flow change of more than 0.1mM·mmm within 1 second, the possibility of an abnormal value is high, so an abnormal value error response is performed. In addition, the presence or absence of body movement can also be detected by a moving body detection image processing operation based on the image data of the camera device 800. As moving body detection, methods such as optical flow, template matching, block matching, and background difference are used.
[0150] When the abnormal value determination unit 810 determines that an abnormal value has occurred in the actual measurement, Fig. 12A and Fig. 12B As shown, the occurrence of abnormal value or the occurrence of deviation of the detection area due to body movement is outputted on the display 500. After the operator takes measures to deal with the abnormal value factor as needed, the measurement environment confirmation before the actual measurement described in the first embodiment is repeated.
Claims
1. A camera system, in, have: A light source, which emits pulsed light for irradiating the organism; a photodetector that detects light returning from the organism in response to the irradiation of the pulsed light and outputs a signal corresponding to cerebral blood flow in the organism; and Operation Department, The calculation unit determines whether or not a temporal change in cerebral blood flow of the living body is within a certain range based on the signal from the photodetector.
2. The camera system according to claim 1, in, Also provided is a guiding device for guiding the above-mentioned organism to a resting state, The calculation unit determines whether the change over time of the cerebral blood flow of the biological body guided to the resting state by the guiding device is within a certain range in the determination of whether the change over time of the cerebral blood flow is within a certain range.
3. The camera system according to claim 1, in, The above-mentioned operation unit, Determine whether the environment of the organism is suitable for the measurement of the interior of the organism, When it is determined that the environment of the living body is suitable for measurement of the interior of the living body, it is determined whether the temporal change of the cerebral blood flow of the living body is within a certain range.
4. The camera system according to claim 3, in, The calculation unit adjusts the light intensity of the pulse light when determining that the environment of the living body is suitable for measuring the inside of the living body.
5. The camera system according to claim 1, in, The above-mentioned operation unit, Based on the change in the luminous intensity of the light source, the stability corresponding to the temperature change of the light source is determined. When it is determined that the light source is unstable, at least one of the following is performed: outputting an instruction to wait until the light source is stable, and correcting a change in the light emission intensity of the light source caused by temperature.
6. A method for photographing, in, The light source emits pulsed light for irradiating the living body. In response to the irradiation of the pulse light, the light detector detects the light returning from the organism and outputs a signal corresponding to the cerebral blood flow of the organism. Based on the signal from the photodetector, it is determined whether the temporal change in the cerebral blood flow of the living body is within a certain range.
7. A computer program product, in, Make the processor execute: causing the light source to emit pulsed light for irradiating the living body; In response to the irradiation of the pulsed light, the light detector detects the light returning from the organism and outputs a signal corresponding to the cerebral blood flow of the organism; as well as Based on the signal from the photodetector, it is determined whether the temporal change in the cerebral blood flow of the living body is within a certain range.
