Imaging device and imaging program

Abstract

To provide an imaging device configured to obtain a high-resolution captured image without artifacts, in which the influence of background light is removed even when a subject moves in any direction.SOLUTION: An imaging device is configured to: perform exposure in synchronization with turning-on and turning-off of a light emitter to capture a turning-on state exposure frame image exposed when the light emitter is turned on, a turning-off state exposure frame image exposed when the light emitter is turned off; calculate a motion vector of each pixel from the two of the turning-on state exposure frame images or the two of the turning-off state exposure frame images; adjust a length of the motion vector of each pixel according to a time difference between the turning-on state exposure frame image of a processing target and the turning-off state exposure frame image used for the processing; correct each pixel of the turning-off state exposure frame image used for the processing, based on the adjusted motion vector to calculate a virtual turning-off state exposure frame image; and subtract the virtual turning-off state exposure frame image from the turning-on state exposure frame image of the processing target.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an imaging device and an imaging program. [Background technology]

[0002] The InCabin Driver Monitoring System is a system that monitors the driver's behavior inside the vehicle. In this system, to remove the influence of background light (external light) on the acquired infrared images, a difference image is calculated between the image when the LED that illuminates the shooting area is turned on and the image when it is turned off. Then, based on this difference image, the driver's line of sight, etc. are detected, and the driver's behavior, such as drowsy driving, can be detected.

[0003] However, when the driver or the camera moves, artifacts occur in the differential image due to the driver's position shift, etc. The occurrence of artifacts reduces the accuracy of detecting the driver's line of sight, etc.

[0004] The following prior art is described in Patent Document 1: A difference image is generated, which is the difference in brightness between a first image captured when the object is not illuminated and a second image captured when the object is illuminated. A pixel region consisting of pixels having a brightness equal to or greater than a predetermined brightness value is extracted from the difference image. Based on the extracted pixel region, the amount of movement of the object between the timing of capturing the first image and the timing of capturing the second image is estimated. Then, based on the estimated amount of movement, the subject portion in either the first image or the second image is shifted, and a difference image is then created again. This makes it possible to obtain an appropriate image of the object even if the object moves slightly, improving the authentication system. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Patent No. 4842374 Summary of the Invention [Problem to be solved by the invention]

[0006] However, the above-mentioned prior art estimates the amount of movement of the object to be imaged from a differential image between an image of the object when illuminated and an image of the object when not illuminated, and therefore there is a problem that there is a limit to the accuracy of estimating the amount of movement, etc., and it is not always possible to obtain a clear image from which the effects of background light have been removed.

[0007] The present invention has been made to solve the above problems, and an object of the present invention is to provide an imaging device, an imaging method, and an imaging program that are capable of obtaining high-resolution images without artifacts and from which the influence of background light is removed, regardless of the direction in which a subject moves. [Means for solving the problem]

[0008] The above problems can be solved by the following means.

[0009] (1) An imaging device having an imaging unit that captures a lit-time exposure frame image exposed when the light is on and an off-time exposure frame image exposed when the light is off by exposing in synchronization with the turning on and off of a light-emitting unit that irradiates light onto a subject; a motion vector calculation unit that calculates a motion vector of each pixel from two of the lit-time exposure frame images or two of the off-time exposure frame images; a motion vector length adjustment unit that adjusts the length of the motion vector of each pixel according to the time difference between the lit-time exposure frame image to be processed and the off-time exposure frame image used for the processing; an off-time exposure frame image correction unit that calculates a virtual off-time exposure frame image that is estimated to be captured if the light-emitting unit is off when the lit-time exposure frame image to be processed is exposed by correcting each pixel of the off-time exposure frame image to be used for the processing based on the adjusted motion vector of each pixel; and a lit-time exposure frame image processing unit that performs the process of subtracting the calculated virtual off-time exposure frame image from the lit-time exposure frame image to be processed.

[0010] (2) An imaging program for causing a computer to execute a process including: an imaging step of capturing a lit-time exposure frame image exposed when the light is on and an off-time exposure frame image exposed when the light is off by exposing in synchronization with the turning on and off of a light-emitting unit that irradiates light onto a subject; a motion vector calculation step of calculating a motion vector of each pixel from the two lit-time exposure frame images or the two off-time exposure frame images; a motion vector length adjustment step of adjusting the length of the motion vector of each pixel in accordance with the time difference between the lit-time exposure frame image to be processed and the off-time exposure frame image to be used for the processing; an off-time exposure frame image correction step of correcting each pixel of the off-time exposure frame image to be used for the processing based on the adjusted motion vector of each pixel, thereby calculating a virtual off-time exposure frame image that is estimated to be captured if the light-emitting unit was off when the lit-time exposure frame image to be processed was exposed; and a lit-time exposure frame image processing step of performing the process of subtracting the calculated virtual off-time exposure frame image from the lit-time exposure frame image to be processed. [Effects of the Invention]

[0011] The system calculates a motion vector for each pixel from two lights-on exposure frame images or two lights-off exposure frame images, and adjusts the length of the motion vector according to the time difference between the lights-on exposure frame image to be processed and the lights-off exposure frame image to be used for the processing.The system then corrects the lights-off exposure frame image to be used for the processing based on the adjusted motion vector and subtracts it from the lights-on exposure frame image to be processed.This makes it possible to obtain high-resolution images without artifacts, with the effects of background light removed, regardless of the direction in which the subject moves. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 2 is a block diagram showing a hardware configuration of the imaging apparatus. [Figure 2]FIG. 2 is a block diagram showing functions of a control unit of the image processing apparatus. [Figure 3] 3A and 3B are explanatory diagrams showing examples of the positional relationship between a light emitting unit and a camera of an imaging unit and a subject. [Figure 4] 10A and 10B are explanatory diagrams illustrating examples of two frame images used to calculate a movement vector and a lights-out exposure frame image used in processing. [Figure 5] FIG. 10 is an explanatory diagram illustrating an example of a process for calculating a movement vector, adjusting the length of the movement vector, calculating a virtual unlit exposure frame image, and subtracting the virtual unlit exposure frame image from a lit exposure frame image. [Figure 6] 4 is a flowchart illustrating the operation of the imaging device. [Figure 7] FIG. 10 is an explanatory diagram illustrating an example of a process for calculating a movement vector, adjusting the length of the movement vector, calculating a virtual unlit exposure frame image, and subtracting the virtual unlit exposure frame image from a lit exposure frame image. [Figure 8] 4 is a flowchart illustrating the operation of the imaging device. [Figure 9] FIG. 10 is an explanatory diagram illustrating an example of a process for calculating a movement vector, adjusting the length of the movement vector, calculating a virtual unlit exposure frame image, and subtracting the virtual unlit exposure frame image from a lit exposure frame image. [Figure 10] 4 is a flowchart illustrating the operation of the imaging device. [Figure 11] FIG. 10 is an explanatory diagram illustrating an example of a process for calculating a movement vector, adjusting the length of the movement vector, calculating a virtual unlit exposure frame image, and subtracting the virtual unlit exposure frame image from a lit exposure frame image. [Figure 12] 4 is a flowchart illustrating the operation of the imaging device. [Figure 13] FIG. 10 is an explanatory diagram illustrating an example of a process for calculating a movement vector, adjusting the length of the movement vector, calculating a virtual unlit exposure frame image, and subtracting the virtual unlit exposure frame image from a lit exposure frame image. [Figure 14] 4 is a flowchart illustrating the operation of the imaging device. [Figure 15] FIG. 10 is an explanatory diagram illustrating an example of a process for calculating a movement vector, adjusting the length of the movement vector, calculating a virtual unlit exposure frame image, and subtracting the virtual unlit exposure frame image from a lit exposure frame image. [Figure 16] 4 is a flowchart illustrating the operation of the imaging device. [Figure 17] FIG. 10 is an explanatory diagram illustrating an example of a process for calculating a movement vector, adjusting the length of the movement vector, calculating a virtual unlit exposure frame image, and subtracting the virtual unlit exposure frame image from a lit exposure frame image. [Figure 18] 4 is a flowchart illustrating the operation of the imaging device. [Figure 19] 10A and 10B are explanatory diagrams illustrating examples of two lit-on exposure frame images used to calculate a movement vector and an unlit-on exposure frame image used in processing. [Figure 20] FIG. 10 is an explanatory diagram illustrating an example of a process for calculating a movement vector, adjusting the length of the movement vector, calculating a virtual unlit exposure frame image, and subtracting the virtual unlit exposure frame image from a lit exposure frame image. [Figure 21] 4 is a flowchart illustrating the operation of the imaging device. [Figure 22] FIG. 10 is an explanatory diagram illustrating an example of a process for calculating a movement vector, adjusting the length of the movement vector, calculating a virtual unlit exposure frame image, and subtracting the virtual unlit exposure frame image from a lit exposure frame image. [Figure 23] 4 is a flowchart illustrating the operation of the imaging device. [Figure 24] FIG. 10 is an explanatory diagram illustrating an example of a process for calculating a movement vector, adjusting the length of the movement vector, calculating a virtual unlit exposure frame image, and subtracting the virtual unlit exposure frame image from a lit exposure frame image. [Figure 25] 4 is a flowchart illustrating the operation of the imaging device. [Figure 26]FIG. 10 is an explanatory diagram illustrating an example of a process for calculating a movement vector, adjusting the length of the movement vector, calculating a virtual unlit exposure frame image, and subtracting the virtual unlit exposure frame image from a lit exposure frame image. [Figure 27] 4 is a flowchart illustrating the operation of the imaging device. [Figure 28] 10A and 10B are explanatory diagrams for explaining capturing a plurality of frame images through a plurality of exposures, and capturing a lit-on exposure frame image and a lights-off exposure frame image by adding together the plurality of frame images. [Figure 29] An explanatory diagram for explaining capturing a light-on exposure frame image by accumulating the signal amount from multiple exposures when the light is on, and capturing a light-off exposure frame image by accumulating the signal amount from multiple exposures when the light is off. [Figure 30] FIG. 10 is a diagram showing a first example of a case where the first to third frame images are not consecutive. [Figure 31] FIG. 10 is a diagram showing a second example in which the first to third frame images are not consecutive. [Figure 32] FIG. 10 is a diagram showing a third example in which the first to third frame images are not consecutive. [Figure 33] FIG. 10 is a diagram showing a modified example in which the first to third frame images are consecutive. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, an imaging device and an imaging program according to an embodiment will be described in detail with reference to the drawings. The described embodiments are merely exemplary, and various modifications are possible from such embodiments. Hereinafter, the same reference numerals in the drawings refer to the same components, and the size of each component in the drawings is expressed in proportions different from the actual size for the sake of clarity and convenience of explanation.

[0014] Hereinafter, the expressions "upper" or "above" include not only what is directly above / below / left / right in contact, but also what is above / below / left / right without contact.

[0015] Terms such as "first" and "second" are used to describe various components, but are used only to distinguish one component from another, and are not intended to limit the materials or structures of the components.

[0016] An element expressed in the singular includes a plurality of elements unless the context clearly dictates otherwise. Furthermore, when a part "comprises" a certain element, this does not mean that other elements are excluded, but that other elements may also be included, unless otherwise specified to the contrary.

[0017] In addition, terms such as "unit" and "module" used in the specification refer to a unit that processes one or more functions or operations, and may be realized by hardware or software, or a combination of hardware and software.

[0018] (First embodiment) Fig. 1 is a block diagram showing the hardware configuration of the imaging device 1. Fig. 2 is a block diagram showing the functions of a control unit 210 of an image processing device 200. Fig. 3 is an explanatory diagram showing an example of the positional relationship between a light emitting unit 110 and a camera 120 of an imaging unit 100 and a subject 500.

[0019] The imaging device 1 includes an imaging unit 100 and an image processing device 200. The imaging unit 100 and the image processing device 200 are connected to each other so that they can communicate with each other. The imaging unit 100 includes a light emitting unit 110, a camera 120, a control unit 130, and a communication unit 140. The image processing device 200 includes a control unit 210, a storage unit 220, and a communication unit 230.

[0020] (Image capture unit 100) The light emitting unit 110 irradiates light onto the subject 500. Specifically, the light emitting unit 110 is turned on or off under the control of the control unit 130, and irradiates light onto the subject 500 when it is turned on. The light emitting unit 110 is configured, for example, by an LED that emits infrared light. The wavelength of the light emitted by the LED is preferably 940 nm. The light emitting unit 110 may be an LED that emits visible light. The light emitting unit 110 may also be configured by a Vicsel.

[0021] Camera 120 has an imaging element including multiple elements (pixels), integrates the light detected by each element, and outputs frame image 700 (see FIG. 4, etc.), which is a digital signal. Camera 120 captures subject 500 as temporally consecutive frame images 700. Specifically, camera 120, under the control of control unit 130, performs exposure in synchronization with the turning on and off of light-emitting unit 110, thereby capturing lit-on exposure frame image 710 exposed when the light is on and lit-off exposure frame image 720 exposed when the light is off (see FIG. 5, etc.). Camera 120 is, for example, an infrared camera. Camera 120 may also be a visible light camera.

[0022] The control unit 130 may be configured with a CPU and a memory. The control unit 130 controls each element constituting the imaging unit 100 and performs various calculation processes. The control unit 130 transmits a control signal to the light emitting unit 110 for turning on and off the light emitting unit 110. The control unit 130 transmits a control signal to the camera 120 for exposing the camera 120 at a predetermined timing, synchronized with the control signal for turning on and off the light emitting unit 110.

[0023] The communication section 140 transmits the frame images 700 to the image processing device 200. The frame images 700 include a lit-on exposure frame image 710 and a light-off exposure frame image 720.

[0024] (Image processing device 200) The control unit 210 may be configured with a CPU and a memory, and controls each element that configures the image processing device 200 and performs various calculation processes.

[0025] As shown in FIG. 2, the control unit 210 functions as a motion vector calculation unit 211, a motion vector length adjustment unit 212, a lights-out exposure frame image correction unit 213, a low-pass filter unit 214, and a lights-on exposure frame image processing unit 215 by the CPU executing a program.

[0026] As will be described below, the control unit 210 performs a process (hereinafter also simply referred to as "processing") of subtracting, from the lit-exposed frame image 710, a lights-out exposed frame image 720 in which the positions of each pixel have been corrected (a virtual lights-out exposed frame image 721 (see FIG. 5, etc.) described below). In this way, the control unit 210 removes the influence of background light (external light) from the lit-exposed frame image 710. The subject 500 may be irradiated with light from the light-emitting unit 110 and external light such as sunlight (see FIG. 3). The subject 500 is, for example, a person, but may also be an animal or a moving object.

[0027] The motion vector calculation unit 211 calculates a motion vector 730 (see FIG. 5, etc.) for each pixel from the two lit-on exposure frame images 710 or the two unlit-exposure frame images 720. The motion vector 730 is a vector whose elements are the length and direction of a line connecting each pixel in the temporally previous frame image 700 with the corresponding pixel in the temporally subsequent frame image 700. The method for calculating the motion vector 730 is well known, so a description thereof will be omitted. For example, an optical flow algorithm may be used to calculate the motion vector.

[0028] The motion vector length adjustment unit 212 adjusts the length of the motion vector 730 for each pixel according to the time difference between the lit-exposed frame image 710 to be processed and the unlights-exposed frame image 720 used in the process (the subtraction process described above). Either one of the two frame images 700 used to calculate the motion vector 730 can be the lit-exposed frame image 710 to be processed or the unlights-exposed frame image 720 to be used in the process. In the present embodiment, two lit-exposed frame images 710 are used to calculate the motion vector, and one of the two lit-exposed frame images 710 is used as the lit-exposed frame image 710 to be processed. The motion vector length adjustment section 212 can adjust the length of the motion vector 730 for each pixel by multiplying the length of the motion vector 730 by the ratio of the time difference between the lit-exposed frame image 710 to be processed and the unlit-exposed frame image 720 used in the processing to the time difference between the two frame images 700 used to calculate the motion vector 730. The time difference between the two frame images 700 is calculated based on the frame rate of the imaging section 100, for example.

[0029] The lights-out exposure frame image correction unit 213 corrects each pixel of the lights-out exposure frame image 720 used for processing based on the movement vector 731 of each pixel after the length adjustment, thereby calculating a virtual lights-out exposure frame image 721 that is estimated to have been captured if the light-emitting unit was off when the lights-out exposure frame image 710 to be processed was exposed. Specifically, the lights-out exposure frame image correction unit 213 calculates the virtual lights-out exposure frame image 721 by shifting each pixel of the lights-out exposure frame image 720 to be used for processing in a direction along the direction of the movement vector 730 by the length of the movement vector 730 after the length adjustment. The direction in which each pixel of the lights-out exposure frame image 720 to be used for processing is shifted is the same direction as or opposite to the movement vector 730. The direction in which each pixel of the lights-out exposure frame image 720 to be used for processing is shifted differs depending on the temporal positional relationship between the lights-out exposure frame image 710 to be processed and the lights-out exposure frame image 720 to be used for processing.

[0030] The low-pass filter unit 214 filters the virtual unlights-exposed frame image 721 using a low-pass filter to reduce the influence of edge noise caused by movement errors in the shifting of each pixel when calculating the virtual unlights-exposed frame image 721. As the low-pass filter, for example, a Gaussian filter, a median filter, or a bilateral filter may be used.

[0031] The lit-on exposure frame image processing unit 215 performs the above process of subtracting the virtual unlit-on exposure frame image 721 from the lit-on exposure frame image 710 that is the processing target.

[0032] 4 is an explanatory diagram illustrating an example of two frame images 700 used to calculate a movement vector 730 and an example of a lights-out exposure frame image 720 used for processing. The illumination light control signal is a control signal that the control unit 130 uses to turn on and off the light emitting unit 110. The image sensor frame signal is a control signal that the control unit 130 uses to cause the camera 120 to perform exposure and readout from each element (pixel).

[0033] The first, second, and third frame images are frame images 700 captured successively in time. In the example of Fig. 4, the first and third frame images are lit-exposure frame images 710, which are frame images 710 captured using illumination light emitted by the light-emitting unit 110 and background light (including reflected light from these). The second frame image is an unlit-exposure frame image 720, which is a frame image 700 captured using only background light (including reflected light from these) without being illuminated with illumination light. When the first frame image is the lit-exposure frame image 710 to be processed, the second frame image is the unlit-exposure frame image 720 used in processing, and the first and third frame images can be the two frame images 700 used to calculate a movement vector 730.

[0034] FIG. 5 is an explanatory diagram illustrating an example of the calculation of the movement vector 730, the adjustment of the length of the movement vector 730, the calculation of the virtual unlighted exposure frame image 721, and the process of subtracting the virtual unlighted exposure frame image 721 from the lit exposure frame image 710.

[0035] The first frame image ((1) in FIG. 5), the second frame image ((2) in FIG. 5), and the third frame image ((3) in FIG. 5) are a lit-exposed frame image 710, an unlit-exposed frame image 720, and a lit-exposed frame image 710, respectively. These frame images 700 are captured in the order of the first frame image, the second frame image, and the third frame image. The first frame image, the second frame image, and the third frame image constitute the first lit-exposed frame image, the first unlit-exposed frame image, and the second lit-exposed frame image, respectively. The first frame image is the lit-exposed frame image 710 that is the target of processing. The second frame image is the unlit-exposed frame image 720 used in processing. The first frame image and the third frame image are two frame images 700 used in calculating a movement vector 730. The second frame image used in processing is interpolated into the first frame image and the third frame image that are used in calculating the movement vector 730. In Figure 5, in the diagram of the second frame image (Figure 5 (2)) and the diagram of the third frame image (Figure 5 (3)), the outline of subject 500 in the first frame image is shown with a dashed line to make the explanation easier to understand.

[0036] A movement vector 730 is calculated for each pixel from the first and third frame images ((4) in FIG. 5). The movement vector 730 is indicated by an arrow in (4) in FIG. 5. In (4) in FIG. 5, the movement vector 730 is also indicated in an enlarged view surrounded by a circle.

[0037] The length of the calculated movement vector 730 is adjusted according to the ratio of the time difference between the first and third frame images to the time difference between the first and second frame images.

[0038] A virtual lights-out exposure frame image 721 is calculated by correcting each pixel of the second frame image by shifting it in the opposite direction to the movement vector 730 by the length of the length-adjusted movement vector 731 (FIG. 5(5)). The direction and amount of shift for each pixel in this correction are indicated by arrows in the diagram of FIG. 5(5). In FIG. 5(5), the length-adjusted movement vector 731 is also shown in an enlarged view surrounded by a circle.

[0039] In order to reduce the influence of edge noise due to movement errors in the shifting of each pixel when calculating the virtual unlit exposure frame image 721, a filter may be applied to the virtual unlit exposure frame image 721 ((6) in FIG. 5).

[0040] The virtual unlit exposure frame image 721 is subtracted from the first frame image ((7) in FIG. 5).

[0041] 6 is a flowchart showing the operation of the imaging device 1. This flowchart can be executed by the cooperative operation of the control unit 130 of the imaging unit 100 and the control unit 210 of the image processing device 200 in accordance with a program.

[0042] The control unit 130 causes the camera 120 to capture a first frame image, a second frame image, and a third frame image in synchronization with the turning on, turning off, and turning on of the light emitting unit 110 (S101). The control unit 130 then transmits the first frame image, the second frame image, and the third frame image to the image processing device 200 via the communication unit 140.

[0043] Control unit 210 receives the first, second, and third frame images via communication unit 230, and calculates movement vector 730 from the first and third frame images (S102).

[0044] Control unit 210 adjusts the length of movement vector 730 in accordance with the ratio of the time difference between the first and third frame images to the time difference between the first and second frame images (S103).

[0045] Control unit 210 calculates virtual lights-out exposure frame image 721 by shifting each pixel of the second frame image by the length of length-adjusted movement vector 731 in the opposite direction to movement vector 730 (S104).

[0046] The control unit 210 executes a process of subtracting the virtual unlit exposure frame image 721 from the first frame image (S105).

[0047] The control unit 210 can execute the processing of the flowchart shown in FIG. 6, treating the sequentially received lit exposure frame images 710 as the lit exposure frame images 710 to be sequentially processed.

[0048] According to this embodiment, the lights-out exposure frame image 720, which is subtracted from the lights-on exposure frame image 710, is corrected by shifting it pixel by pixel, so that no matter in which direction the subject 500 moves or the camera 120 moves, an image without artifacts and with the influence of background light removed can be obtained.

[0049] (Second embodiment) The second embodiment will be described. The differences between this embodiment and the first embodiment are as follows: In the first embodiment, the length of the motion vector 730 is adjusted for each pixel. On the other hand, in this embodiment, the average length of the motion vector 730 for each pixel is adjusted, and the motion vectors 730 of each pixel other than specific pixels whose motion vector 730 length is equal to or less than a predetermined value are uniformly adjusted to the adjusted average. In other respects, this embodiment is similar to the first embodiment, so duplicated explanations will be omitted or simplified.

[0050] The motion vector length adjustment unit 212 shown in FIG. 2 sets pixels whose motion vector 730 length is equal to or less than a predetermined threshold as specific pixels whose motion vector 730 length is not adjusted. The motion vector length adjustment unit 212 calculates the average motion vector 730 for each pixel. The motion vector length adjustment unit 212 adjusts the average length of motion vector 730 according to the time difference between the lit-exposure frame image 710 to be processed and the unlit-exposure frame image 720 used for processing, and corrects each pixel other than the specific pixel in the unlit-exposure frame image 720 used for processing based on the average motion vector 730 of each pixel after adjustment. Correction of each pixel other than the specific pixel can be performed by creating a vector mask in which motion vectors 730 whose motion vector length is equal to or less than a predetermined threshold are set to 0 and motion vectors 730 other than those whose length is equal to or less than the predetermined threshold are set to 1, and using the vector mask. In other words, only pixels with a value of "1" in the vector mask can be corrected based on the average motion vector 730 of each pixel after adjustment. The predetermined threshold value can be set to an appropriate value through experiments from the viewpoint of the image quality of the processed lit exposure frame image 710 and reducing the amount of calculation.

[0051] FIG. 7 is an explanatory diagram illustrating an example of the calculation of a movement vector 730, the adjustment of the length of the movement vector 730, the calculation of a virtual unlit exposure frame image 721, and the process of subtracting the virtual unlit exposure frame image 721 from the lit exposure frame image 710.

[0052] The first frame image ((1) in FIG. 7), the second frame image ((2) in FIG. 7), and the third frame image ((3) in FIG. 7) are a lit-exposed frame image 710, a lights-out exposed frame image 720, and a lit-exposed frame image 710, respectively. These frame images 700 are captured in the order of the first frame image, the second frame image, and the third frame image. The first frame image, the second frame image, and the third frame image constitute the first lit-exposed frame image, the first lights-out exposed frame image, and the second lit-exposed frame image, respectively. The first frame image is the lit-exposed frame image 710 that is the target of processing. The second frame image is the lights-out exposed frame image 720 used in processing. The first and third frame images are the two frame images 700 used in calculating a movement vector 730. The second frame image used in processing is interpolated into the first and third frame images that are used in calculating the movement vector 730. In FIG. 7, in the diagram of the second frame image (FIG. 7(2)) and the third frame image (FIG. 7(3)), the outline of subject 500 in the first frame image is shown by a broken line.

[0053] A movement vector 730 is calculated for each pixel from the first and third frame images ((4) in FIG. 7). The movement vector 730 is indicated by an arrow in (4) in FIG.

[0054] A vector mask is created in which movement vectors 730 whose length is less than a predetermined threshold are set to 0 (the black part in the same figure), and movement vectors 730 other than those whose length is less than a predetermined threshold are set to 1 (the white part in the same figure) ((5) in Figure 7).

[0055] The calculated movement vectors 730 are averaged, and the average is adjusted according to the ratio of the time difference between the first and third frame images to the time difference between the first and second frame images.

[0056] Using a vector mask, each pixel of the second frame image other than the specific pixel is corrected by shifting it in the opposite direction to the average of the adjusted movement vector 731 by the average length of the adjusted movement vector 731, thereby calculating a virtual lights-out exposure frame image 721 ((6) in Figure 7).

[0057] In order to reduce the influence of edge noise due to movement errors in the shifting of each pixel when calculating the virtual unlit exposure frame image 721, a filter may be applied to the virtual unlit exposure frame image 721 ((7) in FIG. 7).

[0058] The virtual unlit exposure frame image 721 is subtracted from the first frame image ((8) in FIG. 7).

[0059] 8 is a flowchart showing the operation of the imaging device 1. This flowchart can be executed by the cooperative operation of the control unit 130 of the imaging unit 100 and the control unit 210 of the image processing device 200 in accordance with a program.

[0060] The control unit 130 causes the camera 120 to capture a first frame image, a second frame image, and a third frame image in synchronization with the turning on, turning off, and turning on of the light emitting unit 110 (S201). The control unit 130 then transmits the first frame image, the second frame image, and the third frame image to the image processing device 200 via the communication unit 140.

[0061] Control unit 210 receives the first, second, and third frame images via communication unit 230, and calculates motion vectors 730 for each pixel from the first and third frame images (S202). Control unit 210 also calculates the average of motion vectors 730.

[0062] Control unit 210 creates a vector mask based on the length of motion vector 730 for each pixel (S203). That is, control unit 210 creates a vector mask in which motion vectors 730 of specific pixels whose lengths are equal to or less than a predetermined threshold are set to 0, and motion vectors 730 other than those whose lengths are equal to or less than the predetermined threshold are set to 1.

[0063] Control unit 210 adjusts the average length of movement vector 730 according to the ratio of the time difference between the first and third frame images to the time difference between the first and second frame images (S204).

[0064] The control unit 210 uses a vector mask to shift each pixel of the second frame image other than the specific pixel by the average length of the length-adjusted movement vector 731 in the opposite direction to the average of the movement vector 730, thereby calculating a virtual unlit exposure frame image 721 (S205).

[0065] The control unit 210 executes a process of subtracting the virtual unlit exposure frame image 721 from the first frame image (S206).

[0066] According to this embodiment, all pixels other than the specific pixel are uniformly shifted based on the average length of the motion vector 730, thereby effectively reducing the amount of calculation required for processing.

[0067] (Third embodiment) The third embodiment will be described. The present embodiment differs from the first embodiment in the following respects: In the first embodiment, movement vectors 730 are calculated for all pixels. On the other hand, in this embodiment, a face area 715 (see FIG. 9) is detected from a lit-on exposure frame image 710 used to calculate movement vectors 730, and movement vectors 730 are calculated only for each pixel in the detected face area 715. In other respects, this embodiment is similar to the first embodiment, so duplicated explanations will be omitted or simplified.

[0068] 2 detects a face area 715 from a lit exposure frame image 710, which is used to calculate a movement vector 730. The movement vector calculation unit 211 calculates a movement vector 730 for each pixel in the detected face area 715.

[0069] FIG. 9 is an explanatory diagram illustrating an example of the calculation of the movement vector 730, the adjustment of the length of the movement vector 730, the calculation of the virtual unlit exposure frame image 721, and the process of subtracting the virtual unlit exposure frame image 721 from the lit exposure frame image 710.

[0070] The first frame image ((1) in FIG. 9), the second frame image ((2) in FIG. 9), and the third frame image ((3) in FIG. 9) are a lit-exposed frame image 710, a lights-out exposed frame image 720, and a lit-exposed frame image 710, respectively. These frame images 700 are captured in the order of the first frame image, the second frame image, and the third frame image. The first frame image, the second frame image, and the third frame image constitute the first lit-exposed frame image, the first lights-out exposed frame image, and the second lit-exposed frame image, respectively. The first frame image is the lit-exposed frame image 710 that is the target of processing. The second frame image is the lights-out exposed frame image 720 used in processing. The first frame image and the third frame image are two frame images 700 used in calculating a movement vector 730. The second frame image used in processing is interpolated into the first frame image and the third frame image that are used in calculating the movement vector 730. In FIG. 9, in the diagram of the second frame image (FIG. 9(2)) and the third frame image (FIG. 9(3)), the outline of subject 500 in the first frame image is shown by a broken line.

[0071] A face region 715, which is a portion including the face of the subject 500, is detected from the first frame image and set as a ROI (Region of Interest) ((4) in FIG. 9). The face region 715 can be detected using a known method. For example, the face region 715 is detected using a neural network model trained to detect the face region 715 from an image.

[0072] A movement vector 730 of the face area 715 is calculated for each pixel from the first and third frame images ((5) in FIG. 9). The movement vector 730 is indicated by an arrow in (5) in FIG.

[0073] The length of the calculated movement vector 730 is adjusted according to the ratio of the time difference between the first and third frame images to the time difference between the first and second frame images.

[0074] A virtual lights-out exposure frame image 721 is calculated by correcting each pixel of the face region 715 of the second frame image by shifting it in the opposite direction to the movement vector 730 by the length of the length-adjusted movement vector 731 (FIG. 9(6)). The shift direction and amount of each pixel in this correction are indicated by arrows in the diagram of FIG. 9(6).

[0075] In order to reduce the influence of edge noise due to movement errors in the shifting of each pixel when calculating the virtual unlit exposure frame image 721, a filter may be applied to the virtual unlit exposure frame image 721 ((7) in FIG. 9).

[0076] The virtual unlit exposure frame image 721 is subtracted from the first frame image ((8) in FIG. 9).

[0077] 10 is a flowchart showing the operation of the imaging device 1. This flowchart can be executed by the cooperative operation of the control unit 130 of the imaging unit 100 and the control unit 210 of the image processing device 200 in accordance with a program.

[0078] The control unit 130 causes the camera 120 to capture a first frame image, a second frame image, and a third frame image in synchronization with the turning on, turning off, and turning on of the light emitting unit 110 (S301). The control unit 130 then transmits the first frame image, the second frame image, and the third frame image to the image processing device 200 via the communication unit 140.

[0079] The control unit 210 receives the first, second, and third frame images via the communication unit 230, and detects the face area 715 in the first frame (S302).

[0080] The control unit 210 calculates the movement vector 730 of the face area 715 from the first and third frame images (S303).

[0081] Control unit 210 adjusts the length of movement vector 730 in accordance with the ratio of the time difference between the first and third frame images to the time difference between the first and second frame images (S304).

[0082] Control unit 210 calculates virtual lights-out exposure frame image 721 by shifting each pixel of face region 715 of the second frame image by the length of length-adjusted movement vector 731 in the opposite direction to movement vector 730 (S305). As a result, virtual lights-out exposure frame image 721 is calculated in which each pixel of face region 715 of the second frame image for which movement vector 730 has been calculated is shifted by the length of length-adjusted movement vector 731 in the opposite direction to the movement vector.

[0083] The control unit 210 executes a process of subtracting the virtual unlit exposure frame image 721 from the first frame image (S306).

[0084] According to this embodiment, the amount of calculation can be reduced because the pixels for which the movement vector 730 is calculated and the pixels to be shifted in calculating the virtual lights-out exposure frame image 721 are limited to the face area 715. Furthermore, because the lights-out exposure frame image 720 to be subtracted from the lights-out exposure frame image 710 is corrected by shifting it pixel by pixel, it is possible to obtain an image without artifacts, with the influence of background light removed, regardless of the direction in which the subject 500 moves.

[0085] (Fourth embodiment) The fourth embodiment will be described. The present embodiment differs from the first embodiment in the following respects. In the first embodiment, the virtual lights-out exposure frame image 721 is filtered using one low-pass filter. On the other hand, in the present embodiment, the virtual lights-out exposure frame image 721 is filtered using multiple types of low-pass filters, and processing is performed using the filtered virtual lights-out exposure frame image 721 that has the smallest standard deviation of the difference from the virtual lights-out exposure frame image 721 before filtering. In all other respects, this embodiment is similar to the first embodiment, so duplicated explanations will be omitted or simplified.

[0086] 2 calculates a plurality of filtered virtual lights-out exposure frame images 721 by filtering the calculated virtual lights-out exposure frame image 721 using a plurality of types of low-pass filters. Examples of the plurality of types of low-pass filters that can be used include a Gaussian filter, a median filter, and a bilateral filter. Low-pass filters with different kernel sizes may also be used as the plurality of types of low-pass filters.

[0087] The lit-on exposure frame image processing unit 215 identifies the filtered virtual off-exposure frame image 721 that has the smallest standard deviation of the difference between the calculated filtered virtual off-exposure frame image 721 and the unfiltered virtual off-exposure frame image 721. The lit-on exposure frame image processing unit 215 performs processing to subtract the identified virtual off-exposure frame image 721 from the lit-on exposure frame image 710 to be processed.

[0088] FIG. 11 is an explanatory diagram illustrating an example of the calculation of the movement vector 730, the adjustment of the length of the movement vector 730, the calculation of the virtual unlit exposure frame image 721, and the process of subtracting the virtual unlit exposure frame image 721 from the lit exposure frame image 710.

[0089] The first frame image ((1) in FIG. 11), the second frame image ((2) in FIG. 11), and the third frame image ((3) in FIG. 11) are a lit-exposed frame image 710, a lights-out exposed frame image 720, and a lit-exposed frame image 710, respectively. These frame images 700 are captured in the order of the first frame image, the second frame image, and the third frame image. The first frame image, the second frame image, and the third frame image constitute the first lit-exposed frame image, the first lights-out exposed frame image, and the second lit-exposed frame image, respectively. The first frame image is the lit-exposed frame image 710 that is the target of processing. The second frame image is the lights-out exposed frame image 720 used in processing. The first frame image and the third frame image are two frame images 700 used in calculating a movement vector 730. The second frame image used in processing is interpolated into the first frame image and the third frame image that are used in calculating the movement vector 730. In FIG. 11, in the diagram of the second frame image (FIG. 11(2)) and the third frame image (FIG. 11(3)), the outline of subject 500 in the first frame image is shown by a broken line.

[0090] A movement vector 730 is calculated for each pixel from the first and third frame images ((4) in FIG. 11). The movement vector 730 is indicated by an arrow in (4) in FIG.

[0091] The length of the calculated movement vector 730 is adjusted according to the ratio of the time difference between the first and third frame images to the time difference between the first and second frame images.

[0092] A virtual lights-out exposure frame image 721 is calculated by correcting each pixel of the second frame image by shifting it in the opposite direction to the movement vector 730 by the length of the length-adjusted movement vector 731 (FIG. 11 (5)). The shift direction and amount of each pixel in this correction are indicated by arrows in the diagram of FIG. 11 (5).

[0093] The virtual unlights-exposed frame image 721 is filtered using a plurality of types of low-pass filters to calculate each filtered virtual unlights-exposed frame image 721 ((6) in FIG. 11).

[0094] The standard deviation of the difference between the virtual unlights-exposed frame image 721 before and after filtering is calculated, and the virtual unlights-exposed frame image 721 with the smallest calculated standard deviation is selected as the virtual unlights-exposed frame image 721 to be used for processing ((7) in Figure 11).

[0095] The selected virtual unlit exposure frame image 721 is subtracted from the first frame image ((8) in FIG. 11).

[0096] 12 is a flowchart showing the operation of the imaging device 1. This flowchart can be executed by the cooperative operation of the control unit 130 of the imaging unit 100 and the control unit 210 of the image processing device 200 in accordance with a program.

[0097] Steps S401 to S404 are the same as steps S101 to S104 in the first embodiment, and therefore a description thereof will be omitted.

[0098] The control unit 210 filters the calculated virtual lights-out exposure frame image 721 using a plurality of low-pass filters, and calculates the difference between the pre-filtered and post-filtered virtual lights-out exposure frame image 721. The control unit 210 selects the virtual lights-out exposure frame image 721 with the smallest standard deviation of the difference as the virtual lights-out exposure frame image 721 to be used for processing (S405).

[0099] The control unit 210 executes a process of subtracting the selected virtual unlit exposure frame image 721 from the first frame image (S406).

[0100] According to this embodiment, by performing processing using an optimally filtered virtual off-lights exposure frame image 721, edge noise caused by errors in the pixel shift amount when calculating the virtual off-lights exposure frame image 721 can be further reduced.

[0101] (Fifth embodiment) The fifth embodiment will be described. The present embodiment differs from the first embodiment in the following respects. In the first embodiment, of the two lit-exposure frame images 710 used to calculate the movement vector 730, the lit-exposure frame image 710 captured earlier is the target of processing. On the other hand, in the present embodiment, of the two lit-exposure frame images 710 used to calculate the movement vector 730, the lit-exposure frame image 710 captured later is the target of processing. In all other respects, this embodiment is similar to the first embodiment, so duplicated explanations will be omitted or simplified.

[0102] FIG. 13 is an explanatory diagram illustrating an example of the calculation of the movement vector 730, the adjustment of the length of the movement vector 730, the calculation of the virtual unlighted exposure frame image 721, and the process of subtracting the virtual unlighted exposure frame image 721 from the lit exposure frame image 710.

[0103] The first frame image ((1) in FIG. 13), the second frame image ((2) in FIG. 13), and the third frame image ((3) in FIG. 13) are a lit-exposed frame image 710, a lights-out exposed frame image 720, and a lit-exposed frame image 710, respectively. These frame images 700 are captured in the order of the first frame image, the second frame image, and the third frame image. The first frame image, the second frame image, and the third frame image constitute the first lit-exposed frame image, the first lights-out exposed frame image, and the second lit-exposed frame image, respectively. The third frame image is the lit-exposed frame image 710 that is the target of processing. The second frame image is the lights-out exposed frame image 720 used in processing. The first frame image and the third frame image are two frame images 700 used in calculating a movement vector 730. The second frame image used in processing is interpolated into the first frame image and the third frame image used in calculating the movement vector 730. In FIG. 13, in the diagram of the second frame image (FIG. 13(2)) and the third frame image (FIG. 13(3)), the outline of subject 500 in the first frame image is shown by a broken line.

[0104] A movement vector 730 is calculated for each pixel from the first and third frame images ((4) in FIG. 13). The movement vector 730 is indicated by an arrow in (4) in FIG.

[0105] The length of the calculated movement vector 730 is adjusted according to the ratio of the time difference between the first and third frame images to the time difference between the second and third frame images.

[0106] A virtual lights-out exposure frame image 721 is calculated by correcting each pixel of the second frame image by shifting it in the same direction as the movement vector 730 by the length of the length-adjusted movement vector 731 (FIG. 13 (5)). The shift direction and amount of each pixel in this correction are indicated by arrows in the diagram of FIG. 13 (5).

[0107] In order to reduce the influence of edge noise due to movement errors in the shifting of each pixel when calculating the virtual unlit exposure frame image 721, a filter may be applied to the virtual unlit exposure frame image 721 ((6) in FIG. 13).

[0108] The virtual unlit exposure frame image 721 is subtracted from the third frame image ((7) in FIG. 13).

[0109] 14 is a flowchart showing the operation of the imaging device 1. This flowchart can be executed by the cooperative operation of the control unit 130 of the imaging unit 100 and the control unit 210 of the image processing device 200 in accordance with a program.

[0110] The control unit 130 causes the camera 120 to capture a first frame image, a second frame image, and a third frame image in synchronization with the turning on, turning off, and turning on of the light emitting unit 110 (S501). The control unit 130 then transmits the first frame image, the second frame image, and the third frame image to the image processing device 200 via the communication unit 140.

[0111] Control unit 210 receives the first, second, and third frame images via communication unit 230, and calculates movement vector 730 from the first and third frame images (S502).

[0112] Control unit 210 adjusts the length of movement vector 730 in accordance with the ratio of the time difference between the first and third frame images to the time difference between the second and third frame images (S503).

[0113] Control unit 210 calculates virtual lights-out exposure frame image 721 by shifting each pixel of the second frame image by the length of length-adjusted movement vector 731 in the same direction as movement vector 730 (S504).

[0114] The control unit 210 executes a process of subtracting the virtual unlit exposure frame image 721 from the third frame image (S505).

[0115] According to this embodiment, the lights-out exposure frame image 720 to be subtracted from the lights-out exposure frame image 710 is corrected by shifting it pixel by pixel, so that an image without artifacts and from which the influence of background light is removed can be obtained no matter in which direction the subject 500 moves or the camera 120 moves. Furthermore, in creating the virtual lights-out exposure frame image 721, this embodiment shifts each pixel of the lights-out exposure frame image 720 captured earlier to advance the exposure time of the lights-out exposure frame image 710 captured later. This embodiment can be suitably employed when the lights-out exposure frame image 710 to be processed is captured after the lights-out exposure frame image 720 used in processing and the capture times of the two are relatively close to each other. On the other hand, in the first embodiment, in creating the virtual lights-out exposure frame image 721, each pixel of the lights-out exposure frame image 720 is shifted in order to return the lights-out exposure frame image 720 captured later to the exposure time of the lights-out exposure frame image 710 captured earlier. The first embodiment can be suitably employed in cases where the lights-out exposure frame image 710 to be processed was captured before the lights-out exposure frame image 720 to be used for processing, and the capture times of the two are relatively close to each other.

[0116] (Sixth embodiment) The sixth embodiment will be described. The present embodiment differs from the first embodiment in the following respects. In the first embodiment, two lit-on exposure frame images 710 are used to calculate the movement vector 730. On the other hand, in the present embodiment, two lights-out exposure frame images 720 are used to calculate the movement vector 730. In other respects, this embodiment is similar to the first embodiment, so duplicated explanations will be omitted or simplified.

[0117] FIG. 15 is an explanatory diagram illustrating an example of the calculation of the movement vector 730, the adjustment of the length of the movement vector 730, the calculation of the virtual unlighted exposure frame image 721, and the process of subtracting the virtual unlighted exposure frame image 721 from the lit exposure frame image 710.

[0118] The first frame image ((1) in FIG. 15), the second frame image ((2) in FIG. 15), and the third frame image ((3) in FIG. 15) are a lights-out exposure frame image 720, a lights-out exposure frame image 710, and a lights-out exposure frame image 720, respectively. These frame images 700 are captured in the order of the first frame image, the second frame image, and the third frame image. The first frame image, the second frame image, and the third frame image constitute a first lights-out exposure frame image, a first lights-out exposure frame image, and a second lights-out exposure frame image, respectively. The second frame image is the lights-out exposure frame image 710 that is the target of processing. The first frame image is the lights-out exposure frame image 720 that is used for processing. The first frame image and the third frame image are two frame images 700 that are used for calculating a movement vector 730. The second frame image that is the target of processing is interpolated into the first frame image and the third frame image that are used for calculating the movement vector 730. In FIG. 15, in the diagram of the second frame image (FIG. 15(2)) and the third frame image (FIG. 15(3)), the outline of subject 500 in the first frame image is shown by a broken line.

[0119] A movement vector 730 is calculated for each pixel from the first and third frame images ((4) in FIG. 15). The movement vector 730 is indicated by an arrow in (4) in FIG.

[0120] The length of the calculated movement vector 730 is adjusted for each pixel according to the ratio of the time difference between the first and third frame images to the time difference between the first and second frame images.

[0121] A virtual lights-out exposure frame image 721 is calculated by correcting each pixel of the first frame image by shifting it in the same direction as the movement vector 730 by the length of the length-adjusted movement vector 731 (FIG. 15 (5)). The shift direction and amount of each pixel in this correction are indicated by arrows in the diagram of FIG. 15 (5).

[0122] In order to reduce the influence of edge noise due to movement errors in the shifting of each pixel when calculating the virtual unlit exposure frame image 721, a filter may be applied to the virtual unlit exposure frame image 721 ((6) in FIG. 15).

[0123] The virtual unlit exposure frame image 721 is subtracted from the second frame image ((7) in FIG. 15).

[0124] 16 is a flowchart showing the operation of the imaging device 1. This flowchart can be executed by the cooperative operation of the control unit 130 of the imaging unit 100 and the control unit 210 of the image processing device 200 in accordance with a program.

[0125] The control unit 130 causes the camera 120 to capture a first frame image, a second frame image, and a third frame image in synchronization with the turning on, turning off, and turning on of the light emitting unit 110 (S601). The control unit 130 then transmits the first frame image, the second frame image, and the third frame image to the image processing device 200 via the communication unit 140.

[0126] Control unit 210 receives the first, second, and third frame images via communication unit 230, and calculates movement vector 730 from the first and third frame images (S602).

[0127] Control unit 210 adjusts the length of movement vector 730 in accordance with the ratio of the time difference between the first and third frame images to the time difference between the first and second frame images (S603).

[0128] Control unit 210 calculates virtual lights-out exposure frame image 721 by shifting each pixel of the first frame image by the length of length-adjusted movement vector 731 in the same direction as movement vector 730 (S604).

[0129] The control unit 210 executes a process of subtracting the virtual unlit exposure frame image 721 from the second frame image (S605).

[0130] According to this embodiment, when the background light is strong, the lit-on exposure frame image 710 is close to the saturation level of the camera 120, and it is difficult to obtain a difference in contrast of the feature amounts of the lit-on exposure frame image 710, the accuracy of the movement vector 730 can be improved by calculating the movement vector 730 using two lights-out exposure frame images 720.

[0131] Seventh embodiment The seventh embodiment will now be described. The present embodiment differs from the sixth embodiment in the following respects. In the sixth embodiment, of the two lights-out exposure frame images 720 used to calculate the movement vector 730, the virtual lights-out exposure frame image 721 is created by shifting each pixel of the lights-out exposure frame image 720 that was captured earlier. On the other hand, in this embodiment, the virtual lights-out exposure frame image 721 is created by shifting each pixel of the lights-out exposure frame image 720 that was captured later. In all other respects, this embodiment is similar to the sixth embodiment, so duplicated explanations will be omitted or simplified.

[0132] FIG. 17 is an explanatory diagram illustrating an example of the calculation of the movement vector 730, the adjustment of the length of the movement vector 730, the calculation of the virtual unlighted exposure frame image 721, and the process of subtracting the virtual unlighted exposure frame image 721 from the lit exposure frame image 710.

[0133] The first frame image ((1) in FIG. 17), the second frame image ((2) in FIG. 17), and the third frame image ((3) in FIG. 17) are a lights-out exposure frame image 720, a lights-out exposure frame image 710, and a lights-out exposure frame image 720, respectively. These frame images 700 are captured in the order of the first frame image, the second frame image, and the third frame image. The first frame image, the second frame image, and the third frame image constitute a first lights-out exposure frame image, a first lights-out exposure frame image, and a second lights-out exposure frame image, respectively. The second frame image is the lights-out exposure frame image 710 that is the target of processing. The third frame image is the lights-out exposure frame image 720 that is used for processing. The first frame image and the third frame image are two frame images 700 that are used for calculating a movement vector 730. The second frame image that is the target of processing is interpolated into the first frame image and the third frame image that are used for calculating the movement vector 730. In FIG. 17, in the diagram of the second frame image (FIG. 17(2)) and the third frame image (FIG. 17(3)), the outline of subject 500 in the first frame image is shown by a broken line.

[0134] A movement vector 730 is calculated for each pixel from the first and third frame images ((4) in FIG. 17). The movement vector 730 is indicated by an arrow in (4) in FIG.

[0135] The length of the calculated movement vector 730 is adjusted for each pixel according to the ratio of the time difference between the first and third frame images to the time difference between the second and third frame images.

[0136] A virtual lights-out exposure frame image 721 is calculated by correcting each pixel of the third frame image by shifting it in the opposite direction to the movement vector 730 by the length of the length-adjusted movement vector 731 (FIG. 17(5)). The shift direction and amount of each pixel in this correction are indicated by arrows in the diagram of FIG. 17(5).

[0137] In order to reduce the influence of edge noise due to movement errors in the shifting of each pixel when calculating the virtual unlit exposure frame image 721, a filter may be applied to the virtual unlit exposure frame image 721 ((6) in FIG. 17).

[0138] The virtual unlit exposure frame image 721 is subtracted from the second frame image ((7) in FIG. 17).

[0139] 18 is a flowchart showing the operation of the imaging device 1. This flowchart can be executed by the cooperative operation of the control unit 130 of the imaging unit 100 and the control unit 210 of the image processing device 200 in accordance with a program.

[0140] The control unit 130 causes the camera 120 to capture a first frame image, a second frame image, and a third frame image in synchronization with the turning on, turning off, and turning on of the light emitting unit 110 (S701). The control unit 130 then transmits the first frame image, the second frame image, and the third frame image to the image processing device 200 via the communication unit 140.

[0141] Control unit 210 receives the first, second, and third frame images via communication unit 230, and calculates movement vector 730 from the first and third frame images (S702).

[0142] Control unit 210 adjusts the length of movement vector 730 in accordance with the ratio of the time difference between the first and third frame images to the time difference between the second and third frame images (S703).

[0143] Control unit 210 calculates virtual lights-out exposure frame image 721 by shifting each pixel of the third frame image by the length of length-adjusted movement vector 731 in the opposite direction to movement vector 730 (S704).

[0144] The control unit 210 executes a process of subtracting the virtual unlit exposure frame image 721 from the second frame image (S705).

[0145] According to this embodiment, in creating the virtual off-time exposure frame image 721, each pixel of the off-time exposure frame image 720 is shifted to return the later captured lights-out exposure frame image 720 to the exposure time of the earlier captured lights-out exposure frame image 710. This embodiment can be suitably employed when the lights-out exposure frame image 710 to be processed was captured before the lights-out exposure frame image 720 used for processing, and the capture times of the two images are relatively close. On the other hand, in the sixth embodiment, in creating the virtual off-time exposure frame image 721, each pixel of the earlier captured lights-out exposure frame image 720 is shifted to advance the exposure time of the later captured lights-out exposure frame image 710. This sixth embodiment can be suitably employed when the lights-out exposure frame image 710 to be processed was captured after the lights-out exposure frame image 720 used for processing, and the capture times of the two images are relatively close. Either the sixth or seventh embodiment can be flexibly adopted depending on the imaging situation, imaging control method, and the like.

[0146] (Eighth embodiment) An eighth embodiment will be described. The present embodiment differs from the first embodiment in the following respects. In the first embodiment, a lit-time exposure frame image 710, an unlit-time exposure frame image 720, and a lit-time exposure frame image 710 are captured in this order, and a virtual unlit-time exposure frame image 721 is created by shifting each pixel of the unlit-time exposure frame image 720 that is interpolated into the two lit-time exposure frame images 710 used to calculate the movement vector 730. On the other hand, in this embodiment, a lit-time exposure frame image 710, an unlit-time exposure frame image 710, and a light-exposed frame image 720 are captured in this order, and a virtual unlit-time exposure frame image 721 is created by shifting each pixel of the unlit-time exposure frame image 720 that is extrapolated into the two lit-time exposure frame images 710 used to calculate the movement vector 730. In other respects, this embodiment is similar to the first embodiment, and therefore, duplicated explanations will be omitted or simplified.

[0147] 19 is an explanatory diagram illustrating an example of two lit-on exposure frame images 710 used to calculate a movement vector 730 and an off-lights exposure frame image 720 used in processing. The illumination light control signal is a control signal that causes the control unit 130 to turn the light-emitting unit 110 on and off. The image capture element frame signal is a control signal that causes the control unit 130 to cause the camera 120 to perform exposure and readout from each element (pixel). In this embodiment, after two lit-on exposure frame images 710 are captured in succession, two off-lights exposure frame images 720 can be captured in succession.

[0148] The first, second, and third frame images are frame images 700 that are captured consecutively in time. In Fig. 19, the first and second frame images are lit-exposure frame images 710. The third frame image is an unlit-exposure frame image 720. The second frame image is the lit-exposure frame image 710 that is the target of processing, the third frame image is the unlit-exposure frame image used for processing, and the first and second frame images can be the two frame images 700 that are used to calculate a movement vector 730.

[0149] FIG. 20 is an explanatory diagram illustrating an example of the calculation of the movement vector 730, the adjustment of the length of the movement vector 730, the calculation of the virtual unlit exposure frame image 721, and the process of subtracting the virtual unlit exposure frame image 721 from the lit exposure frame image 710.

[0150] The first frame image ((1) in FIG. 20), the second frame image ((2) in FIG. 20), and the third frame image ((3) in FIG. 20) are a lit-exposed frame image 710, a lit-exposed frame image 710, and a lights-out exposed frame image 720, respectively. These frame images 700 are captured in the order of the first frame image, the second frame image, and the third frame image. The first frame image, the second frame image, and the third frame image constitute a first lit-exposed frame image, a second lit-exposed frame image, and a first lights-out exposed frame image, respectively. The second frame image is the lit-exposed frame image 710 that is the target of processing. The third frame image is the lights-out exposed frame image 720 used in processing. The first and second frame images are two frame images 700 used in calculating a movement vector 730. The third frame image used in processing is extrapolated to the first and second frame images used in calculating the movement vector 730. In FIG. 20, in the diagram of the second frame image (FIG. 20(2)) and the third frame image (FIG. 20(3)), the outline of subject 500 in the first frame image is shown by a broken line.

[0151] A movement vector 730 is calculated for each pixel from the first and second frame images ((4) in FIG. 20). The movement vector 730 is indicated by an arrow in (4) in FIG.

[0152] The length of the calculated movement vector 730 is adjusted for each pixel according to the ratio of the time difference between the first and second frame images and the time difference between the second and third frame images.

[0153] A virtual lights-out exposure frame image 721 is calculated by correcting each pixel of the third frame image by shifting it in the opposite direction to the movement vector 730 by the length of the length-adjusted movement vector 731 (FIG. 20 (5)). The shift direction and amount of each pixel in this correction are indicated by arrows in the diagram of FIG. 20 (5).

[0154] In order to reduce the influence of edge noise due to movement errors in the shifting of each pixel when calculating the virtual unlit exposure frame image 721, a filter may be applied to the virtual unlit exposure frame image 721 ((6) in FIG. 20).

[0155] The virtual unlit exposure frame image 721 is subtracted from the second frame image ((7) in FIG. 20).

[0156] 21 is a flowchart showing the operation of the imaging device 1. This flowchart can be executed by the cooperative operation of the control unit 130 of the imaging unit 100 and the control unit 210 of the image processing device 200 in accordance with a program.

[0157] The control unit 130 causes the camera 120 to capture a first frame image, a second frame image, and a third frame image in synchronization with the turning on, turning on, and turning off of the light emitting unit 110 (S801). The control unit 130 then transmits the first frame image, the second frame image, and the third frame image to the image processing device 200 via the communication unit 140.

[0158] Control unit 210 receives the first, second, and third frame images via communication unit 230, and calculates movement vector 730 from the first and second frame images (S802).

[0159] Control unit 210 adjusts the length of movement vector 730 in accordance with the ratio of the time difference between the first and second frame images to the time difference between the second and third frame images (S803).

[0160] Control unit 210 calculates virtual lights-out exposure frame image 721 by shifting each pixel of the third frame image by the length of length-adjusted movement vector 731 in the opposite direction to movement vector 730 (S804).

[0161] The control unit 210 executes a process of subtracting the virtual unlit exposure frame image 721 from the second frame image (S805).

[0162] According to this embodiment, by alternately repeating two consecutive exposures when the light-emitting unit 110 is on and two consecutive exposures when it is off, it is possible to properly calculate a virtual off-lights exposure frame image 721 even if an off-lights exposure frame image 720 is not captured between the capture of two on-lights exposure frame images 710.

[0163] (Ninth embodiment) The ninth embodiment will be described. The present embodiment differs from the eighth embodiment in the following respects. In the eighth embodiment, the lights-on exposure frame image 710, the lights-on exposure frame image 710, and the lights-off exposure frame image 720 are captured in this order, and a virtual lights-off exposure frame image 721 is created by shifting each pixel of the lights-off exposure frame image 720 that has been extrapolated to the two lights-on exposure frame images 710 used to calculate the movement vector 730 in the opposite direction to the movement vector 730. On the other hand, in the present embodiment, the lights-off exposure frame image 720, the lights-on exposure frame image 710, and the lights-on exposure frame image 710 are captured in this order, and a virtual lights-off exposure frame image 721 is created by shifting each pixel of the lights-off exposure frame image 720 that has been extrapolated to the two lights-on exposure frame images 710 used to calculate the movement vector 730 in the same direction as the movement vector 730. In all other respects, this embodiment is similar to the first embodiment, and therefore, redundant description will be omitted or simplified.

[0164] FIG. 22 is an explanatory diagram illustrating an example of the calculation of the movement vector 730, the adjustment of the length of the movement vector 730, the calculation of the virtual unlit exposure frame image 721, and the process of subtracting the virtual unlit exposure frame image 721 from the lit exposure frame image 710.

[0165] The first frame image ((1) in FIG. 22), the second frame image ((2) in FIG. 22), and the third frame image ((3) in FIG. 22) are the lights-out exposure frame image 720, the lights-out exposure frame image 710, and the lights-out exposure frame image 710, respectively. These frame images 700 are captured in the order of the first frame image, the second frame image, and the third frame image. The first frame image, the second frame image, and the third frame image constitute the first lights-out exposure frame image, the first lights-out exposure frame image, and the second lights-out exposure frame image, respectively. The second frame image is the lights-out exposure frame image 710 that is the target of processing. The first frame image is the lights-out exposure frame image 720 used in processing. The second and third frame images are the two frame images 700 used to calculate the movement vector 730. The first frame image used in processing is extrapolated to the second and third frame images used to calculate the movement vector 730. In FIG. 22, in the diagram of the second frame image (FIG. 22(2)) and the third frame image (FIG. 22(3)), the outline of subject 500 in the first frame image is shown by a broken line.

[0166] A movement vector 730 is calculated for each pixel from the second and third frame images ((4) in FIG. 22). The movement vector 730 is indicated by an arrow in (4) in FIG.

[0167] The length of the calculated movement vector 730 is adjusted according to the ratio of the time difference between the second and third frame images to the time difference between the first and second frame images.

[0168] A virtual lights-out exposure frame image 721 is calculated by correcting each pixel of the first frame image by shifting it in the same direction as the movement vector 730 by the length of the length-adjusted movement vector 731 (FIG. 22 (5)). The shift direction and amount of each pixel in this correction are indicated by arrows in the diagram of FIG. 22 (5).

[0169] In order to reduce the influence of edge noise due to movement errors in the shifting of each pixel when calculating the virtual unlit exposure frame image 721, a filter may be applied to the virtual unlit exposure frame image 721 ((6) in FIG. 22).

[0170] The virtual unlit exposure frame image 721 is subtracted from the second frame image ((7) in FIG. 22).

[0171] 23 is a flowchart showing the operation of the imaging device 1. This flowchart can be executed by the cooperative operation of the control unit 130 of the imaging unit 100 and the control unit 210 of the image processing device 200 in accordance with a program.

[0172] The control unit 130 causes the camera 120 to capture a first frame image, a second frame image, and a third frame image in synchronization with the turning-off, turning-on, and turning-on of the light emitting unit 110 (S901). The control unit 130 then transmits the first frame image, the second frame image, and the third frame image to the image processing device 200 via the communication unit 140.

[0173] Control unit 210 receives the first, second, and third frame images via communication unit 230, and calculates movement vector 730 from the second and third frame images (S902).

[0174] Control unit 210 adjusts the length of movement vector 730 in accordance with the ratio of the time difference between the second and third frame images to the time difference between the first and second frame images (S903).

[0175] Control unit 210 calculates virtual lights-out exposure frame image 721 by shifting each pixel of the first frame image by the length of length-adjusted movement vector 731 in the same direction as movement vector 730 (S904).

[0176] The control unit 210 executes a process of subtracting the virtual unlit exposure frame image 721 from the second frame image (S905).

[0177] As with the eighth embodiment, this embodiment alternately repeats two consecutive exposures when the light-emitting unit 110 is on and two consecutive exposures when it is off, so that even if a lights-out exposure frame image 720 is not captured between the capture of two lights-out exposure frame images 710, it is possible to properly calculate a virtual lights-out exposure frame image 721. Furthermore, the processing method can be optimized by selecting this embodiment or the eighth embodiment depending on the movement trajectory of the subject 500.

[0178] (Tenth embodiment) A tenth embodiment will be described. The present embodiment differs from the eighth embodiment in the following points. In the eighth embodiment, the lights-on exposure frame image 710, the lights-on exposure frame image 710, and the lights-off exposure frame image 720 are captured in this order, and a virtual lights-off exposure frame image 721 is created by shifting each pixel of the lights-off exposure frame image 720, which is extrapolated to the two lights-on exposure frame images 710 used to calculate the movement vector 730, in the opposite direction to the movement vector 730. On the other hand, in the present embodiment, the lights-off exposure frame image 720, the lights-off exposure frame image 720, and the lights-on exposure frame image 710 are captured in this order, and a virtual lights-off exposure frame image 721 is created by shifting each pixel of the lights-off exposure frame image 720 in the same direction as the movement vector 730, which is used to process the lights-on exposure frame image 710, which is extrapolated to the two lights-off exposure frame images 720 used to calculate the movement vector 730. In other respects, this embodiment is similar to the eighth embodiment, so duplicated explanations will be omitted or simplified.

[0179] FIG. 24 is an explanatory diagram illustrating an example of the calculation of the movement vector 730, the adjustment of the length of the movement vector 730, the calculation of the virtual unlit exposure frame image 721, and the process of subtracting the virtual unlit exposure frame image 721 from the lit exposure frame image 710.

[0180] The first frame image ((1) in FIG. 24), the second frame image ((2) in FIG. 24), and the third frame image ((3) in FIG. 24) are the lights-out exposure frame image 720, the lights-out exposure frame image 720, and the lights-on exposure frame image 710, respectively. These frame images 700 are captured in the order of the first frame image, the second frame image, and the third frame image. The first frame image, the second frame image, and the third frame image constitute the first lights-out exposure frame image, the second lights-out exposure frame image, and the first lights-on exposure frame image, respectively. The third frame image is the lights-on exposure frame image 710 to be processed. The second frame image is the lights-out exposure frame image 720 used in processing. The first and second frame images are the two frame images 700 used to calculate a movement vector 730. The third frame image to be processed is extrapolated to the first and second frame images used to calculate the movement vector 730. In FIG. 24, in the diagram of the second frame image (FIG. 24(2)) and the third frame image (FIG. 24(3)), the outline of subject 500 in the first frame image is shown by a broken line.

[0181] A movement vector 730 is calculated for each pixel from the first and second frame images ((4) in FIG. 24). The movement vector 730 is indicated by an arrow in (4) in FIG.

[0182] The length of the calculated movement vector 730 is adjusted according to the ratio of the time difference between the first and second frame images to the time difference between the second and third frame images.

[0183] A virtual lights-out exposure frame image 721 is calculated by correcting each pixel of the second frame image by shifting it in the same direction as the movement vector 730 by the length of the length-adjusted movement vector 731 (FIG. 24(5)). The shift direction and amount of each pixel in this correction are indicated by arrows in the diagram of FIG. 24(5).

[0184] In order to reduce the influence of edge noise due to movement errors in the shifting of each pixel when calculating the virtual unlit exposure frame image 721, a filter may be applied to the virtual unlit exposure frame image 721 ((6) in FIG. 24).

[0185] The virtual unlit exposure frame image 721 is subtracted from the third frame image ((7) in FIG. 24).

[0186] 25 is a flowchart showing the operation of the imaging device 1. This flowchart can be executed by the cooperative operation of the control unit 130 of the imaging unit 100 and the control unit 210 of the image processing device 200 in accordance with a program.

[0187] The control unit 130 causes the camera 120 to capture a first frame image, a second frame image, and a third frame image in synchronization with the turning on and off of the light emitting unit 110 (S1001). The control unit 130 then transmits the first frame image, the second frame image, and the third frame image to the image processing device 200 via the communication unit 140.

[0188] Control unit 210 receives the first, second, and third frame images via communication unit 230, and calculates movement vector 730 from the first frame images (S1002).

[0189] Control unit 210 adjusts the length of movement vector 730 in accordance with the ratio of the time difference between the first and second frame images to the time difference between the second and third frame images (S1003).

[0190] Control unit 210 calculates virtual lights-out exposure frame image 721 by shifting each pixel of the second frame image by the length of length-adjusted movement vector 731 in the same direction as movement vector 730 (S1004).

[0191] The control unit 210 executes a process of subtracting the virtual unlit exposure frame image 721 from the third frame image (S1005).

[0192] According to this embodiment, even when multiple consecutive exposures when the light-emitting unit 110 is on and multiple consecutive exposures when it is off are repeated alternately, the background light is strong, the frame image 710 exposed when the light is on is close to the saturation level of the camera 120, and it is difficult to obtain a difference in contrast of the feature amount of the frame image 710 exposed when the light is on, the accuracy of the movement vector 730 can be improved by calculating the movement vector 730 using two frame images 720 exposed when the light is off.

[0193] (Eleventh embodiment) An eleventh embodiment will now be described. The differences between this embodiment and the tenth embodiment are as follows: In the tenth embodiment, a lights-out exposure frame image 720, a lights-out exposure frame image 720, and a lights-on exposure frame image 710 are captured in this order, and a virtual lights-out exposure frame image 721 is created by shifting each pixel of the lights-out exposure frame image 720 used in processing the lights-on exposure frame image 710 that has been extrapolated to the two lights-out exposure frame images 720 used to calculate the movement vector 730 in the same direction as the movement vector 730. On the other hand, in this embodiment, a lit-time exposure frame image 710, a lights-out exposure frame image 720, and a lights-out exposure frame image 720 are captured in this order, and a virtual lights-out exposure frame image 721 is used to process the lit-time exposure frame image 710 that is extrapolated to the two lights-out exposure frame images 720 used to calculate the movement vector 730, and the virtual lights-out exposure frame image 721 is created by shifting each pixel of the lights-out exposure frame image 720 in the opposite direction to the movement vector 730. In other respects, this embodiment is similar to the tenth embodiment, and therefore redundant explanations will be omitted or simplified.

[0194] FIG. 26 is an explanatory diagram illustrating an example of the calculation of the movement vector 730, the adjustment of the length of the movement vector 730, the calculation of the virtual unlighted exposure frame image 721, and the process of subtracting the virtual unlighted exposure frame image 721 from the lit exposure frame image 710.

[0195] The first frame image ((1) in FIG. 26), the second frame image ((2) in FIG. 26), and the third frame image ((3) in FIG. 26) are a lit-exposed frame image 710, a lights-out exposed frame image 720, and a lights-out exposed frame image 720, respectively. These frame images 700 are captured in the order of the first frame image, the second frame image, and the third frame image. The first frame image, the second frame image, and the third frame image constitute the first lit-exposed frame image, the first lights-out exposed frame image, and the second lights-out exposed frame image, respectively. The first frame image is the lit-exposed frame image 710 that is the target of processing. The second frame image is the lights-out exposed frame image 720 that is used for processing. The second and third frame images are the two frame images 700 that are used to calculate a movement vector 730. The first frame image that is the target of processing is extrapolated to the second and third frame images that are used to calculate the movement vector 730. In FIG. 26, in the diagram of the second frame image (FIG. 26(2)) and the third frame image (FIG. 26(3)), the outline of subject 500 in the first frame image is shown by a broken line.

[0196] A movement vector 730 is calculated for each pixel from the second and third frame images ((4) in FIG. 26). The movement vector 730 is indicated by an arrow in (4) in FIG.

[0197] The length of the calculated movement vector 730 is adjusted for each pixel according to the ratio of the time difference between the second and third frame images to the time difference between the first and second frame images.

[0198] A virtual lights-out exposure frame image 721 is calculated by correcting each pixel of the second frame image by shifting it in the opposite direction to the movement vector 730 by the length of the length-adjusted movement vector 731 (FIG. 26 (5)). The shift direction and amount of each pixel in this correction are indicated by arrows in the diagram of FIG. 26 (5).

[0199] In order to reduce the influence of edge noise due to movement errors in the shifting of each pixel when calculating the virtual unlit exposure frame image 721, a filter can be applied to the virtual unlit exposure frame image 721 ((6) in FIG. 26).

[0200] The virtual unlit exposure frame image 721 is subtracted from the first frame image ((7) in FIG. 26).

[0201] 27 is a flowchart showing the operation of the imaging device 1. This flowchart can be executed by the cooperative operation of the control unit 130 of the imaging unit 100 and the control unit 210 of the image processing device 200 in accordance with a program.

[0202] The control unit 130 causes the camera 120 to capture a first frame image, a second frame image, and a third frame image in synchronization with the turning on and off of the light emitting unit 110 (S1101). The control unit 130 then transmits the first frame image, the second frame image, and the third frame image to the image processing device 200 via the communication unit 140.

[0203] Control unit 210 receives the first, second, and third frame images via communication unit 230, and calculates movement vector 730 from the second and third frame images (S1102).

[0204] Control unit 210 adjusts the length of movement vector 730 in accordance with the ratio of the time difference between the second and third frame images to the time difference between the first and second frame images (S1103).

[0205] Control unit 210 calculates virtual lights-out exposure frame image 721 by shifting each pixel of the second frame image by the length of length-adjusted vector 731 in the direction opposite to movement vector 730 (S1104).

[0206] The control unit 210 executes a process of subtracting the virtual unlit exposure frame image 721 from the first frame image (S1105).

[0207] According to this embodiment, as in the tenth embodiment, even when multiple consecutive exposures while the light-emitting unit 110 is on and multiple consecutive exposures while it is off are repeated alternately, the background light is strong, the lit-on exposure frame image 710 is close to the saturation level of the camera 120, and it is difficult to obtain a difference in contrast of the feature amount of the lit-on exposure frame image 710, the accuracy of the movement vector 730 can be improved by calculating the movement vector 730 using two lights-out exposure frame images 720. Furthermore, the processing method can be optimized by selecting this embodiment or the tenth embodiment depending on the movement trajectory of the subject 500.

[0208] (Twelfth embodiment) The twelfth embodiment will be described. The present embodiment differs from the first embodiment in the following respects. In the first embodiment, each frame image (first to third frame images) is captured by a single exposure. On the other hand, in this embodiment, each frame image 700 (first to third frame images) is captured by adding multiple frame images captured by multiple exposures to calculate one frame image 700. In other respects, this embodiment is similar to the first embodiment, so duplicated explanations will be omitted or simplified.

[0209] FIG. 28 is an explanatory diagram for explaining capturing a plurality of frame images through a plurality of exposures, and capturing a lit-on exposure frame image 710 and an unlit-on exposure frame image 720 by adding together the plurality of frame images.

[0210] The camera 120 captures a lit-on exposure frame image 710 by adding digital signals of multiple frame images 700 captured by turning on the light-emitting unit 110 and performing exposure synchronized with the lighting multiple times in succession. The camera 120 captures a lights-out exposure frame image 720 by adding digital signals of multiple frame images captured by turning off the light-emitting unit 110 and performing exposure synchronized with the lighting multiple times in succession. That is, the camera 120 captures a first frame image as an added frame image 700 by adding together multiple frame images captured with exposure synchronized with the lighting of the light-emitting unit 110. Next, the camera 120 captures a second frame image as an added frame image 700 by adding together multiple frame images captured with exposure synchronized with the lighting of the light-emitting unit 110. Next, the camera 120 captures a third frame image as an added frame image by adding together multiple frame images 700 captured with exposure synchronized with the lighting of the light-emitting unit 110.

[0211] In this embodiment, the exposure time is set to a relatively short time that does not cause pixel saturation of the image sensor, and can be appropriately set through experiments from the viewpoint of avoiding pixel saturation of the image sensor.

[0212] The addition of the plurality of frame images 700 described above is performed outside the image sensor. This addition may be performed by the camera 120 equipped with a CPU or the like, by the control unit 130 of the image capturing unit 100, or by the control unit 210 of the image processing device 200.

[0213] According to the twelfth embodiment, even when the background light is relatively strong, it is possible to capture a lit-on exposure frame image 710 and an unlit-on exposure frame image 720 in which pixels are not saturated.

[0214] (Thirteenth embodiment) The thirteenth embodiment will be described. The present embodiment differs from the twelfth embodiment in the following respects. In the twelfth embodiment, each frame image 700 (first to third frame images) is captured by adding together multiple frame images captured by multiple exposures to calculate one frame image 700. On the other hand, in this embodiment, each frame image 700 (first to third frame images) is captured by accumulating the signal amount for each pixel by exposure and integrating the accumulated signal amount multiple times in succession to calculate one frame image. In all other respects, this embodiment is similar to the twelfth embodiment, so duplicated explanations will be omitted or simplified.

[0215] Figure 29 is an explanatory diagram for explaining the capturing of a lit-on exposure frame image 710 by accumulating the signal amount from multiple exposures when the light is on, and the capturing of a lights-off exposure frame image 720 by accumulating the signal amount from multiple exposures when the light is off.

[0216] The camera 120 captures a lit-on exposure frame image 710 by turning on the light-emitting unit 110, accumulating the signal amount for each pixel through exposure synchronized with the lighting, and integrating the accumulated signal amount multiple times in succession. The camera 120 captures a lights-off exposure frame image 720 by turning off the light-emitting unit 110, accumulating the signal amount for each pixel through exposure synchronized with the lighting, and integrating the accumulated signal amount multiple times in succession. That is, the camera 120 captures a first frame image by integrating the signal amount accumulated through multiple exposures synchronized with the lighting of the light-emitting unit 110. Next, the camera 120 captures a second frame image by integrating the signal amount accumulated through multiple exposures synchronized with the lighting of the light-emitting unit 110. Next, the camera 120 captures a third frame image by integrating the signal amount accumulated through multiple exposures synchronized with the lighting of the light-emitting unit 110.

[0217] The signal amount can be integrated by storing the signal amount generated by multiple exposures in multiple capacitors and adding the charge amounts stored in the capacitors in an analog manner. The signal amount addition can be performed in the image sensor.

[0218] According to the thirteenth embodiment, even when the background light is relatively strong, it is possible to capture a lit-on exposure frame image 710 and an unlit-on exposure frame image 720 in which pixels are not saturated.

[0219] (Variation 1) FIG. 30 is a diagram showing a first example of a case where the first to third frame images are not consecutive.

[0220] In the first embodiment, the first to third frame images are consecutive, but as in this modified example, the first to third frame images do not have to be consecutive.

[0221] In this modified example, consecutive frames (1) and (2) are designated as the first and second frame images, respectively. Frame (5) is designated as the third frame image. In this way, when the first to third frame images are not consecutive, it is preferable to designate frame (2) as the lights-out exposure frame image 720 used for processing, which is closest to frame (1), the lit exposure frame image 710 to be processed.

[0222] This modification can be applied to the first to fourth embodiments.

[0223] When this modification is applied to the fifth embodiment, it is preferable to use frame (4) closest to the frame (5) to be processed as the second frame image used for processing, which is the lights-out exposure frame image 720. This modification can also be applied to the sixth and seventh embodiments using the same concept.

[0224] (Variation 2) FIG. 31 is a diagram showing a second example in which the first to third frame images are not consecutive.

[0225] In this modified example, when multiple consecutive exposures with the lights on and multiple consecutive exposures with the lights off are performed alternately, non-consecutive frame images are selected as the first to third frame images. Specifically, consecutive frames (2) and (3) are designated as the first and second frame images, respectively. Frame (5) is designated as the third frame image. In this way, when the first to third frame images are not consecutive, it is preferable to select frame (3), which is closest to frame (2), the lit-exposed frame image 710 to be processed, as the lights-off exposed frame image 720 to be used for processing.

[0226] This modification can be applied to the first to fourth embodiments.

[0227] When this modification is applied to the fifth embodiment, it is preferable to use frame (4) closest to the frame (5) to be processed as the second frame image, which is the lights-out exposure frame image 720 used for processing. This modification can also be applied to the sixth and seventh embodiments in a similar way.

[0228] (Variation 3) FIG. 32 is a diagram showing a third example in which the first to third frame images are not consecutive.

[0229] In this modified example, when lights-on exposure and lights-off exposure are performed alternately, non-consecutive frame images are selected as the first to third frame images. Specifically, frame (1) is set as the first frame image. The consecutive frames (3) and (4) are set as the second and third frame images, respectively. In this way, when the first to third frame images are not consecutive, it is preferable to select frame (2) or frame (4), which is closest to frame (3), the lights-on exposure frame image 710 to be processed, as the lights-off exposure frame image 720 to be used for processing. If the lights-off exposure frame image 720 of frame (2) cannot be used due to some error, frame (4) can be selected as the lights-off exposure frame image 720 to be used for processing.

[0230] This modified example can be applied to the eighth embodiment. Based on the same concept as above, this modified example can also be applied to the ninth to eleventh embodiments.

[0231] (Variation 4) FIG. 33 is a diagram showing a modified example in which the first to third frame images are consecutive.

[0232] In this modification, when multiple consecutive exposures with the light on and multiple consecutive exposures with the light off are performed alternately, consecutive frame images are selected as the first to third frame images. Specifically, consecutive frames (1) to (3) are designated as the first to third frame images, respectively. In this way, even when multiple consecutive exposures with the light on and multiple consecutive exposures with the light off are performed alternately, selecting consecutive frame images 700 as the first to third frame images enables external light correction processing to be performed in a short time.

[0233] This modification can be applied to the eighth to eleventh embodiments.

[0234] The embodiment has the following advantages.

[0235] The system calculates a motion vector for each pixel from two lights-on exposure frame images or two lights-off exposure frame images, and adjusts the length of the motion vector according to the time difference between the lights-on exposure frame image to be processed and the lights-off exposure frame image to be used for the processing.The system then corrects the lights-off exposure frame image to be used for the processing based on the adjusted motion vector and subtracts it from the lights-on exposure frame image to be processed.This makes it possible to obtain high-resolution images without artifacts, with the effects of background light removed, regardless of the direction in which the subject moves.

[0236] Furthermore, when the frame images captured in the order of the lit-time exposure frame image, the unlit-time exposure frame image, and the lit-time exposure frame image are designated the first lit-time exposure frame image, the first unlit-time exposure frame image, and the second lit-time exposure frame image, respectively, the movement vector is calculated from the first lit-time exposure frame image and the second lit-time exposure frame image. The length of the movement vector for each pixel is adjusted according to the ratio of the time difference between the first lit-time exposure frame image and the second lit-time exposure frame image to the time difference between the first lit-time exposure frame image and the first unlit-time exposure frame image. A virtual unlit-time exposure frame image is calculated by correcting the first unlit-time exposure frame image by shifting each pixel in the first unlit-time exposure frame image by the length of the adjusted movement vector in the opposite direction to the movement vector. The calculated virtual unlit-time exposure frame image is then subtracted from the first lit-time exposure frame image. This allows for correction by shifting the light-off exposure frame image, which is subtracted from the light-on exposure frame image, in pixel units, so that no matter which direction the subject moves or the camera moves, an image without artifacts can be obtained with the effects of background light removed.

[0237] Furthermore, when the frame images captured in the order of the lit-time exposure frame image, the unlit-time exposure frame image, and the lit-time exposure frame image are designated the first lit-time exposure frame image, the first unlit-time exposure frame image, and the second lit-time exposure frame image, respectively, the movement vector of each pixel is calculated from the first lit-time exposure frame image and the second lit-time exposure frame image. The length of the movement vector of each pixel is adjusted according to the ratio of the time difference between the first lit-time exposure frame image and the second lit-time exposure frame image to the time difference between the first unlit-time exposure frame image and the second lit-time exposure frame image. A virtual unlit-time exposure frame image is calculated by correcting the first unlit-time exposure frame image by shifting each pixel in the same direction as the movement vector by the length of the adjusted movement vector. Then, the above process is performed to subtract the calculated virtual unlit-time exposure frame image from the second lit-time exposure frame image. This allows for correction by shifting the light-off exposure frame image, which is subtracted from the light-on exposure frame image, in pixel units, so that no matter which direction the subject moves or the camera moves, an image without artifacts can be obtained with the effects of background light removed.

[0238] Furthermore, when the frame images captured in this order of lights-out exposure frame image, lights-on exposure frame image, and lights-out exposure frame image are designated the first lights-out exposure frame image, the first lights-on exposure frame image, and the second lights-out exposure frame image, respectively, a movement vector is calculated from the first lights-out exposure frame image and the second lights-out exposure frame image. The length of the movement vector for each pixel is adjusted according to the ratio between the time difference between the first lights-out exposure frame image and the second lights-out exposure frame image and the time difference between the first lights-out exposure frame image and the first lights-on exposure frame image. A virtual lights-out exposure frame image is calculated by correcting each pixel of the first lights-out exposure frame image by shifting it in the same direction as the movement vector by the length of the adjusted movement vector. The lights-on exposure frame image processing unit then performs the above process of subtracting the calculated virtual lights-out exposure frame image from the first lights-on exposure frame image. As a result, even when the background light is strong, the lit-exposed frame image is close to the camera's saturation level, and it is difficult to obtain a difference in contrast between the feature quantities of the lit-exposed frame image, the accuracy of the movement vector can be improved by calculating the movement vector using two lights-out exposed frame images. Furthermore, when the lit-exposed frame image to be processed is captured after the lights-out frame image used for processing and the capture times of the two are relatively close, it is possible to more simply and effectively obtain an image free of artifacts from which the effects of background light have been removed.

[0239] Furthermore, when the frame images captured in this order of lights-out exposure frame image, lights-on exposure frame image, and lights-out exposure frame image are designated the first lights-out exposure frame image, the first lights-on exposure frame image, and the second lights-out exposure frame image, respectively, a movement vector is calculated from the first lights-out exposure frame image and the second lights-out exposure frame image. The length of the movement vector for each pixel is adjusted according to the ratio between the time difference between the first lights-out exposure frame image and the second lights-out exposure frame image and the time difference between the first lights-on exposure frame image and the second lights-out exposure frame image. A virtual lights-out exposure frame image is calculated by correcting each pixel of the second lights-out exposure frame image by shifting it in the opposite direction to the movement vector by the length of the movement vector after the length adjustment. Then, the above process is performed to subtract the calculated virtual lights-out exposure frame image from the first lights-on exposure frame image. This makes it possible to more easily and effectively obtain an image free of artifacts and free of the influence of background light, when the light-exposed frame image to be processed is captured before the light-off frame image to be used for processing and the capture times of the two are relatively close to each other.

[0240] Furthermore, pixels whose motion vector length is equal to or less than a predetermined threshold are designated as specific pixels whose motion vector length is not adjusted. The average of the motion vectors for each pixel is calculated, and the average length of the motion vector is adjusted according to the time difference between the lit-exposure frame image to be processed and the unlit-exposure frame image used for processing. The motion vectors of each pixel other than the specific pixels are then adjusted to the adjusted average. This allows all pixels other than the specific pixels to be uniformly shifted based on the average motion vector length, effectively reducing the amount of calculation required for processing.

[0241] Furthermore, a face area is detected from the lit-exposure frame image or the two unlit-exposure frame images used to calculate the movement vector. Then, the movement vector of each pixel in the detected face area is calculated. This limits the pixels for which movement vectors are calculated and the pixels to be shifted in calculating the virtual unlit-exposure frame image to the face area, thereby reducing the amount of calculations.

[0242] Furthermore, the calculated virtual lights-out exposure frame image is filtered using a plurality of types of low-pass filters to calculate a plurality of filtered virtual lights-out exposure frame images. The filtered virtual lights-out exposure frame image having the smallest standard deviation of the difference between the calculated filtered virtual lights-out exposure frame image and the unfiltered virtual lights-out exposure frame image is subtracted from the lit exposure frame image to be processed. In this way, by processing using the optimally filtered virtual lights-out exposure frame image, edge noise due to errors in the pixel shift amount when calculating the virtual lights-out exposure frame image can be further reduced.

[0243] Furthermore, when the frame images captured in the order of the lit-time exposure frame image, lit-time exposure frame image, and unlit-time exposure frame image are designated the first lit-time exposure frame image, the second lit-time exposure frame image, and the first unlit-time exposure frame image, respectively, a movement vector is calculated from the first lit-time exposure frame image and the second lit-time exposure frame image. The length of the movement vector for each pixel is adjusted according to the ratio of the time difference between the first lit-time exposure frame image and the second lit-time exposure frame image to the time difference between the second unlit-time exposure frame image and the first unlit-time exposure frame image. A virtual unlit-time exposure frame image is calculated by correcting the pixels of the first unlit-time exposure frame image by shifting them in the opposite direction to the movement vector by the length of the adjusted movement vector. The above process is then performed to subtract the calculated virtual unlit-time exposure frame image from the second unlit-time exposure frame image. This allows for an appropriate calculation of a virtual off-lights exposure frame image even in cases where the off-lights exposure frame image 720 is not captured between the capture of two on-lights exposure frame images, for example, due to alternating two consecutive exposures when the light-emitting unit is on and two consecutive exposures when it is off.

[0244] Furthermore, when the frame images captured in the order of lights-out exposure frame image, lights-on exposure frame image, and lights-on exposure frame image are designated as the first lights-out exposure frame image, the first lights-on exposure frame image, and the second lights-on exposure frame image, respectively, a movement vector is calculated from the first lights-on exposure frame image and the second lights-on exposure frame image. The length of the movement vector for each pixel is adjusted according to the ratio of the time difference between the first lights-on exposure frame image and the second lights-on exposure frame image to the time difference between the first lights-out exposure frame image and the first lights-on exposure frame image. A virtual lights-out exposure frame image is calculated by correcting each pixel of the first lights-out exposure frame image by the length of the adjusted movement vector in the same direction as the movement vector. Then, the calculated virtual lights-out exposure frame image is subtracted from the first lights-on exposure frame image. This allows for the virtual off-lights exposure frame image to be calculated appropriately even in cases where an off-lights exposure frame image is not captured between the capture of two on-lights exposure frame images, for example, because two consecutive exposures when the light-emitting unit is on and two consecutive exposures when the light-emitting unit is off are alternately repeated.

[0245] Furthermore, when the frame images captured in the order of lights-out exposure frame image, lights-out exposure frame image, and lights-on exposure frame image are designated as the first lights-out exposure frame image, the second lights-out exposure frame image, and the first lights-on exposure frame image, respectively, a movement vector is calculated from the first lights-out exposure frame image and the second lights-out exposure frame image. The length of the movement vector for each pixel is adjusted according to the ratio of the time difference between the first lights-out exposure frame image and the second lights-out exposure frame image to the time difference between the second lights-out exposure frame image and the first lights-on exposure frame image. A virtual lights-out exposure frame image is calculated by correcting each pixel of the second lights-out exposure frame image by shifting it in the same direction as the movement vector by the length of the adjusted movement vector. Then, the above process is performed to subtract the calculated virtual lights-out exposure frame image from the first lights-on exposure frame image. As a result, even when multiple consecutive exposures when the light-emitting unit is on and multiple consecutive exposures when it is off are repeated alternately, the background light is strong, the frame images exposed when the light is on are close to the saturation level of the camera, and it is difficult to obtain a difference in contrast between the feature amounts of the frame images exposed when the light is on, the accuracy of the movement vector can be improved by calculating the movement vector using two frame images exposed when the light is off 720.

[0246] Furthermore, when the frame images captured in the order of the lit-time exposure frame image, the unlit-time exposure frame image, and the unlit-time exposure frame image are designated the first lit-time exposure frame image, the first unlit-time exposure frame image, and the second unlit-time exposure frame image, respectively, a movement vector is calculated from the first unlit-time exposure frame image and the second unlit-time exposure frame image. The length of the movement vector for each pixel is adjusted according to the ratio of the time difference between the first unlit-time exposure frame image and the second unlit-time exposure frame image to the time difference between the first unlit-time exposure frame image and the first unlit-time exposure frame image. A virtual unlit-time exposure frame image is calculated by correcting the pixels of the first unlit-time exposure frame image by shifting them in the opposite direction to the movement vector by the length of the adjusted movement vector. Then, the above process is performed to subtract the calculated virtual unlit-time exposure frame image from the first unlit-time exposure frame image. This allows the accuracy of the movement vector to be improved by calculating it using two light-exposed frame images when the light is off, even when multiple consecutive exposures when the light-emitting unit is on are repeated alternately, the background light is strong, the light-exposed frame images are close to the saturation level of the camera, and it is difficult to obtain a difference in the contrast of the features of the light-exposed frame images.

[0247] Furthermore, a lit-time exposure frame image is captured by adding together the digital signals of multiple frame images captured by successively turning on the light-emitting unit and performing exposure synchronized with the lighting a plurality of times. A lights-out exposure frame image is then captured by adding together the digital signals of multiple frame images captured by successively turning off the light-emitting unit and performing exposure synchronized with the lighting a plurality of times. This makes it possible to capture a lit-time exposure frame image and a lights-out exposure frame image that are not pixel-saturated, even when the background light is relatively strong.

[0248] Furthermore, a lit-time exposure frame image is captured by turning on the light-emitting unit, accumulating the signal amount for each pixel through exposure synchronized with the lighting, and integrating the accumulated signal amount multiple times in succession. Then, a lights-out exposure frame image is captured by turning off the light-emitting unit, accumulating the signal amount for each pixel through exposure synchronized with the lighting, and integrating the accumulated signal amount multiple times in succession. This makes it possible to capture a lit-time exposure frame image and a lights-out exposure frame image that are not pixel-saturated, even when the background light is relatively strong.

[0249] The above-described embodiments are merely examples, and those skilled in the art will recognize that various modifications and equivalent embodiments are possible. Therefore, the disclosed embodiments should be considered from an illustrative perspective, not a restrictive one. The scope of the invention is defined in the claims, and all structures within the scope of the claims should be construed as being within the scope of the invention. [Explanation of symbols]

[0250] 1 imaging device, 100 imaging unit, 110 light-emitting part, 120 cameras, 130 control section, 140 Communications Department, 200 image processing device, 210 control section, 211 movement vector calculation unit, 212 movement vector length adjustment unit; 213 Lights-out exposure frame image correction unit, 214 low-pass filter section, 215 lighting exposure frame image processing unit; 500 subjects, 700 frame images, 710 lit exposure frame image, 720 Lights-out exposure frame image, 721 Virtual light-off exposure frame images, 730 movement vectors, 731The translation vector after length adjustment.

Claims

1. an imaging unit that captures a light-on exposure frame image exposed when the light is on and a light-off exposure frame image exposed when the light is off by performing exposure in synchronization with the turning on and turning off of a light-emitting unit that irradiates a subject with light; a motion vector calculation unit that calculates a motion vector for each pixel from the two lit-on exposure frame images or the two unlit-on exposure frame images; a motion vector length adjustment unit that adjusts the length of the motion vector of each pixel according to the time difference between the light-exposed frame image to be processed and the light-exposed frame image used in the processing; a lights-out exposure frame image correction unit that calculates a virtual lights-out exposure frame image that is estimated to be captured if the light-emitting unit is turned off when the lit-out exposure frame image that is the target of the processing is exposed by correcting each pixel of the lights-out exposure frame image used in the processing based on the movement vector of each pixel after adjustment; a lit-on exposure frame image processing section that performs the process of subtracting the calculated virtual lights-out exposure frame image from the lit-on exposure frame image that is the target of the process.

2. the movement vector calculation unit calculates the movement vector from the first lit-time exposure frame image and the second lit-time exposure frame image, when the frame images captured in this order of the lit-time exposure frame image, the unlit-time exposure frame image, and the lit-time exposure frame image are respectively defined as a first lit-time exposure frame image, a first unlit-time exposure frame image, and a second lit-time exposure frame image; the motion vector length adjustment unit adjusts the length of the motion vector of each pixel in accordance with a ratio of a time difference between the first lit exposure frame image and the second lit exposure frame image to a time difference between the first lit exposure frame image and the first unlit exposure frame image; the lights-out exposure frame image correction unit calculates the virtual lights-out exposure frame image by correcting each pixel of the first lights-out exposure frame image by shifting the pixel in a direction opposite to the movement vector by the length of the movement vector after the length adjustment; The imaging device according to claim 1 , wherein the lit-on exposure frame image processing unit performs the process of subtracting the calculated virtual unlit-on exposure frame image from the first lit-on exposure frame image.

3. the movement vector calculation unit calculates the movement vector of each pixel from the first lit-time exposure frame image and the second lit-time exposure frame image, when the frame images captured in this order of the lit-time exposure frame image, the unlit-time exposure frame image, and the lit-time exposure frame image are respectively designated as a first lit-time exposure frame image, a first unlit-time exposure frame image, and a second lit-time exposure frame image; the motion vector length adjustment unit adjusts the length of the motion vector of each pixel in accordance with a ratio of a time difference between the first lit-on exposure frame image and the second lit-on exposure frame image to a time difference between the first unlit-on exposure frame image and the second lit-on exposure frame image; the lights-out exposure frame image correction unit calculates the virtual lights-out exposure frame image by correcting each pixel of the first lights-out exposure frame image by shifting the pixel in the same direction as the movement vector by the length of the movement vector after the length adjustment; The imaging device according to claim 1 , wherein the lit-on exposure frame image processing unit performs the process of subtracting the calculated virtual unlit-on exposure frame image from the second lit-on exposure frame image.

4. the movement vector calculation unit calculates the movement vector from the first lights-out exposure frame image and the second lights-out exposure frame image, when the frame images captured in this order of the lights-out exposure frame image, the lights-on exposure frame image, and the lights-out exposure frame image are respectively defined as a first lights-out exposure frame image, a first lights-on exposure frame image, and a second lights-out exposure frame image; the motion vector length adjustment unit adjusts the length of the motion vector of each pixel in accordance with a ratio of a time difference between the first lights-out exposure frame image and the second lights-out exposure frame image to a time difference between the first lights-out exposure frame image and the first lights-on exposure frame image; the lights-out exposure frame image correction unit calculates the virtual lights-out exposure frame image by correcting each pixel of the first lights-out exposure frame image by shifting the pixel in the same direction as the movement vector by the length of the movement vector after the length adjustment; The imaging device according to claim 1 , wherein the lit-on exposure frame image processing unit performs the process of subtracting the calculated virtual unlit-on exposure frame image from the first lit-on exposure frame image.

5. the movement vector calculation unit calculates the movement vector from the first lights-out exposure frame image and the second lights-out exposure frame image, when the frame images captured in this order of the lights-out exposure frame image, the lights-on exposure frame image, and the lights-out exposure frame image are respectively defined as a first lights-out exposure frame image, a first lights-on exposure frame image, and a second lights-out exposure frame image; the motion vector length adjustment unit adjusts the length of the motion vector of each pixel in accordance with a ratio of a time difference between the first lights-out exposure frame image and the second lights-out exposure frame image to a time difference between the first lights-out exposure frame image and the second lights-out exposure frame image; the lights-out exposure frame image correction unit calculates the virtual lights-out exposure frame image by correcting each pixel of the second lights-out exposure frame image by shifting the pixel in a direction opposite to the movement vector by the length of the movement vector after the length adjustment; The imaging device according to claim 1 , wherein the lit-on exposure frame image processing unit performs the process of subtracting the calculated virtual unlit-on exposure frame image from the first lit-on exposure frame image.

6. The movement vector length adjustment unit A pixel whose length of the motion vector is equal to or less than a predetermined threshold is set as a specific pixel whose length of the motion vector is not adjusted; 2. The imaging device according to claim 1, wherein an average of the motion vectors of each pixel is calculated, the length of the average motion vector is adjusted according to the time difference between the lit-exposed frame image to be processed and the unlit-exposed frame image to be used in the processing, and each pixel other than the specific pixel of the unlit-exposed frame image to be used in the processing is corrected based on the adjusted average.

7. The movement vector calculation unit detecting a face area from the light-exposed frame image or the two light-exposed frame images used to calculate the movement vector; The imaging device according to claim 1 , further comprising: a processor configured to calculate the motion vector of each pixel in the detected face area.

8. a low-pass filter unit that calculates a plurality of filtered virtual unlights-exposed frame images by filtering the calculated virtual unlights-exposed frame images using a plurality of types of low-pass filters, The lighting exposure frame image processing unit 2. The imaging device of claim 1, wherein the process subtracts the filtered virtual off-exposure frame image having the smallest standard deviation of the difference between the filtered virtual off-exposure frame image and the unfiltered virtual off-exposure frame image from the lit-on exposure frame image being processed.

9. the movement vector calculation unit calculates the movement vector from the first lit-time exposure frame image and the second lit-time exposure frame image, when the frame images captured in this order of the lit-time exposure frame image, the lit-time exposure frame image, and the unlit-time exposure frame image are respectively defined as a first lit-time exposure frame image, a second lit-time exposure frame image, and a first unlit-time exposure frame image; the motion vector length adjustment unit adjusts the length of the motion vector of each pixel in accordance with a ratio of a time difference between the first lit exposure frame image and the second lit exposure frame image to a time difference between the second lit exposure frame image and the first unlit exposure frame image; the lights-out exposure frame image correction unit calculates the virtual lights-out exposure frame image by correcting each pixel of the first lights-out exposure frame image by shifting the pixel in a direction opposite to the movement vector by the length of the movement vector after the length adjustment; The imaging device according to claim 1 , wherein the lit-on exposure frame image processing unit performs the process of subtracting the calculated virtual unlit-on exposure frame image from the second lit-on exposure frame image.

10. the movement vector calculation unit calculates the movement vector from the first lit-time exposure frame image and the second lit-time exposure frame image, when the frame images captured in this order of the lights-out exposure frame image, the lights-on exposure frame image, and the lights-on exposure frame image are respectively designated as a first lights-out exposure frame image, a first lights-on exposure frame image, and a second lights-on exposure frame image; the motion vector length adjustment unit adjusts the length of the motion vector of each pixel in accordance with a ratio of a time difference between the first lit-on exposure frame image and the second lit-on exposure frame image to a time difference between the first unlit-on exposure frame image and the first lit-on exposure frame image; the lights-out exposure frame image correction unit calculates the virtual lights-out exposure frame image by correcting each pixel of the first lights-out exposure frame image by shifting the pixel in the same direction as the movement vector by the length of the movement vector after the length adjustment; The imaging device according to claim 1 , wherein the lit-on exposure frame image processing unit performs the process of subtracting the calculated virtual unlit-on exposure frame image from the first lit-on exposure frame image.

11. the movement vector calculation unit calculates the movement vector from the first lights-out exposure frame image and the second lights-out exposure frame image, when the frame images captured in this order of the lights-out exposure frame image, the lights-out exposure frame image, and the lights-on exposure frame image are respectively designated as a first lights-out exposure frame image, a second lights-out exposure frame image, and a first lights-on exposure frame image; the motion vector length adjustment unit adjusts the length of the motion vector of each pixel in accordance with a ratio of a time difference between the first lights-out exposure frame image and the second lights-out exposure frame image to a time difference between the second lights-out exposure frame image and the first lights-on exposure frame image; the lights-out exposure frame image correction unit calculates the virtual lights-out exposure frame image by correcting each pixel of the second lights-out exposure frame image by shifting the pixel in the same direction as the movement vector by the length of the movement vector after the length adjustment; The imaging device according to claim 1 , wherein the lit-on exposure frame image processing unit performs the process of subtracting the calculated virtual unlit-on exposure frame image from the first lit-on exposure frame image.

12. the movement vector calculation unit calculates the movement vector from the first lights-out exposure frame image and the second lights-out exposure frame image, when the frame images captured in this order of the lights-out exposure frame image, the lights-out exposure frame image, and the lights-out exposure frame image are respectively designated as a first lights-out exposure frame image, a first lights-out exposure frame image, and a second lights-out exposure frame image; the motion vector length adjustment unit adjusts the length of the motion vector of each pixel in accordance with a ratio of a time difference between the first lights-out exposure frame image and the second lights-out exposure frame image to a time difference between the first lights-out exposure frame image and the first lights-out exposure frame image; the lights-out exposure frame image correction unit calculates the virtual lights-out exposure frame image by correcting each pixel of the first lights-out exposure frame image by shifting the pixel in a direction opposite to the movement vector by the length of the movement vector after the length adjustment; The imaging device according to claim 1 , wherein the lit-on exposure frame image processing unit performs the process of subtracting the calculated virtual unlit-on exposure frame image from the first lit-on exposure frame image.

13. 2. The imaging device of claim 1, wherein the imaging unit captures the light-on exposure frame image by adding together digital signals of a plurality of frame images captured by successively turning on the light-emitting unit and performing exposure synchronized with the lighting multiple times, and captures the light-off exposure frame image by adding together digital signals of a plurality of frame images captured by successively turning off the light-emitting unit and performing exposure synchronized with the lighting multiple times.

14. 2. The imaging device of claim 1, wherein the imaging unit captures the light-on exposure frame image by turning on the light-emitting unit, accumulating the signal amount for each pixel by exposure synchronized with the lighting, and integrating the accumulated signal amount multiple times in succession, and captures the light-off exposure frame image by turning off the light-emitting unit, accumulating the signal amount for each pixel by exposure synchronized with the lighting, and integrating the accumulated signal amount multiple times in succession.

15. an imaging step of capturing a light-on exposure frame image exposed when the light is on and a light-off exposure frame image exposed when the light is off by performing exposure in synchronization with the turning on and turning off of a light-emitting unit that irradiates light onto a subject; a motion vector calculation step of calculating a motion vector for each pixel from the two lit-on exposure frame images or the two unlit-on exposure frame images; a motion vector length adjustment step of adjusting the length of the motion vector for each pixel in accordance with the time difference between the light-exposed frame image to be processed and the light-exposed frame image used in the processing; a lights-out exposure frame image correcting step of correcting each pixel of the lights-out exposure frame image used in the processing based on the movement vector of each pixel after adjustment, thereby calculating a virtual lights-out exposure frame image that is estimated to be captured if the light-emitting unit is turned off when the lit-out exposure frame image that is the target of the processing is exposed; and a lit exposure frame image processing step for performing the process of subtracting the calculated virtual unlit exposure frame image from the lit exposure frame image that is the target of the processing.

16. in the movement vector calculation step, when the frame images captured in the order of the lit-on exposure frame image, the unlit-on exposure frame image, and the lit-on exposure frame image are respectively designated as a first lit-on exposure frame image, a first unlit-on exposure frame image, and a second lit-on exposure frame image, the movement vector is calculated from the first lit-on exposure frame image and the second lit-on exposure frame image; the movement vector length adjustment step adjusts the length of the movement vector for each pixel in accordance with a ratio of a time difference between the first lit-on exposure frame image and the second lit-on exposure frame image to a time difference between the first lit-on exposure frame image and the first unlit-on exposure frame image; in the lights-out exposure frame image correcting step, the virtual lights-out exposure frame image is calculated by correcting each pixel of the first lights-out exposure frame image by shifting the pixel in a direction opposite to the movement vector by the length of the movement vector after length adjustment; 16. The imaging program according to claim 15, wherein the lit-on exposure frame image processing step performs the process of subtracting the calculated virtual unlit-on exposure frame image from the first lit-on exposure frame image.

17. in the movement vector calculation step, when the frame images captured in the order of the lit-on exposure frame image, the unlit-on exposure frame image, and the lit-on exposure frame image are respectively designated as a first lit-on exposure frame image, a first unlit-on exposure frame image, and a second lit-on exposure frame image, the movement vector of each pixel is calculated from the first lit-on exposure frame image and the second lit-on exposure frame image; the movement vector length adjustment step adjusts the length of the movement vector for each pixel in accordance with a ratio of a time difference between the first lit-on exposure frame image and the second lit-on exposure frame image to a time difference between the first unlit-on exposure frame image and the second lit-on exposure frame image; in the lights-out exposure frame image correcting step, the virtual lights-out exposure frame image is calculated by correcting each pixel of the first lights-out exposure frame image by shifting the pixel in the same direction as the movement vector by the length of the movement vector after length adjustment; 16. The imaging program according to claim 15, wherein the lit-on exposure frame image processing step performs the process of subtracting the calculated virtual unlit-on exposure frame image from the second lit-on exposure frame image.

18. the movement vector calculation step calculates the movement vector from the first lights-out exposure frame image and the second lights-out exposure frame image, when the frame images captured in this order of the lights-out exposure frame image, the lights-on exposure frame image, and the lights-out exposure frame image are respectively defined as a first lights-out exposure frame image, a first lights-on exposure frame image, and a second lights-out exposure frame image; the motion vector length adjustment step adjusts the length of the motion vector for each pixel in accordance with a ratio of a time difference between the first lights-out exposure frame image and the second lights-out exposure frame image to a time difference between the first lights-out exposure frame image and the first lights-on exposure frame image; the lights-out exposure frame image correcting step calculates the virtual lights-out exposure frame image by correcting each pixel of the first lights-out exposure frame image by shifting the pixel in the same direction as the movement vector by the length of the movement vector after the length adjustment; 16. The imaging program according to claim 15, wherein the lit-on exposure frame image processing step performs the process of subtracting the calculated virtual unlit-on exposure frame image from the first lit-on exposure frame image.

19. in the movement vector calculation step, when the frame images captured in this order of the lights-out exposure frame image, the lights-on exposure frame image, and the lights-out exposure frame image are respectively designated as a first lights-out exposure frame image, a first lights-on exposure frame image, and a second lights-out exposure frame image, the movement vector is calculated from the first lights-out exposure frame image and the second lights-out exposure frame image; In the movement vector length adjustment step, a length of the movement vector of each pixel is adjusted in accordance with a ratio of a time difference between the first lights-out exposure frame image and the second lights-out exposure frame image to a time difference between the first lights-out exposure frame image and the second lights-out exposure frame image; in the lights-out exposure frame image correcting step, the virtual lights-out exposure frame image is calculated by correcting each pixel of the second lights-out exposure frame image by shifting the pixel in a direction opposite to the movement vector by the length of the movement vector after length adjustment; 16. The imaging program according to claim 15, wherein the lit-on exposure frame image processing step performs the process of subtracting the calculated virtual unlit-on exposure frame image from the first lit-on exposure frame image.

20. In the movement vector length adjustment step, A pixel whose length of the motion vector is equal to or less than a predetermined threshold is set as a specific pixel whose length of the motion vector is not adjusted; 16. The imaging program according to claim 15, wherein an average of the motion vectors of each pixel is calculated, the length of the average motion vector is adjusted according to the time difference between the lit-on exposure frame image to be processed and the unlit-on exposure frame image to be used in the processing, and each pixel other than the specific pixel of the unlit-on exposure frame image to be used in the processing is corrected based on the adjusted average.

21. In the movement vector calculation step, detecting a face area from the light-exposed frame image or the two light-exposed frame images used to calculate the movement vector; The imaging program according to claim 15, further comprising: calculating the movement vector of each pixel in the detected face area.

22. the processing further includes a filtering step of filtering the calculated virtual lights-out exposure frame image using a plurality of types of low-pass filters to calculate a plurality of filtered virtual lights-out exposure frame images; In the lighting exposure frame image processing step, The imaging program according to claim 15, wherein the process subtracts the filtered virtual off-exposure frame image having the smallest standard deviation of the difference between the calculated filtered virtual off-exposure frame image and the unfiltered virtual off-exposure frame image from the lit-on exposure frame image being processed.

23. in the movement vector calculation step, when the frame images captured in the order of the lit-on exposure frame image, the lit-on exposure frame image, and the unlit-on exposure frame image are respectively designated as a first lit-on exposure frame image, a second lit-on exposure frame image, and a first unlit-on exposure frame image, the movement vector is calculated from the first lit-on exposure frame image and the second lit-on exposure frame image; the movement vector length adjustment step adjusts the length of the movement vector for each pixel in accordance with a ratio of a time difference between the first lit-on exposure frame image and the second lit-on exposure frame image to a time difference between the second lit-on exposure frame image and the first unlit-on exposure frame image; in the lights-out exposure frame image correcting step, the virtual lights-out exposure frame image is calculated by correcting each pixel of the first lights-out exposure frame image by shifting the pixel in a direction opposite to the movement vector by the length of the movement vector after length adjustment; 16. The imaging program according to claim 15, wherein the lit-on exposure frame image processing step performs the process of subtracting the calculated virtual unlit-on exposure frame image from the second lit-on exposure frame image.

24. in the movement vector calculation step, when the frame images captured in the order of the lights-out exposure frame image, the lights-on exposure frame image, and the lights-on exposure frame image are respectively designated as a first lights-out exposure frame image, a first lights-on exposure frame image, and a second lights-on exposure frame image, the movement vector is calculated from the first lights-on exposure frame image and the second lights-on exposure frame image; In the movement vector length adjustment step, the length of the movement vector of each pixel is adjusted in accordance with a ratio of a time difference between the first lit-on exposure frame image and the second lit-on exposure frame image to a time difference between the first unlit-on exposure frame image and the first lit-on exposure frame image; in the lights-out exposure frame image correcting step, the virtual lights-out exposure frame image is calculated by correcting each pixel of the first lights-out exposure frame image by shifting the pixel in the same direction as the movement vector by the length of the movement vector after length adjustment; 16. The imaging program according to claim 15, wherein the lit-on exposure frame image processing step performs the process of subtracting the calculated virtual unlit-on exposure frame image from the first lit-on exposure frame image.

25. in the movement vector calculation step, when the frame images captured in the order of the lights-out exposure frame image, the lights-out exposure frame image, and the lights-on exposure frame image are respectively designated as a first lights-out exposure frame image, a second lights-out exposure frame image, and a first lights-on exposure frame image, the movement vector is calculated from the first lights-out exposure frame image and the second lights-on exposure frame image; the movement vector length adjustment step adjusts the length of the movement vector for each pixel in accordance with a ratio of a time difference between the first lights-out exposure frame image and the second lights-out exposure frame image to a time difference between the second lights-out exposure frame image and the first lights-out exposure frame image; in the lights-out exposure frame image correcting step, the virtual lights-out exposure frame image is calculated by correcting each pixel of the second lights-out exposure frame image by shifting the pixel in the same direction as the movement vector by the length of the movement vector after the length adjustment; 16. The imaging program according to claim 15, wherein the lit-on exposure frame image processing step performs the process of subtracting the calculated virtual unlit-on exposure frame image from the first lit-on exposure frame image.

26. in the movement vector calculation step, when the frame images captured in the order of the lit-on exposure frame image, the unlit-on exposure frame image, and the unlit-on exposure frame image are respectively designated as a first lit-on exposure frame image, a first unlit-on exposure frame image, and a second unlit-on exposure frame image, the movement vector is calculated from the first unlit-on exposure frame image and the second unlit-on exposure frame image; In the movement vector length adjustment step, the length of the movement vector of each pixel is adjusted in accordance with a ratio of a time difference between the first lights-out exposure frame image and the second lights-out exposure frame image to a time difference between the first lights-out exposure frame image and the first lights-out exposure frame image; in the lights-out exposure frame image correcting step, the virtual lights-out exposure frame image is calculated by correcting each pixel of the first lights-out exposure frame image by shifting the pixel in a direction opposite to the movement vector by the length of the movement vector after length adjustment; 16. The imaging program according to claim 15, wherein the lit-on exposure frame image processing step performs the process of subtracting the calculated virtual unlit-on exposure frame image from the first lit-on exposure frame image.

27. 16. The imaging program of claim 15, wherein in the imaging step, the light-on exposure frame image is captured by adding up the digital signals of a plurality of frame images captured by turning on the light-emitting unit and performing exposure synchronized with the lighting multiple times in succession, and the light-off exposure frame image is captured by adding up the digital signals of a plurality of frame images captured by turning off the light-emitting unit and performing exposure synchronized with the lighting multiple times in succession.

28. 16. The imaging program of claim 15, wherein in the imaging step, the light-on exposure frame image is captured by turning on the light-emitting unit, accumulating the signal amount for each pixel by exposure synchronized with the lighting, and integrating the accumulated signal amount multiple times in succession, and the light-off exposure frame image is captured by turning off the light-emitting unit, accumulating the signal amount for each pixel by exposure synchronized with the lighting, and integrating the accumulated signal amount multiple times in succession.

Images