Control apparatus and control method

Abstract

To provide "a control device and a control method" for suppressing a decrease in accuracy of a difference image due to a change in position of a subject between timings when a difference image is generated based on an irradiation-time image based on a photographing result of a camera at timing at which infrared rays are radiated, and a non-irradiation-time image based on the photographing result of the camera at timing at which the infrared rays are not radiated.SOLUTION: A control device includes: a photographing image acquisition part 12 that acquires a photographing image of a photographing device 4 at a predetermined cycle; a deviation detection part 13 that detects a deviation of a subject between an irradiation-time image that is a photographing image when an irradiation device 3 is in an irradiation state and a non-irradiation-time image that is a photographing image when the irradiation device is in a non-irradiation state; a deviation correction part 14 that corrects at least one of the irradiation-time image and the non-irradiation-time image so that the deviation detected by the deviation detection part 13 is suppressed; and a difference image generation part 15 that generates a difference image between the irradiation-time image and the non-irradiation-time image after the correction by the deviation correction part 14 is performed.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to a control device and a control method, and is particularly suitable for use in a control device and a control method capable of processing a captured image of a photographing device.

Background Art

[0002] In recent years, an in-vehicle imaging system has become widespread in which a camera for imaging a driver and other passengers is provided in the passenger compartment of a vehicle, and various image processes are executed using a captured image based on the imaging result of the camera. The various image processes are, for example, driver's line-of-sight detection, driver's drowsiness detection, or passenger face authentication. In this type of in-vehicle imaging system, since the brightness inside the vehicle is greatly affected by the time, weather, the environment around the vehicle, etc., it is widely practiced to irradiate a subject with infrared rays, which are invisible light, and capture the reflected light with a camera (infrared camera). However, sunlight contains not only visible light but also infrared components. Therefore, even in the in-vehicle imaging system that uses the above-described infrared rays, in the case where sunlight hits the subject, or in the case where sunlight is reflected by the glasses worn by the subject who is a person, the quality of the captured image of the camera (quality as data used for image processing) deteriorates, and as a result, the accuracy and appropriateness of the image processing performed using the captured image may decrease. As a means for solving this problem, Patent Document 1 describes the following technique.

[0003] That is, the system of Patent Document 1 is a system that detects a driver's line of sight based on a captured image of a camera. The system of Patent Document 1 includes a detection control device 11 and a line-of-sight sensor 12. The line-of-sight sensor 12 includes an infrared LED 22 and an infrared camera 21. Then, the detection control device 11 alternately executes irradiation and non-irradiation of the infrared LED 22 at a predetermined cycle, and causes the infrared camera 21 to perform imaging at the same predetermined cycle. As a result, the infrared camera 21 alternately generates an irradiated image (see FIG. 3(a) of Patent Document 1) based on the imaging result when irradiating infrared rays and a non-irradiated image (see FIG. 3(b) of Patent Document 1) based on the imaging result when not irradiating infrared rays at a predetermined cycle. Then, the detection control device 11 generates a difference image for the irradiated image and the non-irradiated image whose imaging timings are consecutive. The detection control device 11 performs image processing on the generated difference image and detects the driver's line of sight. The above technology is described in Patent Document 1. According to the technology of Patent Document 1, it is possible to suppress the adverse effects of sunlight (in particular, reflection by glasses) on the captured image used for detecting the driver's line of sight, and suppress a decrease in the accuracy and appropriateness of the processing performed using the captured image.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, the technology disclosed in Patent Document 1 had the following problems. That is, regarding a pair of an irradiated image and a non-irradiated image that form the basis for generating a difference image, the position of the subject may change between the timing at which the irradiated image was taken and the timing at which the non-irradiated image was taken. In particular, when the subject is a driver or other passenger boarding a vehicle, such a change is likely to occur due to the vibration of the vehicle. And in the technology of Patent Document 1, when such a change occurs, there is a problem that the accuracy of the generated difference image decreases. The difference image is generated by taking the difference in information between corresponding pixels of a pair of an irradiated image and a non-irradiated image that form the basis for the difference image (see paragraph 0018 of Cited Document 1). When the position of the subject changes between each timing, differences between pixels that do not correspond to each other will be taken. Note that the above problems can occur even when the irradiation device that irradiates infrared rays and the camera are not provided inside the vehicle cabin.

[0006] The present invention was made to solve such problems, and when generating a difference image based on an irradiated image based on the shooting result of a camera at the timing when infrared rays are irradiated and a non-irradiated image based on the shooting result of the camera at the timing when infrared rays are not irradiated, it aims to suppress a decrease in the accuracy of the difference image caused by a change in the position of the subject between these timings.

Means for Solving the Problems

[0007] In order to solve the above-described problems, the present invention detects the displacement of the subject between an irradiated image and a non-irradiated image generated at consecutive timings, corrects at least one of these images so that the displacement is suppressed, and then generates a difference image of these images.

Effects of the Invention

[0008] According to the present invention configured as described above, it is possible to generate a difference image based on these images in a state where the displacement of the subject between the irradiated image and the non-irradiated image is corrected. For this reason, it is possible to suppress a decrease in the accuracy of the difference image caused by a change in the position of the subject during the timing when each image was taken.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Modes for Carrying Out the Invention

[0010] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing a functional configuration example of a control system 1 according to this embodiment. The control system 1 is a system applied to a vehicle 2 and includes an irradiation device 3, a photographing device 4, and a control device 5 as shown in FIG. 1. The irradiation device 3 includes an infrared light emitting diode (infrared LED) that irradiates infrared light to a driver (subject) sitting in the driver's seat. The photographing device 4 is a camera provided at a position where the driver sitting in the driver's seat can be photographed. The photographing device 4 is an infrared camera and has sensitivity in the infrared region. Members necessary for performing photographing based on a filter considering the wavelength of infrared light and other reflected light of infrared light are appropriately provided in the photographing device 4. The irradiation device 3 and the photographing device 4 are respectively provided at positions where the photographing device 4 can receive the reflected light of the infrared light irradiated by the irradiation device 3 to the driver.

[0011] The control device 5 has a function of executing image processing based on a photographed image based on the photographing result of the photographing device 4. The image processing is, for example, driver's line-of-sight detection, drowsiness detection, or face authentication. In particular, the control device 5 according to this embodiment has a function of generating a differential image SX used for image processing, as will be described later. The direction of the optical axis of the photographing device 4 is set to an appropriate direction according to the content of the image processing of the control device 5. For example, when the content of the image processing is driver's line-of-sight detection, the direction of the optical axis of the photographing device 4 is set to a direction such that the face of the driver sitting in the driver's seat in a normal manner is included in the photographing range of the photographing device 4.

[0012] As shown in FIG. 1, the control device 5 includes, as functional components, an irradiation control unit 10, a shooting control unit 11, a captured image acquisition unit 12, a deviation detection unit 13, a deviation correction unit 14, a difference image generation unit 15, and an image processing unit 16. The functions of these functional blocks 10 to 16 are realized by hardware or by the cooperation of hardware and software. This also applies to the modified examples described later. For example, the function of any one of these functional blocks 10 to 16 is realized by an analog circuit or a dedicated chip. Also, for example, any one of these functional blocks 10 to 16 includes a processing device (e.g., a CPU), a primary storage device (e.g., a DRAM), and an auxiliary storage device (e.g., a hard disk drive, a ROM, or a flash memory). And the function of the functional block is realized by the processing device reading out the program stored in the auxiliary storage device to the primary storage device and executing it.

[0013] Based on the shooting result by the shooting device 4, the control device 5 continuously executes image processing during the processing target period. The processing target period is a period during which image processing by the control device 5 is performed and is appropriately set according to the content of the image processing. For example, when the image processing is the driver's line-of-sight detection, the processing target period is, for example, a period during which the accessory power supply of the vehicle 2 is turned on. Also, for example, when the image processing is the driver's dozing detection, the processing target period is, for example, a period during which the engine of the vehicle 2 is started. Hereinafter, the functions and processes of each functional block 10 to 16 of the control device 5 (especially the processes during the processing target period) will be described in detail.

[0014] The illumination control unit 10 has a function of controlling the illumination device 3. During the processing period, the illumination control unit 10 alternately switches the illumination device 3 between an illumination state and a non-illumination state at a frame period (a predetermined period). The frame period corresponds to the frame rate of the image capture device 4 (for example, 60 fps or 30 fps). The illumination control unit 10 detects each timing of the frame period based on a clock supplied by a clock circuit (not shown). This also applies to other functional blocks. The illumination state is a state in which the illumination device 3 irradiates infrared light. The non-illumination state is a state in which the illumination device 3 does not irradiate infrared light (non-illumination state). FIG. 2(A) is a timing chart showing, on a time axis, a period in which the illumination device 3 irradiates infrared light under the control of the illumination control unit 10 according to this embodiment. In FIG. 2, timings T0, T1, T2, etc. indicate timings occurring at the frame period.

[0015] In the example of FIG. 2A, the irradiation control unit 10 does not cause the irradiation device 3 to irradiate infrared light at timing T0. That is, the irradiation control unit 10 sets the state of the irradiation device 3 to a non-irradiation state at timing T0. Hereinafter, the timing of the frame period at which the irradiation control unit 10 does not start irradiation by the irradiation device 3 is referred to as a "non-irradiation timing." Thereafter, at timing T1, the irradiation control unit 10 starts irradiating infrared light by the irradiation device 3. That is, the irradiation control unit 10 sets the state of the irradiation device 3 to an irradiation state at timing T1. Hereinafter, the timing of the frame period at which the irradiation control unit 10 starts irradiating infrared light by the irradiation device 3 is referred to as an "irradiation timing." After starting irradiation by the irradiation device 3 at timing T1, the irradiation control unit 10 continues irradiation by the irradiation device 3 for a fixed period shorter than the length of one frame. In other words, the irradiation control unit 10 stops irradiating infrared light by the irradiation device 3 before timing T2. Hereinafter, the fixed period during which irradiation by the irradiation device 3 is performed is referred to as an "irradiation period."

[0016] Thereafter, when timing T2 arrives, the irradiation control unit 10 sets the state of the irradiation device 3 to the non-irradiation state without starting the irradiation by the irradiation device 3. Thereafter, when timing T3 arrives, the irradiation control unit 10 starts the irradiation by the irradiation device 3 and sets the state of the irradiation device 3 to the irradiation state. After reaching timing T3, the irradiation control unit 10 stops the irradiation by the irradiation device 3 after the elapse of the irradiation period. Thereafter, when timing T4 arrives, the irradiation control unit 10 sets the state of the irradiation device 3 to the non-irradiation state without starting the irradiation by the irradiation device 3. Thereafter, every time the timing of the frame period arrives, the irradiation control unit 10 repeatedly executes the start of the irradiation by the irradiation device 3 and the continuation of the irradiation during the irradiation period, and the non-irradiation of infrared rays by the irradiation device 3. As a result, with respect to the state of the irradiation device 3, the irradiation state and the non-irradiation state appear alternately in the frame period.

[0017] The imaging control unit 11 has a function of controlling the imaging device 4. The imaging control unit 11 causes imaging to be performed at the frame period during the processing target period. FIG. 2(B) is a timing chart showing on the time axis the period during which imaging is performed by the imaging device 4 under the control of the imaging control unit 11. In FIG. 2(B), the period during which imaging is performed by the imaging device 4 means the period during which the shutter of the imaging device 4 is open (the exposed period). As shown in FIG. 2(B), the imaging control unit 11 opens the shutter of the imaging device 4 at the frame period and causes the imaging device 4 to perform imaging at the frame period. The length of the irradiation period is set to include the length of the period during which the shutter is open. As a result, imaging is performed by the imaging device 4 at the frame period, and an imaging image based on the imaging result is generated at the frame period.

[0018] As described above, under the irradiation control unit 10, the irradiation device 3 repeats irradiation and non-irradiation alternately in a frame period. Therefore, regarding the imaging by the imaging device 4, imaging in an irradiation state where infrared rays are irradiated by the irradiation device 3 and imaging in a non-irradiation state where infrared rays are not irradiated by the irradiation device 3 are alternately repeated in a frame period. As a result, the imaging device 4 alternately generates an imaging image based on the imaging result of imaging performed in the irradiation state and an imaging image based on the imaging result of imaging performed in the non-irradiation state in a frame period. Hereinafter, an imaging image based on the imaging result of imaging performed in the irradiation state is referred to as an "image PX during irradiation", and an imaging image based on the imaging result of imaging performed in the non-irradiation state is referred to as an "image QX during non-irradiation". FIG. 2(C) is a diagram showing in time series which of the image PX during irradiation and the image QX during non-irradiation is generated by the imaging device 4 at each timing of the frame period. As shown in FIG. 2(C), the imaging device 4 generates the image QX during non-irradiation at timings T0, T2, T4... which are non-irradiation timings, and generates the image PX during irradiation at timings T1, T3, T5... which are irradiation timings. Although details are omitted, when generating an imaging image, the imaging device 4 executes processing related to filtering and other necessary image adjustments and then generates the imaging image.

[0019] The imaging image acquisition unit 12 acquires the imaging image of the imaging device 4 in a frame period during the processing target period. As described above, the imaging device 4 alternately generates the image PX during irradiation and the image QX during non-irradiation in a frame period. Therefore, the imaging image acquisition unit 12 alternately acquires the image PX during irradiation and the image QX during non-irradiation in a frame period. Regarding the acquisition of the imaging image and the processing related to the acquisition, the imaging image acquisition unit 12 executes different processing at the irradiation timing and the non-irradiation timing, respectively.

[0020] During non-irradiation timing, the captured image acquisition unit 12 executes the following processes. That is, the captured image acquisition unit 12 acquires the non-irradiation image QX from the imaging device 4. Next, the captured image acquisition unit 12 overwrites and stores (expands) the acquired non-irradiation image QX in the frame memory 17. The frame memory 17 is a buffer formed in a predetermined storage area. On the other hand, during irradiation timing, the captured image acquisition unit 12 executes the following processes. That is, the captured image acquisition unit 12 acquires the irradiation image PX from the imaging device 4. Next, the captured image acquisition unit 12 outputs the acquired irradiation image PX to the displacement detection unit 13 and the difference image generation unit 15.

[0021] Figure 2(D) is a diagram showing in time series which of the storage of the non-irradiation image QX in the frame memory 17 and the output of the irradiation image PX to the displacement detection unit 13 and the difference image generation unit 15 the captured image acquisition unit 12 executes at each timing of the frame period. In Figure 2(D), the storage of the non-irradiation image QX in the frame memory 17 is expressed as "storage", and the output of the irradiation image PX to the displacement detection unit 13 and the difference image generation unit 15 is expressed as "output". As shown in Figure 2(D), the captured image acquisition unit 12 stores the non-irradiation image QX in the frame memory 17 at timings T0, T2, T4... which are non-irradiation timings, and outputs the irradiation image PX to the displacement detection unit 13 and the difference image generation unit 15 at timings T1, T3, T5... which are irradiation timings.

[0022] The displacement detection unit 13 detects the displacement of the subject between the irradiation image PX and the non-irradiation image QX acquired at consecutive timings by the captured image acquisition unit 12. In particular, the displacement detection unit 13 according to the present embodiment compares the characteristic portion of the irradiation image PX with the corresponding characteristic portion of the non-irradiation image QX to detect the displacement. Hereinafter, the process of the displacement detection unit 13 will be described in detail.

[0023] The deviation detection unit 13 does not execute processing during non-irradiation timing. On the other hand, the deviation detection unit 13 executes the following processing during irradiation timing. That is, as described above, during irradiation timing, the irradiation-time image PX is output from the imaging image acquisition unit 12 to the deviation detection unit 13. Also, during irradiation timing, the non-irradiation-time image QX acquired by the imaging image acquisition unit 12 at non-irradiation timing one cycle before this irradiation timing is stored in the frame memory 17. Based on the above, during irradiation timing, the deviation detection unit 13 acquires the non-irradiation-time image QX from the frame memory 17. Next, the deviation detection unit 13 executes deviation detection processing on the non-irradiation-time image QX acquired from the frame memory 17 and the irradiation-time image PX input from the imaging image acquisition unit 12, and detects the deviation of the subject between these images. Hereinafter, the non-irradiation-time image QX acquired by the deviation detection unit 13 from the frame memory 17 is referred to as "comparison non-irradiation-time image QX-1", and the irradiation-time image PX input by the deviation detection unit 13 from the imaging image acquisition unit 12 is referred to as "comparison irradiation-time image PX-1".

[0024] Hereinafter, the deviation detection processing will be described in detail. In the deviation detection processing, first, the deviation detection unit 13 detects feature portions in each of the comparison irradiation-time image PX-1 and the comparison non-irradiation-time image QX-1. The detection of the feature portions is performed using existing feature point detection algorithms such as Harris corner detection, corner detection by FAST (Features from Accelerated Segment Test), and feature point detection by SIFT (Scaled Invariance Feature Transform). Further, the deviation detection unit 13 performs matching (association, comparison) between each feature portion of the comparison irradiation-time image PX-1 and each feature portion of the comparison non-irradiation-time image QX-1. The matching is performed using existing matching techniques. For example, the deviation detection unit 13 obtains the feature amounts of each feature portion of each image, and determines that feature portions whose similarity of feature amounts is equal to or greater than a threshold value are in a corresponding relationship. Regarding the similarity, for example, in the case of SIFT, the similarity between the feature portion of the comparison irradiation-time image PX-1 and the feature portion of the comparison non-irradiation-time image QX-1 is derived based on the Euclidean distance of the feature amounts of each feature portion.

[0025] Next, the deviation detection unit 13 derives the overall deviation value (deviation) of the comparison non-irradiation image QX-1 with respect to the comparison irradiation image PX-1 based on the result of matching the feature portions of the respective images. The overall deviation value means the following. That is, assume that the comparison irradiation image PX-1 and the comparison non-irradiation image QX-1 are arranged so as to overlap in the same coordinate system. FIG. 3(A) schematically shows, in a manner suitable for explanation, a state in which the comparison irradiation image PX-1 in which the image PG of the subject is recorded and the comparison non-irradiation image QX-1 in which the image QG of the same subject is recorded are arranged so as to overlap in the coordinate system. In FIG. 3(A), for ease of viewing, the comparison irradiation image PX-1 and the comparison non-irradiation image QX-1 are drawn with a slight shift.

[0026] In FIG. 3(A), a deviation occurs between the image PG of the subject related to the comparison irradiation image PX-1 and the image QG of the subject related to the comparison non-irradiation image QX-1. Hereinafter, the deviation of the subject occurring in the pair of images that are the basis of the difference image SX is referred to as "image deviation". This image deviation may occur due to a change in the position of the subject between the timing when the comparison non-irradiation image QX-1 is taken and the timing when the comparison irradiation image PX-1 is taken. In particular, in the present embodiment, the subject is the driver who gets on the vehicle 2. Therefore, such a change is likely to occur due to the vibration of the vehicle 2. In FIG. 3(A), for convenience of explanation, the image deviation between the image PG and the image QG is drawn very large. However, the period between the timing when the comparison non-irradiation image QX-1 is taken and the timing when the comparison irradiation image PX-1 is taken is a short time such as 1 / 60 second (in the case of 60 fps) or 1 / 30 second (in the case of 30 fps). Therefore, in reality, it is assumed that the image deviation does not become as large as illustrated in FIG. 3(A).

[0027] Now, when the comparison irradiated image PX-1 and the comparison non-irradiated image QX-1 are arranged by overlapping them in the same coordinate system, it is assumed that the image of the subject recorded in the comparison non-irradiated image QX-1 can be made to coincide with the image of the subject recorded in the comparison irradiated image PX-1 by moving the comparison non-irradiated image QX-1 by a predetermined amount in a predetermined direction. Note that "coincidence" does not mean that all the pixels of each image completely coincide, but means that they coincide to such an extent that the required high accuracy can be ensured for the differential image SX described later. At this time, the combination of the said predetermined direction and the said predetermined amount is the "overall deviation value". FIG. 3(B) shows, together with an arrow indicating the overall deviation value, the state where the comparison non-irradiated image QX-1 is moved from the state of FIG. 3(A) so that the image PG and the image QG coincide with each other.

[0028] The deviation detection unit 13 derives the overall deviation value by, for example, the following method. That is, the deviation detection unit 13 derives an individual deviation value of the feature portion of the comparison non-irradiated image QX-1 with respect to the feature portion of the comparison irradiated image PX-1 for each pair of corresponding feature portions. The individual deviation value is a combination of the direction of movement and the amount of movement when moving the feature portion of the comparison non-irradiated image QX-1 in the coordinate system so that there is no deviation from the feature portion of the comparison irradiated image PX-1. After the deviation detection unit 13 derives an individual deviation value for each pair of corresponding feature portions, the overall deviation value is derived by a calculation method using a statistical method for each of the individual deviation values. The calculation method is, for example, an average or a weighted average. The above is the deviation detection process. Regarding the weighted average, the deviation detection unit 13 executes, for example, the following process. That is, the deviation detection unit 13 weights the individual deviation value of the feature portion corresponding to "the image QG that is the main image processing target (for example, the image of the face that is the target when drowsiness detection or face authentication is performed as image processing) among the images constituting the comparison non-irradiated image QX-1" and the feature portion corresponding to "the image PG that is the main image processing target among the images constituting the comparison irradiated image PX-1" so that it has a greater influence on the derived overall deviation value than the individual deviation values of other feature portions (including the individual deviation values corresponding to the background), and executes a weighted average.

[0029] The above described an example of a method for deriving the overall deviation value in the deviation detection process. However, the method for deriving the overall deviation value is not limited to the exemplified method. For example, the following method may also be used. That is, for the comparison irradiated image PX-1 and the comparison non-irradiated image QX-1 arranged overlapping in the coordinate system, while keeping the comparison irradiated image PX-1 fixed, the comparison non-irradiated image QX-1 is moved in various directions and by various amounts of movement. However, the movement of the comparison non-irradiated image QX-1 is performed according to rules within a range considering the image deviation that can actually occur. Each time the deviation detection unit 13 moves the comparison non-irradiated image QX-1, it derives the similarity between the moved comparison non-irradiated image QX-1 and the comparison irradiated image PX-1. Then, the deviation detection unit 13 identifies the position with the highest similarity among the positions of the moved comparison non-irradiated image QX-1, and sets the "direction of movement and amount of movement" corresponding to the identified position as the overall deviation value. The above method may also be used. Also, for example, the method for obtaining the overall deviation value may be the following method. That is, the deviation detection unit 13 derives the individual deviation values between the feature part corresponding to the image QG which is the main image processing target of the comparison non-irradiated image QX-1, and the feature part corresponding to the image PG which is the main image processing target of the comparison irradiated image PX-1. Then, the deviation detection unit 13 inputs the derived individual deviation values into a "calculation formula (model)" that takes the individual deviation values as inputs and outputs the overall deviation value, and derives the overall deviation value. The above method may also be used. In this embodiment, the overall deviation value and the individual deviation values do not include the rotation direction and rotation amount of the comparison non-irradiated image QX-1 as elements, but a configuration including these as elements may also be used.

[0030] After deriving the overall deviation value, the deviation detection unit 13 outputs the overall deviation value to the deviation correction unit 14. The above is the process executed by the deviation detection unit 13 at the irradiation timing. FIG. 2(E) shows the timing at which the deviation detection unit 13 executes the deviation detection process.

[0031] The deviation correction unit 14 corrects at least one of the irradiated image PX and the non-irradiated image QX so that the deviation detected by the deviation detection unit 13 is suppressed. Hereinafter, the processing of the deviation correction unit 14 will be described in detail.

[0032] The deviation correction unit 14 does not execute processing at non-irradiation timing. On the other hand, the deviation correction unit 14 executes the following processing at irradiation timing. That is, as described above, at irradiation timing, the non-irradiated image QX acquired by the captured image acquisition unit 12 at non-irradiation timing one cycle before this irradiation timing is stored in the frame memory 17. Further, the deviation detection unit 13 outputs the overall deviation value to the deviation correction unit 14. Based on the above, at irradiation timing, the deviation correction unit 14 acquires the non-irradiated image QX from the frame memory 17. Next, the deviation correction unit 14 performs deviation correction processing and corrects the non-irradiated image QX acquired from the frame memory 17 so as to move according to the overall deviation value input from the deviation correction unit 14. Hereinafter, the deviation correction processing will be described in detail.

[0033] In the deviation correction processing, the deviation correction unit 14 moves all the pixels of the non-irradiated image QX according to the overall deviation value (converts the pixel value of the destination pixel to the pixel value of the source pixel). At that time, the deviation correction unit 14 discards the pixels whose destination is outside the outer frame of the image. In addition, for the pixels within the outer frame of the image that do not become the destination, the deviation correction unit 14 sets the pixel value to a dummy value (a value indicating that it is not used for generating the differential image SX described later). Hereinafter, the non-irradiated image QX after being corrected by the deviation correction unit 14 is referred to as the corrected image HX. Step S1 in FIG. 4 shows how the corrected image HX is generated from the non-irradiated image QX by the deviation correction processing. In the corrected image HX in FIG. 4, the black-filled portions on the right side and the lower side indicate the pixels of the dummy value. The above is the deviation correction processing. Note that the process in which the deviation correction unit 14 moves the pixels of the non-irradiated image QX according to the overall deviation value and generates the corrected image HX corresponds to the process of correcting the non-irradiated image QX so that the deviation (image deviation) detected by the deviation detection unit 13 is suppressed.

[0034] After generating the corrected image HX through the misalignment correction process, the misalignment correction unit 14 outputs the generated corrected image HX to the differential image generation unit 15. The above is the process executed by the misalignment correction unit 14 at the irradiation timing. FIG. 2(F) shows the timing at which the misalignment correction unit 14 executes the misalignment correction process.

[0035] After the correction by the misalignment correction unit 14, the differential image generation unit 15 generates a differential image between the irradiated image PX and the non-irradiated image QX. Hereinafter, the process of the misalignment correction unit 14 will be described in detail.

[0036] The differential image generation unit 15 does not execute processing at non-irradiation timing. On the other hand, the differential image generation unit 15 executes the following processing at the irradiation timing. That is, as described above, at the irradiation timing, the captured image acquisition unit 12 outputs the irradiated image PX acquired at that timing to the differential image generation unit 15. Also, the misalignment correction unit 14 outputs the corrected image HX generated at that timing to the differential image generation unit 15. Based on the above, at the irradiation timing, the differential image generation unit 15 performs differential image generation processing to generate a differential image SX between the irradiated image PX input from the captured image acquisition unit 12 and the corrected image HX input from the misalignment correction unit 14. In the differential image generation processing, the differential image generation unit 15 derives the difference in pixel information for each pair of corresponding pixels of the irradiated image PX and the corrected image HX, and generates the differential image SX. However, the differential image generation unit 15 does not include in the differential image SX the pixel group with dummy values of the corrected image HX. Step S2 in FIG. 4 shows how the differential image SX is generated from the corrected image HX and the irradiated image PX by the differential image generation processing.

[0037] After generating the differential image SX by the differential image generation processing, the differential image generation unit 15 outputs the generated differential image SX to the image processing unit 16. The above is the process executed by the differential image generation unit 15 at the irradiation timing. FIG. 2(G) shows the timing at which the differential image generation unit 15 executes the differential image generation processing.

[0038] Here, the difference image generation unit 15 generates a difference image SX based on the non-irradiation image QX and the irradiation image PX that are corrected so as to suppress image misalignment. Therefore, the difference image generation unit 15 generates the difference image SX in a state where "the misalignment of the positions of the pixels of the non-irradiation image QX and the pixels of the irradiation image PX" caused by the movement of the subject's position between the shooting timing of the non-irradiation image QX and the shooting timing of the irradiation image PX is suppressed. Therefore, according to the present embodiment, regarding the accuracy of the generated difference image SX, it is possible to suppress a decrease in the accuracy of the difference image SX caused by a change in the position of the subject between the timings at which the pair of images serving as the basis of the difference image SX are taken.

[0039] The image processing unit 16 executes predetermined image processing based on the difference image SX input from the difference image generation unit 15. Examples of the predetermined image processing are as described above.

[0040] Next, the processing of the main part by the control device 5 will be described using a flowchart. The flowchart FA in FIG. 5 is a flowchart showing the control method by the control device 5.

[0041] As shown in the flowchart FA in FIG. 5, the captured image acquisition unit 12 acquires the captured image of the imaging device at a predetermined cycle (step SA1). The misalignment detection unit 13 detects the misalignment of the subject between the irradiation image PX and the non-irradiation image QX acquired at consecutive timings by the captured image acquisition unit 12 (step SA2). The misalignment correction unit 14 corrects at least one of the irradiation image PX and the non-irradiation image QX so as to suppress the image misalignment (misalignment) detected by the misalignment detection unit 13 (step SA3). The difference image generation unit 15 generates a difference image SX between the irradiation image PX and the non-irradiation image QX after the correction by the misalignment correction unit 14 is performed (step SA4).

[0042] As described above, the control device 5 according to the present embodiment detects the displacement of the subject between the irradiated image PX and the non-irradiated image QX generated at consecutive timings, corrects at least one of these images so as to suppress the displacement, and then generates a difference image SX of these images. According to this configuration, it is possible to generate a difference image SX based on these images in a state where the displacement of the subject in the irradiated image PX and the non-irradiated image QX is corrected. Therefore, it is possible to suppress a decrease in the accuracy of the difference image SX caused by a change in the position of the subject between the timings at which each image was taken.

[0043] <First Modification Example> Next, a first modification example of the above embodiment will be described. The irradiation control unit 10 according to the above embodiment alternately sets the state of the irradiation device 3 to an irradiation state and a non-irradiation state at a frame period. In this regard, the irradiation control unit 10 according to this modification example sets the state of the irradiation device 3 to a weak irradiation state in which infrared rays are irradiated with an output smaller than the irradiation state, instead of setting it to a non-irradiation state.

[0044] FIG. 6 is a timing chart showing on the time axis the period during which infrared rays are irradiated by the irradiation device 3 under the control of the irradiation control unit 10 according to this modification example. In FIG. 6, it is assumed that the length of the bar extending upward from the base end at the irradiation timing and the non-irradiation timing indicates the magnitude of the output of the irradiation device 3. As shown in FIG. 6, the irradiation control unit 10 does not completely irradiate the irradiation device 3 with infrared rays at the non-irradiation timings (timings T0, T2, T4,... in FIG. 6), but irradiates the infrared rays with an output sufficiently smaller than the output of the infrared rays at the irradiation timing.

[0045] According to the configuration of this modification example, the following effects can be achieved. That is, for the non-irradiation image QX generated based on the shooting result at the non-irradiation timing, the brightness of the image can be improved as a whole. Thereby, in the deviation detection process, it is possible to improve the accuracy of the feature amount detected for the non-irradiation image QX and the accuracy of the matching between the feature amount related to the irradiation image PX and the feature amount related to the non-irradiation image QX. Moreover, since there is a sufficient difference between the magnitude of the infrared output at the irradiation timing and the magnitude of the infrared output at the non-irradiation timing, the difference in the information amount (brightness) between each pixel of the irradiation image PX and each pixel of the non-irradiation image QX is sufficiently ensured, and the appropriateness of the difference image SX generated by the difference image generation unit 15 is maintained. In this modification example, the magnitude of the infrared output in the weak irradiation state is set to a magnitude that ensures the appropriateness of the finally generated difference image SX under prior tests and simulations.

[0046] <Second Modification Example> Next, a second modification example of the above embodiment will be described. FIG. 7 is a block diagram showing a functional configuration example of the control system 1A according to the second modification example. In FIG. 7, elements having the same functions as those in the above embodiment are denoted by the same reference numerals. As shown in FIG. 7, the control system 1A according to this modification example includes an irradiation device 3, a photographing device 4, a photographing-related device 20, a control device 5A, and an image processing device 30. The control device 5A does not include an irradiation control unit 10 and a photographing control unit 11. On the other hand, the photographing-related device 20 includes an irradiation control unit 10 and a photographing control unit 11. Also, the control device 5A does not include an image processing unit 16. On the other hand, the image processing device 30 includes an image processing unit 16. As shown in this modification example, a configuration in which the control device does not include an irradiation control unit, a photographing control unit, and an image processing unit may be used. Although not shown, a configuration in which the control device does not include any one or any combination of two of the irradiation control unit, the photographing control unit, and the image processing unit may also be used.

[0047] The above describes one embodiment of the present invention (including each modification. The same shall apply hereinafter). However, the above embodiment merely shows an example of the implementation of the present invention, and the technical scope of the present invention should not be construed in a limited manner thereby. That is, the present invention can be implemented in various forms without departing from its gist or its main features.

[0048] For example, in the above embodiment, the control system 1 is a system applied to the vehicle 2. However, the system including the control device is not limited to those provided for vehicles. That is, the system including the control device is widely applicable to a system that generates a difference image based on the irradiated image and the non-irradiated image.

[0049] Also, in the above embodiment, the deviation correction unit 14 is configured to correct the non-irradiated image. In this regard, the deviation correction unit 14 may be configured to correct the irradiated image, or may be configured to correct both the irradiated image and the non-irradiated image. In this case, the deviation detection unit 13 outputs appropriate information to the deviation correction unit 14 based on the object to be corrected.

[0050] Also, in the above embodiment, the deviation detection unit 13 detects, as the deviation between the irradiated image PX and the non-irradiated image QX, in which direction and by how much amount the image is shifted as a whole. In this regard, the deviation detection unit 13 may be configured to detect the mode or correspondence relationship of the deviation for each pixel of the irradiated image PX and the non-irradiated image QX.

[0051] Also, in the above embodiment, the difference image generation unit 15 is configured to generate the difference image SX at each of the irradiation timings. In this regard, the difference image generation unit 15 may be configured to generate the difference image SX at each timing of the frame period. In the case of this configuration, each functional block of the control device 5 executes, for example, the following processing.

[0052] That is, as frame memories, a frame memory in which the irradiation-time image PX is developed (hereinafter referred to as the "irradiation-time image memory") and a frame memory in which the non-irradiation-time image QX is developed (hereinafter referred to as the "non-irradiation-time image memory") are prepared. At the irradiation timing, the captured image acquisition unit 12 overwrites and stores the irradiation-time image PX acquired from the imaging device 4 in the irradiation-time image memory. Also, at the non-irradiation timing, the captured image acquisition unit 12 overwrites and stores the non-irradiation-time image QX acquired from the imaging device 4 in the non-irradiation-time image memory. The deviation detection unit 13 executes the following processing at each timing of the frame period. That is, the deviation detection unit 13 derives an overall deviation value corresponding to the image deviation of the non-irradiation-time image QX with respect to the irradiation-time image PX based on the irradiation-time image PX stored in the irradiation-time image memory and the non-irradiation-time image QX stored in the non-irradiation-time image memory, and outputs it to the deviation correction unit 14. The deviation correction unit 14 executes the following processing at each timing of the frame period. That is, the deviation correction unit 14 corrects the non-irradiation-time image QX developed in the non-irradiation-time image memory based on the overall deviation value input from the deviation detection unit 13 to generate a corrected image HX, and outputs it to the difference image generation unit 15. The difference image generation unit 15, that is, at each timing of the frame period, the difference image generation unit 15 generates a difference image SX based on the corrected image HX input from the deviation correction unit 14 and the irradiation-time image PX stored in the irradiation-time image memory, and outputs it to the image processing unit 16.

[0053] Also, the functional blocks shown in the above embodiment can be realized by any hardware, or by the cooperation of any hardware and any software. That is, these functional blocks are not limited to specific hardware.

[0054] Also, for example, regarding the processing described as being executed by each functional block alone, a configuration in which each functional block cooperates with an external device to execute various processes may be adopted. As an example, a configuration in which the deviation detection unit 13 cooperates with an external device (for example, a cloud server) that can communicate via a network to execute processing may be adopted.

[0055] Furthermore, for example, the provision of a program executed by the computer of the control device 5 can be included in the embodiments. Furthermore, the provision of a recording medium on which the program is recorded so as to be readable by a computer can be included in the embodiments. The recording medium can be a magnetic or optical recording medium or a semiconductor memory device. Specific examples include portable or fixed recording media such as flexible disks, HDDs (Hard Disk Drives), CD-ROMs (Compact Disk Read Only Memory), DVDs (Digital Versatile Disks), Blu-ray (registered trademark) Disks, magneto-optical disks, flash memories, and card-type recording media. [Explanation of symbols]

[0056] 1. 1A control system 2 vehicles 3 Irradiation device 4. Imaging equipment 5, 5A control device 10 Irradiation control unit 11. Imaging control unit 12. Image acquisition unit 13 Misalignment detection unit 14 Misalignment correction unit 15. Differential image generation unit 16 Image processing section

Claims

1. An irradiation control unit that alternates the state of an irradiation device capable of irradiating an object with infrared rays between an irradiation state in which infrared rays are irradiated at a predetermined cycle and a non-irradiation state in which no irradiation is performed; A photographing control unit that causes a photographing device capable of photographing the object to perform photographing at the predetermined cycle; A photographed image acquisition unit that acquires a photographed image of the photographing device at the predetermined cycle; A deviation detection unit that detects a deviation of the object between an irradiated image that is the photographed image obtained at consecutive timings by the photographed image acquisition unit in the irradiation state and a non-irradiated image that is the photographed image in the non-irradiation state; A deviation correction unit that corrects at least one of the irradiated image and the non-irradiated image so that the deviation detected by the deviation detection unit is suppressed; A difference image generation unit that generates a difference image between the irradiated image and the non-irradiated image after the correction by the deviation correction unit. A control device characterized by the above.

2. Instead of setting the state of the irradiation device to the non-irradiation state, the irradiation control unit sets it to a weak irradiation state in which infrared rays are irradiated with an output smaller than the irradiation state. The control device according to claim 1, characterized by the above.

3. At a first timing, the photographed image acquisition unit acquires the photographed image and stores the acquired photographed image in a frame memory; At a second timing following the first timing, the photographed image acquisition unit acquires the photographed image; the deviation detection unit detects a deviation between the photographed image stored in the frame memory and the photographed image acquired by the photographed image acquisition unit at the second timing; the deviation correction unit performs correction based on the deviation detected by the deviation detection unit at the second timing on at least one of the photographed device stored in the frame memory and the photographed image acquired by the photographed image acquisition unit at the second timing; the difference image generation unit generates a difference image between the photographed device stored in the frame memory and the photographed image acquired by the photographed image acquisition unit at the second timing after the correction by the deviation correction unit. The control device according to claim 1 or 2, characterized by the above.

4. The deviation detection unit compares a characteristic portion of the irradiated image with a corresponding characteristic portion of the non-irradiated image to detect the deviation. The control device according to claim 1 or 2, characterized by the above.

5. The irradiation device and the imaging device are provided in the passenger compartment of the vehicle. The control device according to claim 1 or 2, characterized in that.

6. An irradiation device capable of irradiating a subject with infrared rays and alternating between an irradiation state in which infrared rays are irradiated at a predetermined period and a non-irradiation state in which no irradiation is performed, and a control device capable of executing processing related to an imaging device capable of imaging the subject at the predetermined period, An imaging image acquisition unit that acquires an imaging image of the imaging device at the predetermined period; A deviation detection unit that detects a deviation of the subject between an irradiation-time image that is the imaging image at the time of the irradiation state and a non-irradiation-time image that is the imaging image at the time of the non-irradiation state, which are acquired at consecutive timings by the imaging image acquisition unit; A deviation correction unit that corrects at least one of the irradiation-time image and the non-irradiation-time image so that the deviation detected by the deviation detection unit is suppressed; A difference image generation unit that generates a difference image between the irradiation-time image and the non-irradiation-time image after the correction by the deviation correction unit is performed. The control device characterized by the above.

7. A control method by a control device capable of irradiating a subject with infrared rays and alternating between an irradiation state in which infrared rays are irradiated at a predetermined period and a non-irradiation state in which no irradiation is performed, and capable of executing processing related to an imaging device capable of imaging the subject at the predetermined period, A step in which an imaging image acquisition unit of the control device acquires an imaging image of the imaging device at the predetermined period; A step in which a deviation detection unit of the control device detects a deviation of the subject between an irradiation-time image that is the imaging image at the time of the irradiation state and a non-irradiation-time image that is the imaging image at the time of the non-irradiation state, which are acquired at consecutive timings by the imaging image acquisition unit; A step in which a deviation correction unit of the control device corrects at least one of the irradiation-time image and the non-irradiation-time image so that the deviation detected by the deviation detection unit is suppressed; A step in which a difference image generation unit of the control device generates a difference image between the irradiation-time image and the non-irradiation-time image after the correction by the deviation correction unit is performed, including. The control method characterized by the above.

Images