Image processing device and image processing method
The image processing device and method manage shutter operation and display to protect infrared sensors from sunlight, enabling continuous recognition of vehicles and people in vehicles.
Patent Information
- Application Number
- JP2023077187
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Filing Date
- 2023-05-09
- Publication Date
- 2024-11-21
AI Technical Summary
Infrared sensors in vehicles face challenges in recognizing objects like vehicles and people when exposed to sunlight due to the need to protect themselves from sunlight by closing shutters, which disrupts continuous recognition.
An image processing device and method that includes a thermal image data acquisition unit, saturation detection, shutter control, and display control to manage shutter operation and display thermal images before shutter closure during sunlight exposure, ensuring continuous recognition.
Enables the recognition of vehicles and people while protecting the infrared sensor from sunlight damage, allowing continuous monitoring and warning capabilities in vehicles.
Smart Images

Figure 2024162036000001_ABST
Abstract
Description
[Technical field]
[0001] The present disclosure relates to an image processing device and an image processing method. [Background technology]
[0002] Patent Document 1 discloses that when sunlight is incident on an infrared sensor, a shutter is closed to protect the infrared sensor. [Prior art documents] [Patent documents]
[0003] [Patent Document 1] JP 2022-187078 A Summary of the Invention [Problem to be solved by the invention]
[0004] The infrared sensor of Patent Document 1 does not recognize people, etc. while the shutter is closed. When the infrared sensor of Patent Document 1 is applied to a vehicle, since the external conditions change as the vehicle moves, it is desirable to protect the infrared sensor from sunlight by closing the shutter and to issue a warning by recognizing the vehicle and people, etc.
[0005] The present disclosure has been made to solve such problems, and aims to provide an image processing device and an image processing method that are capable of recognizing vehicles, people, etc. while protecting the infrared sensor when exposed to sunlight. [Means for solving the problem]
[0006] Therefore, the present disclosure provides an image processing device comprising: a thermal image data acquisition unit that acquires thermal image data consisting of a plurality of frames from an infrared camera that captures thermal images; a saturation detection unit that detects the presence or absence of saturated pixels in the thermal image data that satisfy a predetermined condition; a shutter control unit that controls the operation of a shutter that blocks light from entering an infrared sensor provided in the infrared camera; and a display control unit that displays the thermal image data acquired by the thermal image data acquisition unit on a display device, wherein when the saturation detection unit detects a saturated pixel that satisfies the predetermined condition, the shutter control unit closes the shutter and controls the shutter to open every predetermined frame, and the display control unit displays the thermal image of the frame immediately before the shutter is closed on the display device during the period when the shutter control unit closes the shutter.
[0007] The present disclosure also provides an image processing method in which an image processing device executes the following steps: a thermal image data acquisition step of acquiring thermal image data consisting of a plurality of frames from an infrared camera that captures thermal images; a saturation detection step of detecting the presence or absence of saturated pixels in the thermal image data that satisfy a predetermined condition; a shutter control step of closing a shutter that blocks light from entering an infrared sensor provided in the infrared camera and controlling the shutter to open for each predetermined frame if a saturated pixel that satisfies the predetermined condition is detected in the saturation detection step; and a display control step of displaying the thermal image data acquired in the thermal image data acquisition step on a display device, and displaying the thermal image of the frame immediately before the shutter is closed on the display device during a period in which the shutter is closed. Effect of the Invention
[0008] The present disclosure makes it possible to provide an image processing device and an image processing method that are capable of recognizing vehicles, people, and the like while protecting a sensor when exposed to sunlight. [Brief description of the drawings]
[0009] [Figure 1]1 is a block diagram illustrating a configuration of an imaging system according to a first embodiment. [Diagram 2] 2 is a block diagram illustrating a schematic example of a detailed configuration of an infrared camera in the imaging system according to the first embodiment. FIG. [Diagram 3] 2 is a block diagram illustrating the configuration of an image processing unit according to the first embodiment. FIG. [Figure 4] FIG. 2 is a conceptual diagram illustrating an example of acquired thermal imaging data and output thermal imaging data under normal conditions in the image processing device according to the first embodiment. [Diagram 5] 4 is a conceptual diagram illustrating an example of acquired thermal imaging data and output thermal imaging data when detecting sunlight in the image processing device according to the first embodiment. FIG. [Figure 6] FIG. 2 is a flowchart illustrating an image processing method according to the first embodiment. [Figure 7] FIG. 11 is a block diagram illustrating the configuration of an imaging system according to a second embodiment. [Figure 8] FIG. 11 is a diagram showing an example of display of thermal imaging data in which a predetermined object is recognized by an image recognition unit in the image processing device according to the second embodiment. [Figure 9] FIG. 11 is a flow chart illustrating an image processing method according to the second embodiment. [Figure 10] FIG. 11 is a block diagram illustrating the configuration of an imaging system according to a third embodiment. [Figure 11] FIG. 11 is a flowchart illustrating an image processing method according to a third embodiment. [Figure 12] FIG. 11 is a block diagram illustrating the configuration of an imaging system according to a fourth embodiment. [Figure 13] FIG. 11 is a flow chart illustrating an image processing method according to a fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings. In each drawing, the same or corresponding elements are denoted by the same reference numerals, and for clarity of explanation, duplicate explanations are omitted as necessary. In addition, some reference numerals are omitted so as not to make the drawings complicated.
[0011] (Embodiment 1) An image processing device and an imaging system according to the first embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram illustrating a configuration of an imaging system 1 according to the first embodiment. As shown in Fig. 1, the imaging system 1 includes an image processing device 101, an infrared camera 200, and a display device 300. The imaging system 1 is, for example, an imaging system 1 for vehicles that is used in a vehicle and acquires a thermal image of the surroundings, an imaging system 1 used for surveillance purposes, etc.
[0012] The infrared camera 200 captures a thermal image (infrared image). The infrared camera 200 has an infrared sensor. The configuration of the infrared camera 200 will be described later. The infrared camera 200 captures a thermal image by detecting infrared rays from a subject with an infrared sensor. The infrared camera 200 is a far infrared ray camera that can visualize far infrared rays emitted from an object. Data of a thermal image captured by the infrared camera 200 is also called thermal imaging data (or infrared image data). The thermal imaging data is a moving image composed of multiple frames, for example, 15 to 60 frames per second.
[0013] The infrared camera 200 outputs the thermal imaging data to the image processing device 101. Note that the thermal imaging data acquired by the image processing device 101 from the infrared camera 200 is also referred to as acquired thermal imaging data. The thermal imaging data refers to the thermal imaging data captured by the infrared camera.
[0014] The image processing device 101 generates display image data by performing image processing on the acquired thermal imaging data. For example, the image processing device 101 performs various processes such as correction of pixel variations of the infrared camera 200.
[0015] The image processing device 101 outputs the display image data that has been subjected to image processing to the display device 300. The display image data that has been subjected to image processing by the image processing device 101 and output to the display device 300 is also referred to as output thermal image data. In other words, the acquired thermal image data is data before processing by the image processing device 101, and the output thermal image data is data after processing.
[0016] The display device 300 includes a display for displaying display image data, etc. The display device 300 displays the display image data that has been subjected to image processing in the image processing device 101.
[0017] The image processing device 101 includes, as physical components, a control IF (interface) 110, a ROM (Read Only Memory) 120, and a RAM (Random Access Memory) 130. The image processing device 101 further includes, as functional blocks realized by programs as processing of a CPU (Central Processing Unit) or a GPU (Graphics Processing Unit), a system control unit 140, a shutter control unit 150, a thermal imaging data acquisition unit 160, an image processing unit 170, a saturation detection unit 173, and a display control unit 180. For convenience, these components are shown as being connected to be able to communicate with each other via a bus.
[0018] The thermal imaging data acquisition unit 160 acquires thermal imaging data consisting of multiple frames from the infrared camera 200 that captures thermal images. The thermal imaging data acquisition unit 160 is provided with, for example, an IF (interface) for acquiring the thermal imaging data from the infrared camera 200. The interface between the thermal imaging data acquisition unit 160 and the infrared camera 200 may be a wired connection or a wireless connection.
[0019] The ROM 120 stores a control program and various parameters for controlling the image processing device 101. For example, the ROM 120 stores an image processing program executed by the image processing unit 170. The ROM 120 also stores a system control program executed by the system control unit 140.
[0020] The RAM 130 stores various programs and parameters used for executing the programs, etc. Furthermore, it stores calculation data of the image processing unit 170, etc.
[0021] The system control unit 140 controls the entire imaging system 1. The processor of the system control unit 140 executes a control program to control the entire system.
[0022] The system control unit 140 includes a shutter control unit 150. The shutter control unit 150 controls the opening and closing of the shutter of the infrared camera 200. Specifically, the shutter control unit 150 controls the operation of the shutter that blocks light from entering an infrared sensor included in the infrared camera 200. The shutter control unit 150 outputs a control signal for opening and closing the shutter.
[0023] The control IF 110 is an interface for controlling the infrared camera 200. For example, the shutter control unit 150 outputs a control signal for opening and closing the shutter of the infrared camera 200 via the control IF 110. Furthermore, the image processing device 101 can transmit various signals and data to the infrared camera 200 via the control IF 110. Furthermore, the control IF 110 can receive various signals, data, and the like from the infrared camera 200. For example, the control IF 110 acquires the environmental temperature measured by a temperature sensor of the infrared camera 200.
[0024] The image processing unit 170 performs predetermined image processing on the thermal imaging data acquired by the thermal imaging data acquisition unit 160 .
[0025] The image processing unit 170 includes a saturation detection unit 173. The saturation detection unit 173 detects the presence or absence of saturated pixels in the thermal imaging data that satisfy a predetermined condition. A saturated pixel is, for example, a pixel whose pixel data is in a state where the pixel data is at an upper limit value. For example, when the brightness level of each pixel in the imaging data is expressed in 14 bits from zero to 16383, a pixel at a coordinate having a brightness level of 16383 may be regarded as a saturated pixel. Note that the predetermined condition for a saturated pixel is not limited to being the upper limit value, and may be another value set in advance.
[0026] The saturation detection unit 173 may detect the presence or absence of saturated pixels corresponding to direct incidence of sunlight as the presence or absence of saturated pixels satisfying a predetermined condition. The saturation detection unit 173 may detect the presence or absence of saturated pixels that cause damage to pixels as the presence or absence of saturated pixels satisfying a predetermined condition.
[0027] The processing in the image processor 170 will be described later. The image processor 170 performs image processing on the acquired thermal imaging data to generate output thermal imaging data.
[0028] The display control unit 180 causes the display device 300 to display the thermal imaging data acquired by the thermal imaging data acquisition unit 160. Specifically, the display control unit 180 causes the display device 300 to display the acquired thermal imaging data acquired by the thermal imaging data acquisition unit 160 after the data is processed by the image processing unit 170. The display control unit 180 is an interface that outputs the output thermal imaging data to the display device 300.
[0029] 2 is a block diagram illustrating a schematic example of a detailed configuration of the infrared camera 200 in the imaging system 1 according to embodiment 1. As shown in FIG. 2, the infrared camera 200 includes a lens 210, a shutter 220, an infrared sensor 230, a transmission device 240, and a temperature sensor 250.
[0030] The lens 210 forms an image of infrared light from the subject on the light receiving surface of the infrared sensor 230. The infrared sensor 230 includes a plurality of pixels. Each pixel of the infrared sensor 230 receives infrared light from the subject. This allows a thermal image of the subject to be captured. For example, the infrared sensor 230 may include a microbolometer for detecting far-infrared rays. The infrared sensor 230 includes a plurality of pixels arranged in a two-dimensional array. The detection values (detection signals) of each pixel form a thermal image of the subject.
[0031] The transmission device 240 serves as an interface for transmitting various signals and data to the image processing device 101. The transmission device 240 transmits thermal imaging data captured by the infrared sensor 230 to the image processing device 101. In addition, the transmission device 240 receives a control signal from the image processing device 101.
[0032] The temperature sensor 250 measures the temperature in the environment in which the infrared camera 200 is used. The transmission device 240 transmits the environmental temperature measured by the temperature sensor 250 to the image processing device 101.
[0033] The shutter 220 is disposed on the front side of the infrared sensor 230. The shutter 220 blocks light, including infrared light, that enters the infrared sensor 230 from the outside. The shutter 220 has a mechanism that can be opened and closed. The shutter 220 opens and closes by a control signal from the shutter control unit 150. When the shutter 220 is closed (hereinafter, also simply referred to as the closed state), infrared light from the lens 210 does not enter the infrared sensor 230. That is, in the closed state, infrared light from the shutter 220, not from the lens 210, enters the infrared sensor 230. When the shutter 220 is open (hereinafter, also simply referred to as the open state), infrared light from the lens 210 enters the infrared sensor 230. The shutter 220 can make the imaging range of the infrared sensor 230 a thermally uniform surface. For example, the shutter 220 may be something like a barrier or a cover disposed on the front side of the lens 210.
[0034] Next, the configuration of the image processing unit 170 will be described with reference to Fig. 3. Fig. 3 is a block diagram illustrating the configuration of the image processing unit 170 according to the first embodiment. As shown in Fig. 3, the image processing unit 170 may further include a defective pixel correction unit 171 and a NUC (Non-Uniformity Correction) unit 172 in addition to a saturation detection unit 173. The image processing unit 170 cooperates with the system control unit 140 and the RAM 130 to execute predetermined processing.
[0035] The defective pixel correction unit 171 performs an interpolation process on the pixel value of a defective pixel of the infrared sensor 230, which is pre-stored in the RAM 130, from the pixel values of pixels surrounding the defective pixel. The defective pixel correction unit 171 performs the above-mentioned interpolation process on the acquired thermal imaging data acquired by the thermal imaging data acquisition unit 160, and outputs the image data after the interpolation process to the NUC unit 172.
[0036] The NUC unit 172 performs processing to correct output variations between pixels.
[0037] As described above, the saturation detection unit 173 detects saturated pixels in the thermal imaging data. When the saturation detection unit 173 detects saturated pixels in the acquired thermal imaging data, the saturation detection unit 173 outputs a signal indicating the detection result of the saturated pixels to the system control unit 140.
[0038] Next, the operation of the shutter control unit 150, the display control unit 180, etc. in the image processing device 101 according to the present embodiment will be described. FIG. 4 is a conceptual diagram illustrating the acquired thermal imaging data and the output thermal imaging data in the image processing device 101 according to the first embodiment in a normal state. As shown in FIG. 4, in a normal state, the thermal imaging data acquisition unit 160 in the image processing device 101 acquires thermal imaging data including a plurality of frames F1 to F30 from the infrared camera 200 that captures a thermal image. The normal state includes, for example, a state in which the infrared sensor 230 does not detect direct incidence of sunlight. Therefore, the normal state includes a state in which the saturation detection unit 173 does not detect a saturated image caused by direct incidence of sunlight. In a normal state, the display control unit 180 causes the display device 300 to display the acquired thermal imaging data acquired by the thermal imaging data acquisition unit 160 as output thermal pixel data.
[0039] Fig. 5 is a conceptual diagram illustrating acquired thermal imaging data and output thermal imaging data when detecting sunlight in the image processing device 101 according to the first embodiment. As shown in Fig. 5, when detecting direct incidence of sunlight, that is, when the saturation detection unit 173 detects a saturated pixel that satisfies a predetermined condition, the shutter control unit 150 controls to close the shutter 220. At the same time, the shutter control unit 150 controls to open the shutter 220 every predetermined number of frames. The predetermined number of frames is, for example, every 5 frames.
[0040] Specifically, when saturation detection unit 173 detects a saturated pixel due to direct incidence of sunlight, shutter control unit 150 closes shutter 220 and opens shutter 220 every five frames. For example, shutter control unit 150 opens shutter 220 when acquiring frame F1, and closes shutter 220 when acquiring frames F2 to F5. Shutter control unit 150 opens shutter 220 when acquiring frame F6, and closes shutter 220 when acquiring frames F7 to F10. Shutter control unit 150 opens shutter 220 when acquiring frame F11, and closes shutter 220 when acquiring frames F12 to F15.
[0041] In this way, when the saturation detection unit 173 detects a saturated pixel due to direct incidence of sunlight, the shutter control unit 150 repeats an opening and closing operation including an open state of the shutter 220 over a predetermined number of frames and a closed state of the shutter 220 over a predetermined number of frames. For example, in FIG. 5, the number of frames in which the shutter 220 is open in one opening and closing operation is one frame, and the number of frames in which the shutter 220 is closed in one opening and closing operation is four frames. The number of frames in one opening and closing operation is five frames. Therefore, when the saturation detection unit 173 detects a saturated pixel due to direct incidence of sunlight, the shutter control unit 150 repeats the opening and closing operation in a cycle of five frames until a saturated pixel due to direct incidence of sunlight is no longer detected.
[0042] The display control unit 180, during a period in which the shutter control unit 150 opens the shutter 220, causes the display device 300 to display a thermal image of a frame acquired through the open shutter 220. On the other hand, during a period in which the shutter control unit 150 closes the shutter 220, the display control unit 180 causes the display device 300 to display a thermal image of a frame immediately before the shutter 220 is closed.
[0043] Specifically, when acquiring frame F1, the shutter control unit 150 opens the shutter 220. The display control unit 180 causes the display device 300 to display a thermal image of frame F1 acquired through the open shutter 220. When acquiring frames F2 to F5, the shutter control unit 150 closes the shutter 220. The display control unit 180 causes the display device 300 to display a thermal image of frame F1 immediately before the shutter 220 is closed, during the period of frames F2 to F5 in which the shutter control unit 150 closes the shutter 220.
[0044] When acquiring frame F6, the shutter control unit 150 opens the shutter 220. The display control unit 180 causes the display device 300 to display the thermal image of frame F6 acquired through the open shutter 220. When acquiring the thermal image of frame F6, the display control unit 180 detects saturated pixels due to direct incidence of sunlight in the thermal image of frame F6. If saturated pixels due to direct incidence of sunlight are detected in the thermal image of frame F6, the display control unit 180 repeats the opening and closing operation again in a cycle of five frames. Therefore, when acquiring frames F7 to F10, the shutter control unit 150 closes the shutter 220. During the period of frames F7 to F10 when the shutter control unit 150 closes the shutter 220, the display control unit 180 causes the display device 300 to display the thermal image of frame F6 immediately before the shutter 220 is closed. If saturated pixels due to direct incidence of sunlight are not detected in the thermal image of frame F6, the display control unit 180 does not perform the process of closing the shutter 220. As a result, the opening and closing operation of the shutter 220 is repeated during the period in which the saturation detection unit 173 detects direct incidence of sunlight.
[0045] In this way, during the period when saturation detection unit 173 detects saturated pixels due to direct incidence of sunlight, display control unit 180 repeats the open / close display operation including displaying frames acquired through shutter 220 open for a predetermined number of frames and displaying frames acquired immediately before shutter 220 closes for a predetermined number of frames. For example, in FIG. 5, the number of frames in which shutter 220 is open in one open / close display operation is 1 frame, and the number of frames in which shutter 220 is closed in one open / close display operation is 4 frames. The number of frames in one open / close display operation is 5 frames. Therefore, display control unit 180 repeats the open / close display operation in a cycle of 5 frames during the period when saturation detection unit 173 detects saturated pixels due to direct incidence of sunlight.
[0046] Next, the image processing method according to this embodiment will be described with reference to FIG. 6. In the image processing method according to this embodiment, the image processing device 101 executes the following steps. FIG. 6 is a flow chart illustrating the image processing method according to the first embodiment. As shown in FIG. 6, first, the system control unit 140 causes the thermal image data acquisition unit 160 to start acquiring thermal image data (step S101). The start of acquiring thermal image data shown in step S101 is started when the image processing device 101 is mounted on a vehicle, when the vehicle becomes operable, or by a user operation. The image processing method according to this embodiment executes a thermal image data acquisition step of acquiring thermal image data consisting of a plurality of frames from an infrared camera 200 that captures thermal images.
[0047] Next, the system control unit 140 causes the saturation detection unit 173 to start detecting saturated pixels caused by direct incidence of sunlight (step S102). Next, the system control unit 140 causes the display control unit 180 to start normal display of the thermal imaging data (step S103).
[0048] Next, the system control unit 140 judges whether to end the display of the thermal imaging data (step S104). The end of display shown in step S104 may be rephrased as the end of acquisition of the thermal imaging data. For example, when the image processing device 101 is mounted on a vehicle, the end occurs when the vehicle operation is terminated or by a user operation. When the display of the thermal imaging data is to be ended (Yes in step S104), the system control unit 140 causes the display control unit 180 to end the display of the thermal imaging data, or causes the thermal imaging data acquisition unit 160 to end the acquisition of the thermal imaging data. The system control unit 140 may end the operation of the image processing device 101. When the display of the thermal imaging data is not to be ended (No in step S104), the system control unit 140 judges whether the saturation detection unit 173 detects a saturated pixel (step S105). Therefore, the image processing method of this embodiment executes a saturation detection step of detecting the presence or absence of saturated pixels in the thermal imaging data that satisfy a predetermined condition. In step S105, the presence or absence of saturated pixels corresponding to direct incidence of sunlight is detected as the "presence or absence of saturated pixels that satisfy a predetermined condition". In other words, the presence or absence of saturated pixels that cause damage to pixels is detected.
[0049] If the saturation detection unit 173 does not detect any saturated pixels due to direct incidence of sunlight (No in step S105), the process returns to step S104 and repeats the process. In this case, as shown in FIG. 4, if no saturated pixels of the predetermined condition are detected, the output thermal imaging data corresponding to each frame of the acquired thermal imaging data is displayed on the display device 300. On the other hand, if the saturation detection unit 173 detects any saturated pixels due to direct incidence of sunlight (Yes in step S105), the system control unit 140 controls the shutter control unit 150 to close the shutter 220 (step S106). Then, the system control unit 140 controls the display control unit 180 to start displaying the thermal image of the frame immediately before the shutter 220 is closed (step S107). In step S107, for example, the thermal image of frame F1 immediately before the shutter 220 is closed is displayed during the shooting period of frames F2 to F5, as shown in FIG. 5.
[0050] Next, the system control unit 140 determines whether or not a predetermined number of frames have been photographed with the shutter 220 closed (step S108). If the predetermined number of frames have not been photographed (No in step S108), the system control unit 140 repeats step S108 until the predetermined number of frames have been photographed. If the predetermined number of frames have been photographed (Yes in step S108), the system control unit 140 controls the shutter control unit 150 to open the shutter 220 (step S109), and the process proceeds to step S105.
[0051] Therefore, in the image processing method of this embodiment, when a saturated pixel satisfying a predetermined condition is detected in the saturation detection step, the shutter 220 that blocks light from entering the infrared sensor 230 of the infrared camera 200 is closed, and a shutter control step is executed to control the shutter 220 to open for each predetermined frame. The image processing method of this embodiment also executes a display control step to display the thermal imaging data acquired in the thermal imaging data acquisition step on the display device 300, and to display, during the period when the shutter 220 is closed, the thermal image of the frame immediately before the shutter 220 is closed on the display device 300.
[0052] In step S109, the predetermined frame shooting period may be set to, for example, one frame, as shown in Fig. 5. Then, the process returns to step S105 and repeats. In step S105, if a saturated pixel that satisfies the predetermined condition is detected during the shooting period of frame F6, the same process is repeated.
[0053] Next, the effects of this embodiment will be described. In the image processing device 101 of this embodiment, when the saturation detection unit 173 detects a saturated pixel, the image processing device 101 closes the shutter 220 and opens the shutter 220 for each predetermined number of frames. During the period in which the shutter 220 is closed, the display control unit 180 causes the display device 300 to display a thermal image of the frame immediately before the shutter 220 is closed. Therefore, when exposed to sunlight, the infrared sensor 230 is protected, and the user can recognize vehicles, people, and the like displayed on the display device 300.
[0054] Specifically, for example, when the sun is close to the horizon and a person is standing with the sun in the background, if an image is taken in this state, the infrared sensor 230 will be burned in due to the influence of sunlight. Therefore, in this embodiment, the shutter 220 of the infrared camera 200 is closed and then opened every predetermined number of frames, repeating the open and closed states. This makes it possible to prevent the infrared sensor 230 from being burned in due to sunlight, and allows the user to recognize people.
[0055] (Embodiment 2) Next, an image processing device and an imaging system according to embodiment 2 will be described. Fig. 7 is a block diagram illustrating the configuration of an imaging system 2 according to embodiment 2. As shown in Fig. 7, in the imaging system 2, the image processing device 102 further includes an image recognition unit 190. The image recognition unit 190 detects a predetermined object from the thermal imaging data.
[0056] The image recognition unit 190 refers to a learning model that has been machine-learned using thermal imaging data in which a specified object has been photographed, and detects the specified object by determining that the specified object has been photographed in a section of the acquired thermal imaging data where the threshold value indicating the likelihood of the specified object being the specified object is equal to or greater than a specified value.
[0057] FIG. 8 is a diagram illustrating a case where the image recognition unit 190 in the image processing device 102 according to the second embodiment displays thermal imaging data in which a predetermined object is recognized on the display device 300. As shown in FIG. 8, the image recognition unit 190 detects another vehicle 90 traveling in the thermal imaging data in a thermal image of a frame F1. The image recognition unit 190 may output the detected part of the other vehicle 90 to the system control unit 140. The system control unit 140 causes the display control unit 180 to display the part including the other vehicle 90 surrounded by a detection frame 91. The image recognition unit 190 may detect a person (pedestrian, person riding a bicycle) as the predetermined object, in addition to the other vehicle 90.
[0058] When the shutter 220 is open, the image recognition unit 190 detects a predetermined object from a thermal image of a frame acquired through the open shutter 220. On the other hand, as described above, when the saturation detection unit 173 detects a saturated pixel due to direct incidence of sunlight, the shutter control unit 150 performs control to close the shutter 220 and open the shutter 220 for each predetermined frame. In accordance with this control, the image recognition unit 190 detects a predetermined object from a thermal image of a frame immediately before the shutter 220 is closed while the shutter control unit 150 closes the shutter 220.
[0059] When displaying a thermal image of a previous frame on the display device 300 during a shutter closed period when the shutter 220 is closed, the following two methods may be used. That is, the image recognition unit 190 may perform image recognition for each identical thermal image in succession, and the display control unit 180 may display the detection result. Alternatively, the display control unit 180 may display the detection result detected by the image recognition unit 190 in the previous frame during the shutter closed period.
[0060] Next, an image processing method of this embodiment will be described. Fig. 9 is a flow chart illustrating an image processing method according to embodiment 2. Steps S201 and S202 in Fig. 9 are similar to steps S101 and S102 in embodiment 1 shown in Fig. 6, and therefore will not be described.
[0061] After step S202, the system control unit 140 causes the image recognition unit 190 to start detecting a predetermined object (step S203). Next, the display control unit 180 starts normal display of the detection result of the predetermined object recognized by the image recognition unit 190 in addition to the thermal pixel data acquired by the thermal imaging data acquisition unit 160 (step S204).
[0062] Next, the system control unit 140 judges whether or not to end the display of the thermal imaging data and the detection results (step S205). If the display of the thermal imaging data and the detection results is to be ended (Yes in step S205), the system control unit 140 causes the display control unit 180 to end the display of the thermal imaging data and the detection results. If the display of the thermal imaging data and the detection results is not to be ended (No in step S205), the system control unit 140 proceeds to step S206. Steps S206 and S207 in FIG. 9 are the same processes as steps S105 and S106 in the first embodiment shown in FIG. 6, and therefore the description thereof will be omitted. However, if the result in step S206 is No, the process returns to step S205.
[0063] Next, the system control unit 140 causes the display control unit 180 to start displaying the thermal image and detection result of the frame immediately before the shutter 220 is closed (step S208). In step S208, as shown in Fig. 5, for example, the thermal image of frame F1 immediately before the shutter 220 is closed is displayed during the shooting period of frames F2 to F5. Also, if a predetermined object is detected in frame F1 immediately before the shutter 220 is closed, a thermal image with a detection frame 91 added is displayed during the shooting period of frames F2 to F5. If a predetermined object is not detected in frame F1 immediately before the shutter 220 is closed, a thermal image without a detection frame 91 added is displayed during the shooting period of frames F2 to F5.
[0064] Steps S209 and S210 in Fig. 9 are similar to steps S108 and S109 in the first embodiment shown in Fig. 6, and therefore the description thereof will be omitted. However, if the result of step S209 is No, the process returns to step S209. After step S210, the process returns to step S206.
[0065] Next, the effects of this embodiment will be described. In this embodiment, the image recognition unit 190 detects a predetermined object from a thermal image of a frame immediately before the shutter 220 is closed while the shutter control unit 150 closes the shutter 220. Therefore, while protecting the infrared sensor 230, the user of the image processing device 102 can recognize people, other vehicles 90, etc., even while the shutter 220 is closed.
[0066] In the related imaging systems so far, when sunlight enters the angle of view, the infrared sensor 230 is burned in, so the shutter 220 is closed. Therefore, when the sunlight is close to the horizon, the infrared camera 200 cannot capture an image of a person or the like even if the person or the like is present, and is therefore unable to recognize the person or the like. In contrast, in this embodiment, even during the period when the shutter 220 is closed, the shutter 220 opens for a predetermined number of frames, so that although the image is like a frame-by-frame video, the image recognition unit 190 can recognize the person or the like. Therefore, when the imaging system 2 is mounted on a moving body such as a vehicle, a warning to avoid a collision with a person or the like and the user can recognize the person or the like can be issued.
[0067] (Embodiment 3) Next, an imaging system according to the third embodiment will be described. FIG. 10 is a block diagram illustrating the configuration of the imaging system according to the third embodiment. As shown in FIG. 10, the imaging system 3 of this embodiment further includes a prediction unit 151. Specifically, the system control unit 140 includes the prediction unit 151. The prediction unit 151 may be a function of the image recognition unit 190. The prediction unit 151 predicts the position of a predetermined object detected by the image recognition unit 190. For example, the prediction unit 151 predicts the position of the detected predetermined object from the movement transition for each frame. The prediction unit 151 predicts the position of the predetermined object during the period when the shutter 220 is closed from the transition of the position of the predetermined object detected before the shutter 220 is closed.
[0068] When a predetermined object detected before the shutter 220 is closed moves at a constant speed in a predetermined direction for each frame, the prediction unit 151 predicts that the object will move at a constant speed in the same direction even during the period when the shutter 220 is closed. When a predetermined object detected before the shutter 220 is closed moves at a predetermined acceleration / deceleration in each frame in a predetermined direction, the prediction unit 151 predicts that the object will move at the same acceleration / deceleration in the same direction even during the period when the shutter 220 is closed. Furthermore, the prediction unit 151 predicts the movement direction of the predetermined object during the period when the shutter 220 is closed based on the movement direction of the predetermined object before the shutter 220 is closed.
[0069] During the period in which the shutter control unit 150 closes the shutter 220, the display control unit 180 performs display showing the position predicted by the prediction unit 151 for the predetermined object recognized immediately before the shutter 220 is closed.
[0070] Next, an image processing method of this embodiment will be described. Fig. 11 is a flow chart illustrating an image processing method according to embodiment 3. Steps S301 and S302 in Fig. 11 are similar to steps S201 and S202 in embodiment 2 shown in Fig. 9, and therefore will not be described.
[0071] After step S302, the image recognition unit 190 starts detecting a predetermined object, and the prediction unit 151 starts predicting the position of the detected predetermined object (step S303). The processes from step S304 to step S307 in Fig. 11 are similar to the processes from step S204 to step S207 in the second embodiment shown in Fig. 9, and therefore the description will be omitted. However, if the result of step S306 is No, the process returns to step S305.
[0072] In step S307, when the shutter 220 is closed, the system control unit 140 causes the display control unit 180 to start displaying the thermal image of the frame immediately before the shutter 220 is closed and the predicted movement position of the predetermined object (step S308). In step S308, as shown in Fig. 5, for example, the thermal image of the frame F1 immediately before the shutter 220 is closed is displayed during the shooting period of frames F2 to F5. Also, if a predetermined object is detected in the frame F1 immediately before the shutter 220 is closed, a thermal image with a detection frame 91 indicating the predicted position of the predetermined object predicted from the position of the predetermined object detected in the frame F1 immediately before the shutter 220 is closed and the frame before that is displayed during the shooting period of frames F2 to F5.
[0073] Steps S309 and S310 in Fig. 11 are the same as steps S209 and S210 in the second embodiment shown in Fig. 9, so the description will be omitted. However, if the result of step S309 is No, the process returns to step S309. After step S310, the process returns to step S306.
[0074] Next, the effect of this embodiment will be described. In this embodiment, the prediction unit 151 predicts the position of a predetermined object detected by the image recognition unit 190. Furthermore, during the period when the shutter control unit 150 closes the shutter 220, a detection frame or the like is displayed indicating a predicted position from the position of the predetermined object recognized immediately before the shutter 220 is closed. This allows the user of the image processing device 103 to recognize the predicted destination of a person, another vehicle 90, etc., even during the period when the shutter 220 is closed, while protecting the infrared sensor 230. Therefore, when the imaging system 3 is mounted on a moving body such as a vehicle, a warning to avoid a collision with a person, etc., and a user to confirm the position of a person, etc. can be issued.
[0075] (Embodiment 4) Next, an imaging system according to embodiment 4 will be described. The imaging system of this embodiment changes the period of "predetermined frame period imaging" in embodiments 1 to 3. For example, the system control unit 140 changes the period of "predetermined frame period imaging" depending on the moving state of a moving object on which the imaging system is installed.
[0076] Fig. 12 is a block diagram illustrating a configuration of an imaging system 4 according to a fourth embodiment. As shown in Fig. 12, in the imaging system 4 of this embodiment, an image processing device 104 is installed in a moving body such as a vehicle, and is connected to a CAN (Controller Area Network) 400. The CAN 400 can acquire information indicating the moving state of the moving body from a control device of the moving body or the like. The moving state of the moving body includes information such as the running speed and turning state of the moving body.
[0077] Moreover, the image processing device 104 further includes a movement information acquisition unit 152. Specifically, the system control unit 140 includes the movement information acquisition unit 152. The movement information acquisition unit 152 acquires the movement state of the moving object via the CAN 400.
[0078] The shutter control unit 150 of this embodiment changes the period during which the saturation detection unit 173 detects saturated pixels that satisfy predetermined conditions, and the period for each predetermined frame during which the shutter 220 is closed and then opened, depending on the movement state of the moving object acquired by the movement information acquisition unit 152.
[0079] This embodiment has the objective of preventing the position of the sun relative to the infrared sensor 230 from overlapping for each open state of the shutter 220. Therefore, for example, if the vehicle is turning, the position of the sun relative to the infrared sensor 230 changes in a short period of time. If the vehicle is stopped, sunlight is captured at the same position on the infrared sensor 230 when the shutter 220 is opened multiple times.
[0080] Therefore, for example, when a saturated pixel due to direct incidence of sunlight is detected, if the vehicle is stopped, the shutter control unit 150 closes the shutter for a period during which ten frames are captured, and opens the shutter for a period during which one frame is captured, thereby determining again whether or not there is a saturated pixel due to direct incidence of sunlight. For example, when pixel saturation due to direct incidence of sunlight is detected, if the vehicle is turning (when the steering angle is large, such as when turning at an intersection), the shutter is closed for a period during which three frames are captured, and opens the shutter for a period during which one frame is captured, thereby determining again whether or not there is a saturated pixel due to direct incidence of sunlight. In this way, in a moving state in which the change in the position of the sunlight incident on the infrared sensor 230 is large, the interval of the open state may be shortened (the period of the closed state may be shortened), and in a moving state in which the change in the position of the sunlight incident on the infrared sensor 230 is small, the interval of the open state may be lengthened (the period of the closed state may be lengthened).
[0081] The shutter control unit 150 may shorten the period of time between closing the shutter 220 and opening the shutter 220 for each predetermined frame as the moving speed of the moving object in the imaging angle of view of the infrared camera 200 increases. Here, the faster the moving speed of the moving object in the imaging angle of view increases, the faster and / or slower the traveling speed may be, or the presence or absence of movement in a turning direction may be, or a combination of both.
[0082] Next, an image processing method of this embodiment will be described. Fig. 13 is a flow chart illustrating an image processing method according to the fourth embodiment. Steps S401 and S402 in Fig. 13 are similar to steps S101 and S102 in the first embodiment shown in Fig. 6, and therefore will not be described.
[0083] After step S402, the movement information acquisition unit 152 starts acquiring the movement state of the moving object (S403).
[0084] Steps S404 to S410 are the same as steps S103 to S109 in the first embodiment shown in Fig. 6, and so a description thereof will be omitted. However, if No in step S406, the process proceeds to step S405, if No in step S409, the process proceeds to step S409, and after step S410, the process proceeds to step S406. Also, in steps S409 and S410, the shutter control unit 150 changes the period of time for each predetermined frame during which the shutter 220 is opened after closing the shutter 220, depending on the moving state of the moving object acquired by the movement information acquisition unit 152.
[0085] According to this embodiment, it is possible to prevent the positions of the sun relative to the infrared sensor 230 from overlapping for each open state of the shutter 220, thereby reducing burn-in of the infrared sensor 230. Other configurations and effects are included in the description of embodiments 1 to 3.
[0086] The present invention is not limited to the above-described embodiment, and can be modified as appropriate without departing from the spirit of the present invention. For example, a combination of any of the configurations of the first to fourth embodiments is also within the scope of the technical idea of the present embodiment. In addition, as described below, an image processing program that causes a computer to execute an image processing method is also within the scope of the technical idea of the embodiment.
[0087] (Appendix 1) A thermal image data acquisition step of acquiring thermal image data consisting of a plurality of frames from an infrared camera that captures thermal images; a saturation detection step of detecting the presence or absence of saturated pixels in the thermal image data that satisfy a predetermined condition; a shutter control step of closing a shutter that blocks light from entering an infrared sensor of the infrared camera and opening the shutter for each predetermined frame when the saturated pixel that satisfies the predetermined condition is detected in the saturation detection step; a display control step of displaying the thermal imaging data acquired in the thermal imaging data acquisition step on a display device, and displaying the thermal image of the frame immediately before the shutter is closed on the display device during a period in which the shutter is closed; An image processing program that causes a computer to execute the above. (Appendix 2) Further comprising an image recognition step of detecting a predetermined object from the thermal image data, In the image recognition step, during a period in which the shutter is closed, the predetermined object is detected from the thermal image of the frame immediately before the shutter is closed. 2. An image processing program according to claim 1. (Appendix 3) an image recognition step of detecting a predetermined object from the thermal image data; a prediction step of predicting a movement of the position of the predetermined object detected in the image recognition step; Further equipped with In the display control step, during a period in which the shutter is closed, a display is performed showing a predicted position of the predetermined object recognized immediately before the shutter is closed. 2. An image processing program according to claim 1. (Appendix 4) The method further includes a movement information acquisition step of acquiring a movement state of a moving object, In the shutter control step, when the saturated pixel satisfying the predetermined condition is detected in the saturation detection step, a period of time for each predetermined frame during which the shutter is opened after the shutter is closed is changed according to a moving state of the moving body acquired in the movement information acquisition step. 2. An image processing program according to claim 1.
[0088] Moreover, the above-mentioned program includes a set of instructions (or software code) for making the computer perform one or more functions described in the embodiment when the program is loaded into the computer. The program may be stored in a non-transitory computer-readable medium or a tangible storage medium. As an example, but not limited to, the computer-readable medium or the tangible storage medium includes a random-access memory (RAM), a read-only memory (ROM), a flash memory, a solid-state drive (SSD), or other memories or various storage devices. The program may be transmitted on a temporary computer-readable medium or a communication medium. [Explanation of symbols]
[0089] 1, 2, 3, 4 Imaging System 90 Other vehicles 91 Detection Frame 101, 102, 103, 104 Image processing device 110 Control IF 120 ROM 130 RAM 140 System control section 150 Shutter control section 151 Prediction Department 152 Movement information acquisition unit 160 Thermal image data acquisition unit 170 Image Processing Unit 171 Defective pixel correction section 172 NUC Department 173 Saturation detector 180 Display control section 190 Image Recognition Unit 200 Infrared Camera 210 Lens 210 220 Shutter 230 Infrared Sensor 240 Transmission Devices 250 Temperature Sensor 300 display device
Claims
1. A thermal image data acquisition unit that acquires thermal image data consisting of a plurality of frames from an infrared camera that captures thermal images; A saturation detection unit that detects the presence or absence of saturated pixels in the thermal image data that satisfy a predetermined condition; a shutter control unit that controls an operation of a shutter that blocks light from entering an infrared sensor included in the infrared camera; A display control unit that displays the thermal imaging data acquired by the thermal imaging data acquisition unit on a display device; Equipped with the shutter control unit performs control to close the shutter when the saturation detection unit detects the saturated pixel that satisfies the predetermined condition, and to open the shutter every predetermined frame; the display control unit causes the display device to display the thermal image of the frame immediately before the shutter is closed, during a period in which the shutter control unit closes the shutter; Image processing device.
2. An image recognition unit that detects a predetermined object from the thermal image data is further provided. the image recognition unit detects the predetermined object from the thermal image of the frame immediately before the shutter is closed, during a period in which the shutter control unit closes the shutter; The image processing device according to claim 1 .
3. an image recognition unit for detecting a predetermined object from the thermal image data; A prediction unit that predicts the movement of the position of the predetermined object detected by the image recognition unit; Further equipped with the display control unit performs a display indicating the position predicted by the prediction unit for the predetermined object recognized immediately before the shutter control unit closes the shutter, during a period in which the shutter control unit closes the shutter; The image processing device according to claim 1 .
4. The image processing device is installed in a mobile body, A moving information acquisition unit that acquires a moving state of the moving object, when the saturation detection unit detects the saturated pixel that satisfies the predetermined condition, the shutter control unit changes a period of time for each predetermined frame during which the shutter is opened after the shutter is closed, depending on a moving state of the moving object acquired by the movement information acquisition unit. The image processing device according to claim 1 .
5. A thermal image data acquisition step of acquiring thermal image data consisting of a plurality of frames from an infrared camera that captures thermal images; a saturation detection step of detecting the presence or absence of saturated pixels in the thermal image data that satisfy a predetermined condition; a shutter control step of closing a shutter that blocks light from entering an infrared sensor of the infrared camera and opening the shutter for each predetermined frame when the saturated pixel that satisfies the predetermined condition is detected in the saturation detection step; a display control step of displaying the thermal imaging data acquired in the thermal imaging data acquisition step on a display device, and displaying the thermal image of the frame immediately before the shutter is closed on the display device during a period in which the shutter is closed; The image processing device executes the above steps. Image processing methods.
Citation Information
Patent Citations
Control system, control method, and program
JP2022187078A
