Background updating device, method and program

Abstract

A background updating device 2 comprises a background updating unit that, if a prescribed condition is satisfied, obtains a thermal image at a target time as a new background thermal image, by using: (A) a thermal image which is an image of heat generated by a real object and which is obtained by imaging a certain imaging range at the target time with a thermal camera for imaging heat generated by a real object; (B) a real object image which is an image of the real object and which is obtained by imaging the certain imaging range at the target time with a real object camera for imaging a real object; (C) a background thermal image which is a thermal image of the background of the certain imaging range; and (D) a background real object image which is a real object image of the background of the certain range. The prescribed condition is a first condition in which the difference between the thermal image and the background thermal image is greater than or equal to a first prescribed amount, and the difference between the real object image and the background real object image is smaller than or equal to a second prescribed amount.

Description

[Technical Field]

[0001] The disclosed technology relates to a technology for acquiring a background image for obtaining a background difference. [Background technology]

[0002] Patent Document 1 describes a technology for extracting a thermal trace area by performing image processing using the background difference between a thermal image and an actual object image. A thermal trace is a trace of a subject's touch that is identified by the heat remaining at the location where the subject touched an object in the past, such as a person. In a technology for performing image processing using the background difference between a thermal image and an actual object image, as in Patent Document 1, it is important to obtain a background thermal image and a background actual object image with high accuracy.

[0003] Non-Patent Document 1 discloses a technique for obtaining a background image from a visible image with high accuracy. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2022 / 080350 [Non-patent literature]

[0005] [Non-Patent Document 1] Masaru Tsuchida, Takahito Kawanishi, Hiroshi Murase, Shigeru Takagi, "Background Estimation by Sequential Monte Carlo Method for Object Detection Using Background Subtraction," IEICE Transactions on Computer Vision, Vol. J87-D2, No. 5, pp. 1062-1070, May 1, 2004 Summary of the Invention [Problem to be solved by the invention]

[0006] Although Non-Patent Document 1 can accurately obtain a background image from a visible image, it has a problem of requiring a large amount of calculation processing. Furthermore, Non-Patent Document 1 does not anticipate obtaining a background image from a thermal image.

[0007] The purpose of the disclosed technology is to obtain at least one of a background image (background entity image) for an entity image such as a visible image and a background image (background thermal image) for a thermal image with less computational processing power than Non-Patent Document 1. [Means for solving the problem]

[0008] One aspect of the disclosed technology is a background updating device that includes a background updating unit that uses (A) a thermal image that is an image of heat emitted by a physical object obtained by photographing a certain shooting range at a target time with a thermal camera for photographing heat emitted by the physical object, (B) a physical object image that is an image of the physical object obtained by photographing a certain shooting range at a target time with a physical object camera for photographing the physical object, (C) a background thermal image that is a thermal image of the background of the certain shooting range, and (D) a background physical object image that is an image of the physical object in the background of the certain shooting range to obtain a thermal image of the target time as a new background thermal image when a predetermined condition is met, the predetermined condition being a first condition that the difference between the thermal image and the background thermal image at the target time is greater than or equal to a first predetermined amount, and a first condition that the difference between the physical object image and the background physical object image at the target time is less than or equal to a second predetermined amount.

[0009] One aspect of the disclosed technology is a background updating device that includes a background updating unit that uses (A) a thermal image that is an image of heat emitted by a physical object obtained by photographing a certain photographing range at a target time with a thermal camera for photographing heat emitted by the physical object, (B) a physical object image that is an image of the physical object obtained by photographing a certain photographing range at a target time with a physical object camera for photographing physical objects, (C) a background thermal image that is a thermal image of the background of the certain photographing range, and (D) a background physical object image that is an image of the physical object of the background of the certain photographing range to obtain a new background physical object image of the physical object at the target time when a predetermined condition is satisfied, wherein the predetermined condition is a first condition that the difference between the thermal image and the background thermal image at the target time is smaller than or equal to a first predetermined amount, and a first condition that the difference between the physical object image at the target time and the background physical object image is greater than or equal to a second predetermined amount. [Effects of the Invention]

[0010] According to the disclosed technology, it is possible to obtain at least one of a background image (background entity image) for an entity image such as a visible image and a background image (background thermal image) for a thermal image with less computational processing than Non-Patent Document 1. [Brief explanation of the drawings]

[0011] [Figure 1] FIG. 1 is a diagram illustrating an example of the functional configuration of the background updating device 2. As shown in FIG. [Figure 2] FIG. 2 is a diagram showing an example of the processing procedure of the background updating method and the image processing method. [Figure 3] FIG. 3 is a diagram showing an example of a procedure for the background update device 2 and the image processing device 3 to acquire an initial image of the background substance image and an initial image of the background thermal image. [Figure 4] FIG. 4 is a diagram showing an example of a processing procedure of the background update unit 25. As shown in FIG. [Figure 5] FIG. 5 is a diagram showing an example of the (3) background thermal image adjustment process of the background update unit 25. [Figure 6] FIG. 6 is a diagram showing an example of a processing procedure of the background update unit 25. [Figure 7]FIG. 7 is a diagram showing an example of a processing procedure of the background update unit 25. As shown in FIG. [Figure 8] FIG. 8 is a diagram showing an example of a processing procedure of the background update unit 25. [Figure 9] FIG. 9 is a diagram showing an example of a processing procedure of the background update unit 25. [Figure 10] FIG. 10 is a diagram showing an example of a processing procedure of the background update unit 25. [Figure 11] FIG. 11 is a diagram for explaining a specific example of the process of the background update unit 25. In FIG. [Figure 12] FIG. 12 is a diagram illustrating an example of the functional configuration of the image processing device 3. [Figure 13] FIG. 13 is a diagram illustrating an example of the functional configuration of the image processing unit 31. [Figure 14] FIG. 14 is a diagram illustrating an example of a functional configuration of a computer. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments of the disclosed technology will be described with reference to the drawings. Note that components having the same functions in the drawings are given the same reference numerals, and redundant description will be omitted.

[0013] [Background update device and method] 1, the background updating device 2 includes, for example, a real object image acquisition unit 21, an initial background real object image acquisition unit 22, a thermal image acquisition unit 23, an initial background thermal image acquisition unit 24, and a background updating unit 25. The background updating device 2 may further include a real object camera 11 and a thermal camera 13.

[0014] The background updating method is realized, for example, by each component of the background updating device 2 performing the processes of steps S11, S21, S13, S23, and S25 shown in FIG. 2 at each time (hereinafter referred to as "target time").

[0015] In addition, the process of acquiring the initial image of the background entity image and the initial image of the background thermal image used by the background updating device 2 in the process of step S25 is realized, for example, by each component of the background updating device 2 performing the processes of steps S11, S21, S13, S23, S22-1, S22-2, S24-1, and S24-2 shown in Figure 3.

[0016] The processing of each component of the background updating device 2 will be described below.

[0017] <Real Object Camera 11> The real object camera 11 is a camera for photographing real objects at each time after the start of operation of the background updating device 2. The real object camera 11 photographs images of real objects in a certain photographing range, more specifically, real object images which are images formed by images of real objects existing in the photographing range as seen from the real object camera 11 side, at each time (step S11). The real object images photographed by the real object camera 11 at each time are input to the real object image acquisition unit 21.

[0018] The wavelength band of the electromagnetic waves captured by the stereoscopic camera 11, in other words, the wavelength band of the electromagnetic waves acquired by the stereoscopic camera 11, may be any wavelength band capable of capturing an image of a real object. However, the wavelength band of the electromagnetic waves acquired by the stereoscopic camera 11 and the wavelength band of the electromagnetic waves acquired by the thermal camera 13 are set so as not to overlap. Here, the wavelength band refers to a range of wavelengths in which the intensity of the electromagnetic waves or visible light is equal to or greater than a predetermined intensity. For example, the wavelength band of the electromagnetic waves acquired by the stereoscopic camera 11 may be a near-infrared wavelength band (e.g., 780 nm to 2500 nm), and the wavelength band of the electromagnetic waves acquired by the thermal camera 13 may be a heat ray (far-infrared wavelength band) wavelength band of 8 μm to 14 μm. Alternatively, the lower limit of the wavelength band of the electromagnetic waves acquired by the stereoscopic camera 210 may be 600 nm, the wavelength of visible light, and the wavelength band of the electromagnetic waves acquired by the thermal camera 13 may be 8 μm to 14 μm.

[0019] In cases where the wavelength band of electromagnetic waves captured by the stereoscopic camera 11 does not include the wavelength band of electromagnetic waves present in the imaging range due to lighting or the like, an illuminator 12 may also be provided as shown by the dashed line in Fig. 1, and the imaging range may be irradiated with electromagnetic waves of the wavelength band captured by the stereoscopic camera 11 using the illuminator 12 that irradiates electromagnetic waves of the wavelength band captured by the stereoscopic camera 11. A real object is something that has a physical form. Examples of real objects present in the imaging range include objects present in the background of the imaging range, people who are within the imaging range, and people who have entered the imaging range.

[0020] <Actual object image acquisition unit 21> The entity image acquisition unit 21 operates at each time after the start of operation of the background updating device 2. The entity image acquisition unit 21 acquires entity images at each time photographed by the entity camera 11 (step S21). The acquired entity images at each time are input to the initial background entity image acquisition unit 22 until a predetermined time T1 has elapsed since the start of operation of the background updating device 2, and are input to the background updating unit 25 after the predetermined time T1 has elapsed since the start of operation of the background updating device 2.

[0021] <Initial background entity image acquisition unit 22> The initial background entity image acquisition unit 22 operates only when a predetermined time T1 has elapsed since the background updating device 2 started operating. When entity images input from the entity image acquisition unit 21 have been accumulated for the predetermined time T1, i.e., when the predetermined time T1 has elapsed since the background updating device 2 started operating (YES in step S22-1), the initial background entity image acquisition unit 22 acquires a background entity image, which is an entity image of the background of the shooting range, based on the entity images input from the entity image acquisition unit 21 for the predetermined time T1 (step S22-2), and outputs the acquired background entity image. The entity image of the background of the shooting range is an image of an object existing as the background within the shooting range, more specifically, an image composed of an image of an object existing as the background within the shooting range as viewed from the entity camera 11. The acquired background entity image is input to the background updating unit 25 as an initial image of the background entity image.

[0022] The initial background entity image acquisition unit 22 may acquire the background entity image using any method. For example, the initial background entity image acquisition unit 22 may acquire as the background entity image an image in which the pixel value of each pixel is the average or median of the pixel values ​​of corresponding pixels in a plurality of entity images over a predetermined time T1 from the start of operation of the background updating device 2, or may acquire as the background entity image a certain entity image within the predetermined time T1 from the start of operation of the background updating device 2. Furthermore, the initial background entity image acquisition unit 22 may accept input of a background entity image acquired by a device other than the background updating device 2 and output the input background entity image.

[0023] <Thermal Camera 13> The thermal camera 13 is a camera that captures images of heat emitted by actual objects at each time after the start of operation of the background updating device 2. The thermal camera 13 captures images of heat emitted by actual objects in the same capture range as the actual object camera 11, more specifically, thermal images that are images obtained from the thermal camera 13 of heat radiated by actual objects present in the capture range at each time (step S13). The thermal images captured by the thermal camera 13 at each time are input to the thermal image acquisition unit 23.

[0024] <Thermal image acquisition unit 23> The thermal image acquisition unit 23 operates at each time after the start of operation of the background updating device 2. The thermal image acquisition unit 23 acquires thermal images taken by the thermal camera 13 at each time (step S23). The acquired thermal images at each time are input to the initial background thermal image acquisition unit 24 until a predetermined time T1 has elapsed since the start of operation of the background updating device 2, and are input to the background updating unit 25 after the predetermined time T1 has elapsed since the start of operation of the background updating device 2. Each pixel value of the thermal image represents the temperature of each pixel in the thermal image.

[0025] <Initial background thermal image acquisition unit 24> The initial background thermal image acquisition unit 24 operates only when a predetermined time T1 has elapsed since the background update device 2 started operating. When the thermal images input from the thermal image acquisition unit 23 have been accumulated for the predetermined time T1, i.e., when the predetermined time T1 has elapsed since the background update device 2 started operating (YES in step S24-1), the initial background thermal image acquisition unit 24 acquires a background thermal image, which is a thermal image of the background of the shooting range, based on the thermal images input from the thermal image acquisition unit 23 for the predetermined time T1 (step S24-2), and outputs the acquired background thermal image. The background thermal image of the shooting range is an image of heat emitted by objects present as background in the shooting range, more specifically, an image composed of images captured by the thermal camera 13 of heat emitted by objects present as background in the shooting range. The acquired background thermal image is input to the background update unit 25.

[0026] The initial background thermal image acquisition unit 24 may acquire a background thermal image using any method. For example, the initial background thermal image acquisition unit 24 may acquire as the background thermal image an image in which the pixel value of each pixel is the average or median of the pixel values ​​of corresponding pixels of multiple thermal images taken over a predetermined time T1 from the start of operation of the background updating device 2, or may acquire as the background thermal image a certain thermal image taken within the predetermined time T1 from the start of operation of the background updating device 2. Furthermore, the initial background thermal image acquisition unit 24 may receive an input of a background thermal image acquired by a device other than the background updating device 2 and output the input background thermal image.

[0027] <Background update section 25> Background updating unit 25 operates at each target time after a predetermined time T1 has elapsed since the start of operation of background updating device 2. Since each target time can be said to be the current time when background updating unit 25 operates, "target time" in the following explanation may be read as "current time", and "at the target time" may be read as "at the current time" or "current".

[0028] The background update unit 25 receives the thermal image at the target time obtained by the thermal image acquisition unit 23 and the entity image at the target time obtained by the entity image acquisition unit 21. When a predetermined time T1 has elapsed since the background updating device 2 started operating, the background update unit 25 also receives the initial image of the background thermal image obtained by the initial background thermal image acquisition unit 24 and the initial image of the background entity image obtained by the initial background entity image acquisition unit 22.

[0029] The background update unit 25 performs at least one of (1) background thermal image replacement processing, (2) background entity image replacement processing, and (3) background thermal image adjustment processing as necessary for the target time (step S25). Note that (1) background thermal image replacement processing, (2) background entity image replacement processing, and (3) background thermal image adjustment processing are all convenient names given to the processing performed by the background update unit 25. In the following, (1) background thermal image replacement processing, (2) background entity image replacement processing, and (3) background thermal image adjustment processing may be collectively referred to as the "background update processing" for convenience.

[0030] When the background update unit 25 obtains a new background thermal image by performing either (1) background thermal image replacement processing or (3) background thermal image adjustment processing at the target time, it outputs the new background thermal image and sets the new background thermal image as the background thermal image for background update processing so that it can be used in the background update processing performed at a later time by the background update unit 25. However, when the background update unit 25 obtains a new background thermal image by performing (3) background thermal image adjustment processing at the target time, it may output the new background thermal image, but set the original background thermal image instead of the new background thermal image as the background thermal image for background update processing so that it can be used in the background update processing performed by the background update unit 25 at a later time.

[0031] When the background update unit 25 performs (2) the background entity image replacement process at the target time, it outputs a new background entity image and sets the new background entity image as the background entity image for the background update process so that the new background entity image can be used in the background update process performed by the background update unit 25 at a later time.

[0032] The new background thermal image and / or new background entity image output by the background update unit 25 becomes the output of the background update device 2. The new background thermal image and / or new background entity image output by the background update device 2 is input to another device including an image processing unit 31 that performs image processing. However, as will be described later, the background update device 2 may be included in the image processing device 3 that will be described later. When the background update device 2 is included in the image processing device 3 that will be described later, (1) the new background thermal image obtained by the background thermal image replacement process, (2) the new background entity image obtained by the background entity image replacement process, and (3) the new background thermal image obtained by the background thermal image adjustment process are input to the image processing unit 31 of the image processing device 3. The image processing unit 31 is indicated by a dashed line in FIG. 1.

[0033] The image processing unit 31 performs predetermined image processing using the most recent background entity image and the most recent background thermal image. That is, if the background update unit 25 has not yet obtained a new background entity image, the image processing unit 31 uses the background entity image obtained by the initial background entity image acquisition unit 22 (i.e., the initial image of the background entity image). If the background update unit 25 has already obtained one or more new background entity images, the image processing unit 31 uses the most recent background entity image among the new background entity images obtained by the background update unit 25. If the background update unit 25 has not yet obtained a new background thermal image, the image processing unit 31 uses the background thermal image obtained by the initial background thermal image acquisition unit 24 (i.e., the initial image of the background thermal image). If the background update unit 25 has already obtained one or more new background thermal images, the image processing unit 31 uses the most recent background thermal image among the new background thermal images obtained by the background update unit 25. The predetermined image processing will be described later.

[0034] Hereinafter, examples of when the background update unit 25 performs (1) background thermal image replacement processing, (2) background entity image replacement processing, and (3) background thermal image adjustment processing at the target time will be described with reference to FIG. 4. Then, examples of each of (1) background thermal image replacement processing, (2) background entity image replacement processing, and (3) background thermal image adjustment processing will be described. In the following description, the target time is set to t2, and a predetermined time T interval The explanation will be given assuming that the past time is t1.

[0035] The background update unit 25 updates the real object image I at time t1 in the past. t1 and the object image I at target time t2 t2 The difference d(I t1 ,I t2 ) is smaller than or equal to a predetermined amount ε1, and the thermal image J at the past time t1 t1 and thermal image J at target time t2 t2 The difference d(J t1 ,J t2 It is determined whether the predetermined time T interval is, for example, a few seconds. interval may be determined appropriately depending on the situation of use.

[0036] In Fig. 4, for the sake of simplicity, only "<" is used as a comparison operator for determining whether a value A is larger or smaller than a value B. Similarly, in other figures, only "<" and ">" are used as comparison operators.

[0037] An image P1 and an image P2 are composed of N pixels, and each pixel value of the image P1 is V1 n (n=1,...,N), and each pixel value of image P2 is V2 n (n=1,...,N). In this case, an example of the difference d(P1,P2) between image P1 and image P2 is Σ n=1 N |V1 n -V2 n |, Σ n=1 N |V1 n-V2 n | 2 Of course, other existing distance measure values ​​may be used as the difference d(P1, P2) between the image P1 and the image P2.

[0038] The background update unit 25 may determine whether the difference d(P1, P2) between the image P1 and the image P2 is less than or equal to a predetermined amount by using a pre-trained deep neural network.

[0039] In step S251, the difference d(I t1 ,I t2 ) is less than or equal to a predetermined amount ε1, and the difference d(J t1 ,J t2 ) is not determined to be less than or equal to the predetermined amount ε2 (i.e., the difference d(I t1 ,I t2 ) is equal to or greater than a predetermined amount ε1, and / or the difference d(J t1 ,J t2 If it is determined that ) is equal to or greater than the predetermined amount ε2, the background update unit 25 terminates the processing at the target time t2 without performing background update processing. The reason why no background update processing is performed in this case is that something may be temporarily captured in the object image or thermal image, and it is not appropriate to replace the background object image or background thermal image or adjust the background thermal image using the object image or thermal image in which something is temporarily captured.

[0040] The predetermined amounts ε1 and ε2 may be determined in advance, for example, through experiments, etc., so as to obtain the desired results. The same applies to other predetermined amounts, predetermined times, thresholds, etc., which must be determined in advance.

[0041] In step S251, the difference d(I t1 ,I t2 ) is less than or equal to a predetermined amount ε1, and the difference d(J t1 ,J t2 ) is smaller than or equal to the predetermined amount ε2, the background updating unit 25 updates the background entity image Ib and the object image I at time t2 t2 The difference d(I b ,I t2 It is determined whether the background object image I is smaller than a predetermined amount ε3 (step S252). b is the background entity image for background update processing currently set in the background update unit 25, and is the most recent background entity image; specifically, if the background update unit 25 has not yet obtained any new background entity image, it is the background entity image acquired by the initial background entity image acquisition unit 22 and input to the background update unit 25; if the background update unit 25 has already obtained one or more new background entity images, it is the background entity image obtained in the most recent processing among the new background entity images obtained by the background update unit 25.

[0042] In step S252, the difference d(I b ,I t2 ) is not determined to be less than or equal to the predetermined amount ε3 (i.e., the difference d(I b ,I t2 If it is determined that the background thermal image J is equal to or greater than the predetermined amount ε3, the background update unit 25 updates the background thermal image J b and thermal image J at time t2 t2 The difference d(J b ,J t2 It is determined whether the background thermal image J is smaller than or equal to a predetermined amount ε4 (step S253). b is the background thermal image for background update processing currently set in the background update unit 25, and is the most recent background thermal image for background update processing; specifically, if the background update unit 25 has not yet obtained any new background thermal image for background update processing, it is the background thermal image acquired by the initial background thermal image acquisition unit 24 and input to the background update unit 25; if the background update unit 25 has already obtained one or more new background thermal images for background update processing, it is the background thermal image for background update processing obtained in the most recent processing among the new background thermal images for background update processing obtained by the background update unit 25.

[0043] In step S253, the difference d(J b ,Jt2 If it is determined that the background image I is smaller than or equal to the predetermined amount ε4, the background update unit 25 performs (2) the process of replacing the background image (step S254). In this case, the process of replacing the background image I is performed by the background image I determined in step S252. b and real object image I t2 The reason for the substantial difference between the thermal image J and the background thermal image J is that, as a result of the determination in step S253, there is a person in the shooting range at the target time t2, and b The lighting in the shooting range has not changed since the image was acquired, and the background of the object in the shooting range has changed from the background object image I b This is because it is assumed that the time when the value of the parameter t1 is acquired differs from the target time t2.

[0044] In step S253, the difference d(J b ,J t2 ) is not determined to be less than or equal to the predetermined amount ε4 (i.e., the difference d(J b ,J t2 If it is determined that the background thermal image I is equal to or greater than the predetermined amount ε4, the background update unit 25 performs (3) the background thermal image adjustment process (step S255). In this case, the background thermal image adjustment process (3) is performed when the background thermal image I is equal to or greater than the predetermined amount ε4 as determined in step S252. b and real object image I t2 and the background thermal image J as determined in step S253. b and Thermal Image J t2 Although the cause of the substantial difference in the background thermal image J at target time t2 may be that there is a person or a thermal trace area in the shooting range at target time t2, b The background thermal image J may be affected by changes in the lighting in the shooting area since the time the thermal image was acquired. b This is because there is a possibility that the difference in the thermal radiation of the background of the photographed area between the time when the image was taken and the target time t2 may be included.

[0045] In step S252, the difference d(I b ,I t2If it is determined that the difference ε is smaller than or equal to the predetermined amount ε3, the background update unit 25 updates the background thermal image J b and thermal image J at time t2 t2 The difference d(J b ,J t2 ) is smaller than or equal to a predetermined amount ε4 (step S256).

[0046] In step S256, the difference d(J b ,J t2 ) is determined to be smaller than or equal to the predetermined amount ε4, the background update unit 25 ends the processing at the target time t2 without performing background update processing. The reason why no background update processing is performed in this case is that there is little need for background update from the perspective of the background regarding the actual objects in the shooting range and the background regarding the thermal radiation in the shooting range.

[0047] In step S256, the difference d(J b ,J t2 ) is not determined to be less than or equal to the predetermined amount ε4 (i.e., the difference d(J b ,J t2 If it is determined that the value of the thermal image J is equal to or greater than the predetermined value ε4, the background update unit 25 updates the thermal image J t2 It is determined whether the area of ​​the heat trace is included in the image (step S257).

[0048] In order to determine whether or not a region of a heat trace is included, the background update unit 25 may extract the region of a heat trace by performing processing of a differential substance image generation unit 311, a differential thermal image generation unit 312, and a heat trace extraction unit 313, which will be described later. t2 and background object image I b The background update unit 25 generates a differential entity image, which is a differential image between the thermal image J and the t2 and background thermal image J b Then, the background update unit 25 generates a differential thermal image J based on the differential actual object image generated by the differential actual object image generation unit 311 and the differential thermal image generated by the differential thermal image generation unit 312. t2For details of these processes, refer to the descriptions of the differential actual object image generation unit 311, differential thermal image generation unit 312, and thermal trace extraction unit 313.

[0049] In step S257, the thermal image J t2 If it is determined that the background thermal image does not include a thermal trace area, the background update unit 25 (1) performs a process of replacing the background thermal image (step S258). In this case, the (1) process of replacing the background thermal image is performed when the background entity image I is not included in the background thermal image as determined in step S252. b and real object image I t2 Although there is no substantial difference between the background thermal image J and the background thermal image J as determined in step S256, b and Thermal Image J t2 The substantial difference between the background thermal image J and the actual thermal image J is due to calibration, temperature changes in the shooting area, etc. b This is because it is assumed that this is due to the difference in the thermal radiation of the background of the shooting range between the time when the image was acquired and the target time t2.

[0050] In step S257, if it is not determined that the thermal image does not contain a region of a thermal trace, in other words, if it is determined that the thermal image contains a region of a thermal trace, the background update unit 25 performs (3) background thermal image adjustment processing (step S255). In this case, the reason for performing (3) background thermal image adjustment processing is that a thermal trace is temporary, and it is not appropriate to replace the background thermal image with a thermal image that includes such a temporary thermal trace, but it is appropriate to replace the background thermal image J determined in step S256 with the thermal image J. b and Thermal Image J t2 Substantial differences between the background thermal image and the actual thermal image may include calibration, temperature changes in the field of view, and other factors such as temperature rises or falls. b This is because there is a possibility that the difference in the thermal radiation of the background of the photographed area between the time when the image was taken and the target time t2 may be included.

[0051] <<(1) Background thermal image replacement process>> The (1) background thermal image replacement process performed by the background update unit 25 is a process of obtaining the thermal image at the target time t2 as a new background thermal image (step S258). In other words, the (1) background thermal image replacement process performed by the background update unit 25 is a process of replacing the background thermal image with the thermal image at the target time t2. In the example of FIG. 4, the (1) background thermal image replacement process is performed in step S258 after the judgments of steps S251, S252, S256, and S257. However, the judgments in the example of FIG. 4 are merely an example, and the background update unit 25 may perform the (1) background thermal image replacement process at a time that satisfies the following predetermined conditions:

[0052] An example of the predetermined condition is the first condition described below.

[0053] The first condition is that the difference between the thermal image at the target time and the background thermal image is greater than or equal to a first predetermined amount, and the difference between the actual object image at the target time and the background actual object image is less than or equal to a second predetermined amount.

[0054] The part of the first condition that reads "the difference between the thermal image at the target time and the background thermal image is greater than or equal to a first predetermined amount" corresponds to the "background thermal image J" in step S256. b and thermal image J at target time t2 t2 The difference d(J b ,J t2 ) is smaller than or equal to the predetermined amount ε4. The first predetermined amount corresponds to the predetermined amount ε4. In the above example, (1) as a prerequisite for the replacement process of the background thermal image, in step S256, the difference d(J b ,J t2 ) is judged to be smaller than or equal to the predetermined amount ε4, in other words, the difference d(J b ,J t2 ) is determined to be equal to or greater than a predetermined amount ε4.

[0055] The part of the first condition that says "the difference between the actual object image at the target time and the background actual object image is smaller than or equal to a second predetermined amount" corresponds to "the difference between the actual object image at the target time and the background actual object image I" in step S252. b and the object image I at target time t2t2 The difference d(I b ,I t2 The second predetermined amount corresponds to the predetermined amount ε3. In the above example, (1) as a prerequisite for the replacement process of the background thermal image, in step S252, the background actual object image I b and the object image I at target time t2 t2 The difference d(I b ,I t2 ) is determined to be less than or equal to a predetermined amount ε3.

[0056] Therefore, in the above example, (1) the first condition is met as a prerequisite for the background thermal image replacement process to be performed.

[0057] The predetermined condition may be a condition that the first condition described above and the second condition described below are satisfied.

[0058] The second condition is that the difference between the thermal image at the target time and the thermal image at a time before the target time is smaller than or equal to a third predetermined amount, and the difference between the physical object image at the target time and the physical object image at a time before the target time is smaller than or equal to a fourth predetermined amount.

[0059] This second condition is satisfied in step S251 when the actual object image I at time t1 in the past is t1 and the object image I at target time t2 t2 The difference d(I t1 ,I t2 ) is smaller than or equal to a predetermined amount ε1, and the thermal image J at the past time t1 t1 and thermal image J at target time t2 t2 The difference d(J t1 ,J t2 The third predetermined amount corresponds to the predetermined amount ε2. The fourth predetermined amount corresponds to the predetermined amount ε1. In the above example, (1) as a prerequisite for the replacement process of the background thermal image, in step S251, the actual object image I at the past time t1 is replaced. t1 and the object image I at target time t2 t2The difference d(I t1 ,I t2 ) is smaller than or equal to a predetermined amount ε1, and the thermal image J at the past time t1 t1 and thermal image J at target time t2 t2 The difference d(J t1 ,J t2 ) is determined to be less than or equal to a predetermined amount ε2.

[0060] Therefore, in the above example, (1) the second condition is met as a prerequisite for the background thermal image replacement process to be performed.

[0061] The predetermined condition may be a condition that the first condition mentioned above and a third condition described below are satisfied.

[0062] The predetermined condition may also be a condition that the first and second conditions described above, as well as a third condition described below, are satisfied.

[0063] The third condition is that the thermal image at the target time does not contain any area of ​​a thermal trace.

[0064] This third condition corresponds to the step S257, "Determining whether the thermal image at the target time contains a thermal trace area." In the above example, (1) it is determined in step S257 that the thermal image at the target time does not contain a thermal trace area, as a prerequisite for the background thermal image replacement process.

[0065] Therefore, in the above example, (1) the third condition is met as a prerequisite for the background thermal image replacement process to be performed.

[0066] <<(2) Replacing background images>> The (2) background entity image replacement process performed by the background update unit 25 is a process of obtaining the entity image at the target time as a new background entity image (step S254). In other words, the (2) background entity image replacement process performed by the background update unit 25 is a process of obtaining the entity image at the target time t2 as the background entity image I bIn the example of Fig. 4, the (2) background entity image replacement process is performed in step S254 after the determinations in steps S251, S252, and S253, but the determinations in the example of Fig. 4 are merely an example, and the background update unit 25 may perform the (2) background entity image replacement process at a time when a predetermined condition, exemplified below, is satisfied.

[0067] An example of the predetermined condition is the first condition described below.

[0068] The first condition is that the difference between the thermal image at the target time and the background thermal image is smaller than or equal to a first predetermined amount, and the difference between the actual object image at the target time and the background actual object image is larger than or equal to a second predetermined amount.

[0069] The part of the first condition that reads "the difference between the actual object image at the target time and the background actual object image is greater than or equal to a second predetermined amount" corresponds to the "background actual object image I" in step S252. b and the object image I at target time t2 t2 The difference d(I b ,I t2 The second predetermined amount corresponds to the predetermined amount ε3. In the above example, (2) as a prerequisite for the replacement process of the background substantial object image, in step S252, the background substantial object image I b and the object image I at target time t2 t2 The difference d(I b ,I t2 ) is judged to be smaller than or equal to the predetermined amount ε3, in other words, the difference d(I b ,I t2 ) is determined to be equal to or greater than a predetermined amount ε3.

[0070] The part of the first condition that says "the difference between the thermal image at the target time and the background thermal image is less than or equal to a first predetermined amount" corresponds to the "background thermal image J" in step S253. b and thermal image J at target time t2 t2 The difference d(J b ,J t2) is smaller than or equal to the predetermined amount ε4. The first predetermined amount corresponds to the predetermined amount ε4. In the above example, (2) as a prerequisite for the replacement process of the background substantial object image, in step S253, the difference d(J b ,J t2 ) is determined to be less than or equal to a predetermined amount ε4.

[0071] Therefore, in the above example, (2) the first condition is satisfied as a prerequisite for the replacement process of the background entity image.

[0072] The predetermined condition may be a condition that the first condition described above and the second condition described below are satisfied.

[0073] The second condition is that the difference between the thermal image at the target time and the thermal image at a time before the target time is smaller than or equal to a third predetermined amount, and the difference between the physical object image at the target time and the physical object image at a time before the target time is smaller than or equal to a fourth predetermined amount.

[0074] This second condition is satisfied in step S251 when the actual object image I at time t1 in the past is t1 and the object image I at target time t2 t2 The difference d(I t1 ,I t2 ) is smaller than or equal to a predetermined amount ε1, and the thermal image J at the past time t1 t1 and thermal image J at target time t2 t2 The difference d(J t1 ,J t2 The third predetermined amount corresponds to the predetermined amount ε2. The fourth predetermined amount corresponds to the predetermined amount ε1. In the above example, (2) as a prerequisite for the replacement process of the background actual object image, in step S251, the actual object image I at the past time t1 is replaced. t1 and the object image I at target time t2 t2 The difference d(I t1 ,I t2 ) is smaller than or equal to a predetermined amount ε1, and the thermal image J at the past time t1 t1and thermal image J at target time t2 t2 The difference d(J t1 ,J t2 ) is determined to be less than or equal to a predetermined amount ε2.

[0075] Therefore, in the above example, (2) the second condition is satisfied as a prerequisite for the replacement process of the background entity image to be performed.

[0076] <<(3) Adjustment of background thermal image>> The background update unit 25 performs the (3) background thermal image adjustment process on the thermal image J at the target time t2. t2 When the predetermined adjustment requirements are met, the background area in the thermal image J at the target time t2 is defined as the "background area." t2 Typical temperature T' of the background region t2 Background thermal image from J b Typical temperature T' of the background region b The value obtained by subtracting b If the predetermined adjustment requirement is not satisfied, a new background thermal image is not obtained (step S255). When a new background thermal image is obtained in step S255, the background update unit 25 may use the newly obtained background thermal image as the background thermal image to be used in the background update process of the background update unit 25 at the next time, or may use the current background thermal image J instead of the newly obtained background thermal image. b The background thermal image itself may be used as the background thermal image for the background update process of the background update unit 25 at the next time. If no new background thermal image is obtained in step S255, the background update unit 25 uses the current background thermal image J. b This image is used as the background thermal image for the next background update process by the background update unit 25. The predetermined adjustment requirements will be described later.

[0077] Thermal Image J t2 Typical temperature T' of the background region t2 is a thermal image t2 This is a value that represents the temperature of the background area of ​​the thermal image. t2 Typical temperature T' of the background region t2 For example, thermal image Jt2 The average temperature of the pixels in the background area of ​​the thermal image J t2 The median temperature of the pixels in the background area of ​​the thermal image J t2 is the most frequent value of the temperature of the pixels included in the background region.

[0078] Background Thermal Image J b Typical temperature T' of the background region b is the background thermal image J b This is a value that represents the temperature of the background area of ​​the background thermal image J. b Typical temperature T' of the background region b For example, the background thermal image J b The average temperature of the pixels included in the background region of the background thermal image J b The median temperature of the pixels included in the background region of the background thermal image J b is the most frequent value of the temperature of the pixels included in the background region.

[0079] An example of the (3) background thermal image adjustment process by the background update unit 25 will be described below with reference to FIG.

[0080] The background update unit 25 first updates the thermal image J at the target time t2. t2 Typical temperature T t2 and background thermal image J b Typical temperature T b Difference from |T t2 -T b It is determined whether | is greater than or equal to a predetermined first threshold δ1 (step S2551).

[0081] Thermal Image J t2 Typical temperature T t2 is a thermal image t2 This is a value that represents the temperature of the t2 Typical temperature T t2 For example, thermal image J t2 The average temperature of the pixels in the thermal image J t2 The median temperature of the pixels contained in the thermal image J t2 is the most frequent temperature value of the pixels included in

[0082] Background Thermal Image Jb Typical temperature T b is the background thermal image J b The representative temperature value of the background thermal image is T b is, for example, the average temperature of the pixels included in the background thermal image, the median temperature of the pixels included in the background thermal image, or the mode temperature of the pixels included in the background thermal image.

[0083] difference|T t2 -T b If it is determined that | is not greater than or equal to the predetermined first threshold δ1 (i.e., the difference |T t2 -T b is equal to or smaller than the predetermined first threshold δ1), the background update unit 25 ends the background thermal image adjustment process (3) without obtaining a new background thermal image.

[0084] difference|T t2 -T b If it is determined that | is greater than or equal to the predetermined first threshold δ1, the background update unit 25 generates a mask image for extracting the background region (step S2552).

[0085] For example, the background update unit 25 updates the actual object image I at the target time t2. t2 and background object image I b An image showing the foreground region is generated by comparing the image with the image, and the generated image is used as a mask image.

[0086] The image (mask image) showing the foreground region is, for example, the actual object image I at the target time t2. t2 pixel values ​​and background object image I b The pixel value corresponding to a pixel whose difference with the corresponding pixel value in the image is greater than or equal to a predetermined value (in other words, a pixel corresponding to the foreground region) is set to v1, and the pixel value corresponding to a pixel other than the predetermined value is set to v2. For example, v1=1, v2=0.

[0087] The background update unit 25 updates the thermal image J at the target time t2. t2 , background thermal image J b, and the thermal image J at target time t2 is obtained using the mask image. t2 Typical temperature T' of the background region t2 Background thermal image from J b Typical temperature T' of the background region b The value obtained by subtracting dT=T' t2 -T' b is calculated (step S2553).

[0088] For this purpose, the background update unit 25 updates, for example, the thermal image J at the target time t2 corresponding to the pixel having the pixel value v2 in the mask image. t2 The area consisting of the pixels in the figure is considered as the background area, and the representative value of the temperature in this background area is calculated. The calculated representative value is used as the thermal image J at the target time t2. t2 Typical temperature T' of the background region t2 Let's say.

[0089] In addition, the background update unit 25 updates, for example, the background thermal image J corresponding to the pixel having the pixel value v2 in the mask image. b The area consisting of the pixels in the image is taken as the background area, and the representative temperature value of this background area is calculated. b Typical temperature T' of the background region b Let's say.

[0090] Then, the background update unit 25 calculates dT=T' t2 -T' b Calculate.

[0091] The background update unit 25 determines whether the absolute value of dT is greater than or equal to a predetermined second threshold δ2 (step S2554). In other words, the background update unit 25 updates the thermal image J at the target time t2. t2 Typical temperature T' of the background region t2 and background thermal image J b Typical temperature T' of the background region b It is determined whether the difference |dT| is greater than or equal to a predetermined second threshold value δ2.

[0092] In step S2554, if it is determined that the absolute value of dT is not greater than or equal to the predetermined second threshold δ2, the background update unit 25 ends the (3) background thermal image adjustment process without obtaining a new background thermal image.

[0093] In step S2554, if it is determined that the absolute value of dT is greater than or equal to the predetermined second threshold δ2, the background update unit 25 updates the background thermal image J b The image consisting of pixels whose pixel values ​​are obtained by adding dT to each pixel value of the image is set as a new background thermal image (step S2555). In this way, the background update unit 25 performs offset adjustment on the background thermal image using dT, and sets the background thermal image after the offset adjustment process as a new background thermal image.

[0094] That is, the background update unit 25 updates the thermal image J at the target time t2. t2 Typical temperature T' of the background region t2 Background thermal image from J b Typical temperature T' of the background region b The value obtained by subtracting the above is the background thermal image J b The predetermined adjustment requirement for obtaining a new background thermal image by adding the pixel values ​​of the thermal image J at the target time t2 is t2 Typical temperature T t2 and background thermal image J b Typical temperature T b Difference from |T t2 -T b | is greater than or equal to the predetermined first threshold δ1 (YES in step S2551), and the thermal image J at the target time t2 t2 Typical temperature T' of the background region t2 Background thermal image from J b Typical temperature T' of the background region b The value obtained by subtracting dT=T' t2 -T' b is greater than or equal to a predetermined second threshold value δ2 (YES in step S2554).

[0095] The background update unit 25 updates the actual object image I at the target time t2. t2 and background object image Ib In addition to the foreground region obtained by comparing the image with the image, the mask image may be generated by taking into consideration the region of the thermal signature extracted up to time t2 where heat still remains.

[0096] In this case, in step S2552, the background update unit 25 updates the actual object image I at the target time t2. t2 pixel values ​​and background object image I b and the pixel values ​​of the pixels (in other words, pixels corresponding to the foreground region) whose difference from the corresponding pixel values ​​of the thermal image J at the target time t2 is greater than or equal to a predetermined value. t2 A mask image is generated by defining v1 as the pixel value corresponding to the pixel included in the area of ​​the thermal trace extracted up to time t2 where heat still remains, and v2 as the pixel value corresponding to the pixel not included.

[0097] By masking the areas of the thermal traces extracted up to time t2 that still have residual heat, the temperatures of the pixels in these areas can be excluded from the calculation of the representative temperature value, allowing for more appropriate adjustment of the background thermal image.

[0098] So far, the process of obtaining a background region using a mask image has been described. However, the process of obtaining a background region is not limited to the process using a mask image. Other processes may be used as the process of obtaining a background region.

[0099] The background update unit 25 performs (1) a process of replacing the background thermal image, (2) a process of replacing the background real object image, and (3) a process of adjusting the background thermal image, thereby making it possible to replace the background real object image and replace or adjust the background thermal image with less computational effort than in Non-Patent Document 1.

[0100] As described above, the background update unit 25 only needs to perform at least one of the following processes: (1) replacing the background thermal image; (2) replacing the background entity image; and (3) adjusting the background thermal image. In other words, the background update unit 25 does not need to perform all of the processes from step S251 to step S258. Below, Modifications 1 to 5 will be described as examples of such a background update unit 25.

[0101] <<Modification 1 of the background update unit 25>> The background update unit 25 may perform only the (1) background thermal image replacement process.

[0102] For example, the background update unit 25 does not need to perform the processes from step S253 to step S255 described above. That is, as shown in FIG. 6, the background update unit 25 calculates the difference d(I b ,I t2 If it is not determined that ε3 is smaller than or equal to the predetermined amount ε3, and if it is determined in step S257 that there is a thermal trace in the thermal image, the background update unit 25 may end the processing for the target time t2 without performing the background update processing.

[0103] The processes from step S251 to step S252 and step S256 to step S258 are the same as those described above, and therefore will not be described again.

[0104] <<Modification 2 of the background update unit 25>> The background update unit 25 may perform only (2) the process of replacing the background entity image.

[0105] For example, the background update unit 25 does not need to perform the processes from step S255 to step S258 described above. That is, as shown in FIG. 7, the background update unit 25 calculates the difference d(I b ,I t2 ) is smaller than or equal to the predetermined amount ε3, and in step S253, the difference d(J b ,J t2If it is not determined that the difference ε is smaller than or equal to the predetermined amount ε4, the background update unit 25 may end the processing at the target time t2 without performing the background update processing.

[0106] The processing from step S251 to step S254 is the same as the processing described above, and therefore a duplicated description will be omitted.

[0107] <<Modification 3 of the background update unit 25>> The background update unit 25 may perform only (1) the process of replacing the background thermal image and (2) the process of replacing the background actual object image.

[0108] For example, the background update unit 25 does not need to perform the process of step S255 described above. That is, as shown in FIG. 8, the background update unit 25 updates the difference d(J b ,J t2 If it is not determined that ε4 is smaller than or equal to the predetermined amount ε4, and if it is determined in step S257 that there is a thermal trace in the thermal image, the background update unit 25 may end the processing for the target time t2 without performing the background update processing.

[0109] The processes from step S251 to step S254 and step S256 to step S258 are the same as those described above, and therefore will not be described again.

[0110] <<Fourth Modification of the Background Update Unit 25>> The background update unit 25 may perform only (1) the process of replacing the background thermal image and (3) the process of adjusting the background thermal image.

[0111] For example, the background update unit 25 does not need to perform the process of step S254 described above. That is, as shown in FIG. 9, the background update unit 25 updates the difference d(J b ,J t2 If it is determined that the difference ε is smaller than or equal to the predetermined amount ε4, the background update unit 25 may end the processing at the target time t2 without performing the background update processing.

[0112] The processes from step S251 to step S253 and step S255 to step S258 are the same as those described above, and therefore will not be described again.

[0113] <<Modification 5 of the background update unit 25>> The background update unit 25 may perform only the (3) adjustment process of the background thermal image.

[0114] For example, the background update unit 25 does not need to perform the processes of steps S254 and S258. That is, as shown in FIG. 10, the background update unit 25 updates the difference d(J b ,J t2 If it is determined that ε₄ is smaller than or equal to the predetermined amount ε₄, and if it is determined in step S257 that there is no thermal trace in the thermal image, the background update unit 25 may end the processing for the target time t₂ without performing the background update processing.

[0115] The processes from step S251 to step S253 and step S255 to step S257 are the same as those described above, and therefore will not be described again.

[0116] <<Specific example of processing by the background update unit 25>> A specific example of the process of the background update unit 25 will be described below.

[0117] The thermal images and background thermal images at each time point from time 1 to time 5 are shown in FIG.

[0118] For the sake of simplicity, the following description will be given on the assumption that the determination in step S251 in FIG. 4 is YES at each of times 1 to 5.

[0119] Assume that at time 2, the overall temperature of the thermal image has risen compared to time 1 due to calibration. Also, assume that at time 2, a person who was not in the shooting range at time 1 now enters the shooting range. Since the overall temperature of the thermal image is higher than at time 1 due to calibration, it is preferable to update the background thermal image at time 2. However, since the person is in the shooting range at time 2, it is not appropriate to use the thermal image at time 2 as the background thermal image by (1) background thermal image replacement processing. Therefore, at time 2, the background update unit 25 determines NO in step S252 and NO in step S253 of FIG. 4, and (3) performs background thermal image adjustment processing (step S255).

[0120] Assume that a person leaves the imaging range between time 2 and time 3, and that there is no person in the imaging range at time 3. Assume also that at time 3, the overall temperature of the thermal image rises from time 2 due to calibration. Assume also that at time 3, there is no thermal trace remaining in the imaging range due to the person's presence. Assume also that as a result of the (3) background thermal image adjustment process at time 2, the pixel values ​​of the background thermal image at time 1 are adjusted and this is used as the new background thermal image at time 2, becoming the background thermal image at time 3. Since the overall temperature of the thermal image is higher than at time 2 due to calibration, it is preferable to update the background thermal image at time 3. Since there is no person in the imaging range at time 3 and no thermal trace remains in the imaging range, it is appropriate to use the thermal image at time 3 as the background thermal image by (1) background thermal image replacement process. Therefore, at time 3, the background update unit 25 determines YES in step S252, NO in step S256, and YES in step S257 in FIG. 4, and (1) performs the background thermal image replacement process (step S258).

[0121] At time 4, the temperature has changed from time 3 due to air conditioning, and the overall temperature of the thermal image has dropped compared to time 3. Also, at time 4, the thermal image contains a thermal trace left by a person who entered the shooting area after time 3 but left the shooting area before time 4. Furthermore, as a result of the (1) background thermal image replacement process at time 3, the thermal image at time 3 is designated as the new background thermal image and becomes the background thermal image at time 4. Since the overall temperature of the thermal image has dropped compared to time 3 due to air conditioning, it is preferable to update the background thermal image at time 4. However, since the thermal image contains the thermal trace at time 4, it is not appropriate to use the (1) background thermal image replacement process as the background thermal image. Therefore, at time 4, the background update unit 25 determines YES at step S252, NO at step S256, and NO at step S257 in FIG. 4, and then performs (3) background thermal image adjustment process (step S255).

[0122] Assume that at time 5, the thermal traces present at time 4 have disappeared from the thermal image. Furthermore, assume that due to air conditioning, the overall temperature of the thermal image at time 5 is lower than the overall temperature of the thermal image at time 4. Furthermore, assume that as a result of the (3) background thermal image adjustment process at time 4, the pixel values ​​of the background thermal image at time 3 are adjusted and used as the new background thermal image at time 4, which becomes the background thermal image at time 5. Since the overall temperature of the thermal image is lower than that at time 4 due to air conditioning, it is preferable to update the background thermal image at time 5. Since there is no person in the shooting range and no thermal traces remain in the shooting range at time 5, it is appropriate to use the thermal image at time 5 as the background thermal image by (1) background thermal image replacement process. Therefore, at time 5, the background update unit 25 determines YES in step S252, NO in step S256, and YES in step S257 of FIG. 4, and performs (1) background thermal image replacement process (step S258).

[0123] As in this example, the above process allows the background thermal image to be updated appropriately even if there is a temperature change.

[0124] [Image processing device and method] 12, the image processing device 3 includes, for example, a background updating device 2 and an image processing unit 31. The image processing device 3 may further include a stereo camera 11 and a thermal camera 13.

[0125] The image processing method is realized, for example, by each component of the image processing device 3 performing the processes of steps S11, S21, S13, S23, S25, and S31 shown in Fig. 2 at each target time for extracting a heat trace (hereinafter referred to as "target time for heat trace extraction") after a predetermined time T1 has elapsed since the image processing device 3 started operating. Note that the image processing device 3 may perform the processes of steps S11, S21, S13, S23, S22-1, S22-2, S24-1, and S24-2 shown in Fig. 3 by each component, for example, to obtain an initial image of the background entity image and an initial image of the background thermal image used in the processes of steps S25 and S31.

[0126] The image processing unit 31 performs predetermined image processing using the most recent background entity image and the most recent background entity image at each heat trace extraction target time. That is, if the background update unit 25 has not yet obtained any new background entity image, the image processing unit 31 performs predetermined image processing using the background entity image acquired by the initial background entity image acquisition unit 22. If the background update unit 25 has already obtained one or more new background entity images, the image processing unit 31 performs predetermined image processing using the most recent background entity image among the new background entity images acquired by the background update unit 25. Similarly, if the background update unit 25 has not yet obtained any new background thermal image, the image processing unit 31 performs predetermined image processing using the background thermal image acquired by the initial background thermal image acquisition unit 24. If the background update unit 25 has already obtained one or more new background thermal images, the image processing unit 31 performs predetermined image processing using the most recent background thermal image among the new background thermal images acquired by the background update unit 25. That is, the image processing unit 31 performs predetermined image processing using at least the background actual object image and the background thermal image obtained in step S25.

[0127] The following description will be given taking as an example a case where the image processing unit 31 performs a process of extracting a heat trace. Of course, the image processing unit 31 may perform a process other than the extraction of a heat trace.

[0128] As shown in FIG. 13, the image processing unit 31 includes a differential actual object image generating unit 311, a differential thermal image generating unit 312, and a thermal trace extracting unit 313, for example.

[0129] <Differential entity image generation unit 311> The differential entity image generating unit 311 receives the entity image at the target time for heat trace extraction acquired by the entity image acquiring unit 21. If the background updating unit 25 has not yet acquired a new background entity image, the background entity image acquired by the initial background entity image acquiring unit 22 is input to the differential entity image generating unit 311. If the background updating unit 25 has already acquired one or more new background entity images, the most recent background entity image among the new background entity images acquired by the background updating unit 25 is input to the differential entity image generating unit 311. In other words, the most recent background entity image at the target time for heat trace extraction is input to the differential entity image generating unit 311.

[0130] The differential entity image generating unit 311 generates a differential entity image, which is a difference image between the entity image at the heat trace extraction target time and the most recent background entity image at the heat trace extraction target time (step S311), and outputs the generated differential entity image at the heat trace extraction target time. In short, the differential entity image generating unit 311 generates a differential entity image, which is a difference image between the entity image at the heat trace extraction target time, which is an image of an entity obtained by photographing a certain photographing range with the entity camera 11 for photographing an entity, and the background entity image, which is the most recent background entity image of the certain photographing range at the heat trace extraction target time. The generated differential entity image is input to the heat trace extraction unit 313.

[0131] <Differential Thermal Image Generator 312> The differential thermal image generating unit 312 receives the thermal image at the target time of thermal trace extraction acquired by the thermal image acquiring unit 23. If the background updating unit 25 has not yet acquired any new background thermal image, the background thermal image acquired by the initial background thermal image acquiring unit 24 is input to the differential thermal image generating unit 312. If the background updating unit 25 has already acquired one or more new background thermal images, the most recent background thermal image among the new background thermal images acquired by the background updating unit 25 is input to the differential thermal image generating unit 312. That is, the most recent background thermal image at the target time of thermal trace extraction is input to the differential thermal image generating unit 312.

[0132] The differential thermal image generating unit 312 generates a differential thermal image, which is an image of the difference between the thermal image at the target time for heat trace extraction and the most recent background thermal image at the target time for heat trace extraction (step S312), and outputs the generated differential thermal image at the target time for heat trace extraction. In short, the differential thermal image generating unit 312 generates a differential thermal image, which is an image of the difference between the thermal image at the target time for heat trace extraction, which is an image of heat emitted by an actual object obtained by photographing the same photographing range as the actual object camera 11 with the thermal camera 13 for photographing heat emitted by the actual object, and the background thermal image, which is the most recent background thermal image of the photographing range at the target time for heat trace extraction. The generated differential thermal image is input to the heat trace extracting unit 313.

[0133] <Heat trace extraction part 313> The heat trace extraction unit 313 extracts the area of ​​the heat trace by removing the area of ​​the actual object from the thermal image based on the differential actual object image at the target time for heat trace extraction generated by the differential actual object image generation unit 311 and the differential thermal image at the target time for heat trace extraction generated by the differential thermal image generation unit 312 (step S313), and outputs information indicating the extracted area of ​​the heat trace at the target time for heat trace extraction.

[0134] Specifically, the heat trace extraction unit 313 defines each region in the differential object image that differs from the background object image as a differential object region, defines each region in the differential thermal image that differs from the background thermal image as a differential thermal region, and extracts, from the one or more differential thermal regions, a region dissimilar to any of the one or more differential object regions as a heat trace region. The heat trace extraction unit 17 may generate a binary image in which the heat trace region is colored white and the rest is colored black, and output the generated binary image. The target time for heat trace extraction may be each time at a predetermined interval, or may be a time specified by the operator of the image processing unit 31, etc.

[0135] [Programs, recording media, etc.] The processing of each unit of each of the above-mentioned devices may be realized by a computer, in which case the processing content of the functions that each device should have is described by a program. Then, by loading this program into storage unit 1020 of computer 1000 shown in Fig. 14 and operating arithmetic processing unit 1010, input unit 1030, output unit 1040, display unit 1060, etc., various processing functions of each of the above-mentioned devices are realized on the computer.

[0136] Each of the above-mentioned devices, as a single hardware entity, has, for example, an input unit capable of inputting signals from outside the hardware entity, an output unit capable of outputting signals to outside the hardware entity, a communication unit to which a communication device (e.g., a communication cable) can be connected for communication with outside the hardware entity, a CPU (which may also include a central processing unit, cache memory, registers, etc.) as an arithmetic processing unit, RAM and ROM as memories, an external storage device such as a hard disk, and buses connecting these input unit, output unit, communication unit, CPU, RAM, ROM, and external storage device so that data can be exchanged between them. If necessary, the hardware entity may also be provided with a device (drive) capable of reading and writing to a recording medium such as a CD-ROM. An example of a physical entity equipped with such hardware resources is a general-purpose computer.

[0137] The external storage device of the hardware entity stores the programs required to realize the above-mentioned functions and the data required for processing these programs (the programs may be stored in a ROM, which is a read-only storage device, for example, instead of an external storage device). Data obtained by processing these programs is stored in RAM, the external storage device, etc. as appropriate.

[0138] In the hardware entity, each program stored in an external storage device (or ROM, etc.) and data required for processing each program are loaded into memory as needed, and interpreted, executed, and processed by the CPU as appropriate. As a result, the CPU realizes predetermined functions (each component represented as a "... unit" above). In other words, each component in the embodiments of the present invention may be configured by a processing circuitry.

[0139] As mentioned above, when the processing functions of the hardware entities (the above-mentioned devices) described in the above embodiments are realized by a computer, the processing contents of the functions that the hardware entities should have are described by a program. Then, by executing this program on a computer, the processing functions of the hardware entities are realized on the computer.

[0140] The program describing the processing contents can be recorded on a computer-readable recording medium, such as a non-transitory recording medium, specifically a magnetic recording device, an optical disk, or the like.

[0141] The program may be distributed, for example, by selling, transferring, lending, etc. a portable recording medium such as a DVD or CD-ROM on which the program is recorded. Furthermore, the program may be stored in a storage device of a server computer, and then transferred from the server computer to another computer via a network, thereby distributing the program.

[0142] A computer that executes such a program, for example, first stores the program recorded on a portable recording medium or transferred from a server computer in its own non-transitory storage device, auxiliary storage unit 1050. Then, when executing a process, the computer loads the program stored in auxiliary storage unit 1050, its own non-transitory storage device, into storage unit 1020 and executes processing in accordance with the loaded program. Alternatively, as another form of execution of this program, the computer may load the program directly from a portable recording medium into storage unit 1020 and execute processing in accordance with the program. Furthermore, each time a program is transferred from a server computer to this computer, the computer may execute processing in accordance with the received program. Alternatively, the server computer may not transfer the program to this computer, but may instead execute the processing function by issuing an execution instruction and obtaining the results, thereby executing the above-described processing through a so-called ASP (Application Service Provider) type service. Note that the program in this embodiment includes information used for processing by a computer that is equivalent to a program (such as data that is not a direct instruction to a computer but has properties that define computer processing).

[0143] Furthermore, in this embodiment, the device is configured by executing a predetermined program on a computer, but at least a part of the processing contents may be realized by hardware.

[0144] It goes without saying that other modifications are possible without departing from the spirit of the present invention.

[0145] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

Claims

1. a background updating unit that uses (A) a thermal image, which is an image of heat emitted by an actual object, obtained by photographing a certain photographing range at a target time with a thermal camera for photographing heat emitted by the actual object, (B) an actual object image, which is an image of the actual object, obtained by photographing the certain photographing range at the target time with a real object camera for photographing the actual object, (C) a background thermal image, which is a thermal image of the background of the certain photographing range, and (D) a background actual object image, which is an image of the actual object in the background of the certain photographing range, to obtain the thermal image at the target time as a new background thermal image when a predetermined condition is satisfied; The predetermined condition is a first condition that a difference between the thermal image and the background thermal image at the target time is greater than or equal to a first predetermined amount, and a difference between the object image and the background object image at the target time is less than or equal to a second predetermined amount. Background update device.

2. a background updating unit that uses (A) a thermal image that is an image of heat emitted by an entity obtained by photographing a certain photographing range at a target time with a thermal camera for photographing heat emitted by the entity, (B) an entity image that is an image of the entity obtained by photographing the certain photographing range at the target time with a entity camera for photographing entity, (C) a background thermal image that is a thermal image of the background of the certain photographing range, and (D) a background entity image that is an entity image of the background of the certain photographing range to obtain the entity image at the target time as a new background entity image when a predetermined condition is satisfied; The predetermined condition is a first condition that a difference between the thermal image and the background thermal image at the target time is smaller than or equal to a first predetermined amount, and a difference between the object image and the background object image at the target time is larger than or equal to a second predetermined amount. Background update device.

3. 2. The background updating device of claim 1, a second condition being that a difference between the thermal image at the target time and a thermal image at a time before the target time is smaller than or equal to a third predetermined amount, and a difference between the actual object image at the target time and a actual object image at a time before the target time is smaller than or equal to a fourth predetermined amount; the predetermined condition is a condition that the first condition and the second condition are satisfied; Background update device.

4. 3. The background updating device of claim 2, a second condition being that a difference between the thermal image at the target time and a thermal image at a time before the target time is smaller than or equal to a third predetermined amount, and a difference between the actual object image at the target time and a actual object image at a time before the target time is smaller than or equal to a fourth predetermined amount; the predetermined condition is a condition that the first condition and the second condition are satisfied; Background update device.

5. 4. A background updating device according to claim 1 or 3, A third condition is that there is no thermal trace in the thermal image at the target time. the predetermined condition is a condition that the first condition and the third condition are satisfied, or a condition that the first condition, the second condition, and the third condition are satisfied; Background update device.

6. a background updating step of obtaining a new background thermal image of the thermal image at the target time when a predetermined condition is satisfied, using (A) a thermal image, which is an image of heat emitted by a real object, obtained by photographing a certain photographing range at a target time with a thermal camera for photographing heat emitted by the real object; (B) a real object image, which is an image of the real object, obtained by photographing the certain photographing range at the target time with a real object camera for photographing real objects; (C) a background thermal image, which is a thermal image of the background of the certain photographing range; and (D) a background real object image, which is an image of the real object in the background of the certain photographing range; The predetermined condition is a first condition that a difference between the thermal image and the background thermal image at the target time is greater than or equal to a first predetermined amount, and a difference between the object image and the background object image at the target time is less than or equal to a second predetermined amount. How to update the background.

7. a background updating step of obtaining the entity image at the target time as a new background entity image when a predetermined condition is satisfied using (A) a thermal image, which is an image of heat emitted by an entity, obtained by photographing a certain photographing range at a target time with a thermal camera for photographing heat emitted by the entity, (B) an entity image, which is an image of the entity, obtained by photographing the certain photographing range at the target time with a entity camera for photographing entity, (C) a background thermal image, which is a thermal image of the background of the certain photographing range, and (D) a background entity image, which is an entity image of the background of the certain photographing range; The predetermined condition is a first condition that a difference between the thermal image and the background thermal image at the target time is smaller than or equal to a first predetermined amount, and a difference between the object image and the background object image at the target time is larger than or equal to a second predetermined amount. How to update the background.

8. A program for causing a computer to function as the background updating unit of the background updating device of claim 1 or 2.

Images