Method for extracting areas of change over time using background difference analysis.

Abstract

This invention provides a new background subtraction method that can reliably and quickly detect changes in the state of a region or location over time using images acquired from imaging equipment mounted on a mobile device, and a method for extracting locations that have changed over time using a background subtraction method that can quickly and reliably extract locations that have changed over time using images acquired from imaging equipment mounted on a mobile device that are expected to show changes over time. [Solution] This method extracts areas of change over time using background subtraction with images acquired by a moving object. When acquiring multiple images at different acquisition times, it detects matching feature points in the different acquired images, matches the matching feature points, deforms one of the images so that the composition of one image and the composition of the other image are substantially the same, calculates the difference in image brightness between the deformed image and the other image, and extracts areas of change over time based on the calculated image brightness difference.

Description

Technical Field

[0001] The present invention relates to an image processing method using a background subtraction method for an object or object area assumed to change over time, and particularly to a method for extracting a location that changes over time by a background subtraction method using an image assumed to change over time acquired by a photographing device mounted on a moving body.

Background Art

[0002] Conventionally, as a method for detecting a location that changes over time based on an image, a background subtraction method is generally known, in which a predetermined area of a certain region or location is photographed at different times, and the image is used.

[0003] Here, the background subtraction method refers to a method for detecting changes over time in a region or location by photographing the background at a previous time and the background in a state where something new has entered, something has disappeared, or the state has changed at a later time, and calculating the luminance difference between the background images at the photographed times before and after. As an application example of the background subtraction method, it is possible to process images photographed before and after a change to count, for example, the number of customers entering a store over a long period of time, or to count the traffic flow of automobiles over a long period of time.

[0004] However, in the application of the background subtraction method, it is assumed that the composition and angle of view of the images match during photographing when acquiring images before and after a change over time. Here, the composition refers to the arrangement of objects or people to be photographed, and the angle of view refers to the range to be photographed.

[0005] In the application of the background subtraction method using a fixed camera, it is relatively easy to take pictures with the same composition or the same composition and angle of view when taking pictures at different times. However, it is extremely difficult to take pictures of images before and after a change over time with the same composition or the same composition and angle of view using a photographing device mounted on a moving body and acquire them.

[0006] In recent years, methods have been devised to apply background subtraction to images acquired by cameras mounted on mobile devices such as drones. However, when applying background subtraction to images acquired by cameras mounted on mobile devices such as drones, as mentioned above, it is necessary to take or process images to match the composition and field of view of two images acquired before and after the change over time.

[0007] However, conventionally, matching the composition and field of view of two images acquired before and after a change using a shooting device mounted on a mobile device has been difficult due to the complex and cumbersome shooting or image processing procedures, as well as the extremely high cost.

[0008] When performing high-precision positioning using a mobile device, there are device limitations, such as the requirement that the device must support RTK-GNSS positioning. Furthermore, high-precision positioning using a mobile device cannot be applied in environments where GNSS satellite signals are blocked (in the shade of trees or mountains, indoors, or underground). When generating orthomosaic images based on aerial images, complex tasks such as preliminary work (installation and surveying of ground control markers) and analysis are required, resulting in extra effort, cost, and time. In addition, generating orthomosaic images is difficult with image acquisition devices that have a viewpoint close to the ground (such as handheld smartphone cameras or cameras mounted on quadruped robots).

[0009] Therefore, there was a need for a method to extract locations where the background subtraction method can be applied to images acquired by imaging equipment mounted on a mobile body, which is unaffected by the image acquisition device or image acquisition environment, and which can reliably and quickly detect changes in the state of a region or location over time. [Prior art documents] [Patent Documents]

[0010] [Patent Document 1] Japanese Patent Publication No. 2023-142894 [Overview of the project] [Problems that the invention aims to solve]

[0011] This invention was devised to meet the aforementioned demands and solve the problems of the past, and aims to provide a new background subtraction method that can reliably and quickly detect changes in the state of a region or location over time using images acquired from an imaging device mounted on a mobile body, and a method for extracting locations that have changed over time using a background subtraction method that can quickly and reliably extract locations that have changed over time using images acquired by an imaging device mounted on a mobile body in which changes over time are expected. [Means for solving the problem]

[0012] This invention is a method for extracting areas of change over time using background subtraction with moving images. When acquiring multiple images at different acquisition times, the system detects matching feature points in the different acquired images, matches the matching feature points, deforms one image so that its composition matches that of the other image, calculates the difference in image brightness between the deformed image and the other image, performs image processing to binarize each image based on the calculated image brightness difference, and extracts areas that change over time from the binarized images. It is characterized by the following: or The deformation operation of one of the images can be performed while checking the amount of deformation of the deformed image. It is characterized by the following: or The feature point matching process involves generating multiple sets of different images, selecting an image set with a high feature point matching rate from the multiple image sets, and performing feature point matching on the images in the selected image set. It is characterized by the following: or After selecting the image set with a high feature point matching rate, if the selected feature point matching rate is below a predetermined threshold, the camera used to capture the comparison image performs an operation to improve the feature point matching rate, and the comparison image is captured by the camera that has performed the operation. It is characterized by the following: or The aforementioned binarization image processing is an image processing that generates a mask image by binarizing the image and hiding a portion of it. It is characterized by the following: or This is a method for extracting areas that change over time using background subtraction with moving images. When acquiring multiple images at different acquisition times, the system detects matching feature points in the different acquired images, matches the matching feature points to deform one of the images so that its composition matches that of the other image, calculates the difference in image brightness between the deformed image and the other image, generates a mask image by binarizing each image based on the calculated image brightness difference and hiding a portion of the image, performs image processing by overlaying the generated mask image with the deformed image and / or the other image, and extracts areas that have changed over time from the processed image. It is characterized by the following: [Effects of the Invention]

[0013] According to the present invention, a novel background subtraction method is available that can reliably and quickly detect changes in the state of a region or location over time using images acquired from an imaging device mounted on a mobile body. In other words, the present invention provides a method for extracting locations that have changed over time using a background subtraction method that allows for the rapid and reliable extraction of locations that have changed over time using images acquired by an imaging device mounted on a mobile body that are expected to show changes over time.

[0014] This makes it possible to improve the freedom of the image acquisition device. That is, it is also applicable to image data acquired by a device without a high-precision positioning function. And, it is possible to extract portions that change over time by applying the background difference method, using combinations such as drones, quadruped walking robots, smartphones, wearable cameras, or combinations of different devices.

[0015] Also, it is possible to improve the freedom of the image acquisition environment. That is, in the present invention, it is possible to extract portions that change over time by applying the background difference method even in environments such as under the shade of trees or mountains where GNSS satellite radio waves are blocked, indoors, underground spaces, city streets, and outer space.

[0016] Furthermore, it is possible to reduce the computational cost of image processing. That is, high-cost calculations such as SfM (Structure from Motion) may not be performed, and real-time on-site processing and evaluation are possible.

[0017] Also, in the present invention, there is an advantage that the image data remains as it is, and only necessary portions are displayed, for example, only in black and white, and a mask image with unnecessary portions hidden is superimposed on the original image, thereby making the extraction of portions that change over time clearer and presenting it to the user. This can also be expected to improve the efficiency of operations such as appearance inspection of structures, abnormal monitoring of facilities, detection of suspicious objects, and progress management of construction sites. [[ID=X]] [[ID=Y]]

Brief Description of the Drawings

[0018] [[ID=Y]] [[ID=X]] [Figure 1] [[ID=Y]] It is an explanatory diagram of images with different acquisition times. [[ID=X]] [Figure 2] [[ID=Y]] It is an explanatory diagram of an example of performing feature point matching on an image. [[ID=X]] [Figure 3] [[ID=Y]] It is an explanatory diagram for explaining the configuration of a deformed reference image. [[ID=X]] [Figure 4] [[ID=Y]] It is an explanatory diagram (1) for calculating the luminance difference between a deformed reference image and a comparison image to generate a mask image. [[ID=X]] [Figure 5] [[ID=Y]] This is an explanatory diagram (2) showing how a mask image is generated by calculating the brightness difference between a deformed reference image and a comparison image. [Figure 6] This is an explanatory diagram illustrating the process of generating a final image by superimposing a mask image onto a comparison image. [Figure 7] This is an explanatory diagram of a reference image of an indoor bookshelf taken at a previous time. [Figure 8] This is an explanatory diagram of comparative images of an indoor bookshelf taken at different times. [Figure 9] This is an explanatory diagram illustrating an example of feature point matching for two images of an indoor bookshelf. [Figure 10] This is an explanatory diagram illustrating the configuration of a deformation reference image in an indoor bookshelf. [Figure 11] This is an explanatory diagram (1) showing how a mask image was generated by calculating the brightness difference between a deformation reference image and a comparison image on an indoor bookshelf. [Figure 12] This is an explanatory diagram (2) showing how a mask image was generated by calculating the brightness difference between a deformation reference image and a comparison image on an indoor bookshelf. [Figure 13] This is an explanatory diagram illustrating the process of generating a final image by superimposing a mask image onto a comparison image. [Figure 14] This is a schematic diagram (1) illustrating the general configuration of the present invention. [Figure 15] This is a schematic diagram (2) illustrating the general configuration of the present invention. [Figure 16] This is a schematic diagram (3) illustrating the general configuration of the present invention. [Figure 17] This is a schematic diagram (4) illustrating the general configuration of the present invention. [Modes for carrying out the invention]

[0019] The outline of the present invention is a method for extracting areas of change over time using background subtraction, which utilizes images acquired by a mobile device, that is, images acquired by a camera mounted on a mobile device 1. When acquiring multiple images at different acquisition times, the system detects matching feature points in the different acquired images, matches the matching feature points to deform one image so that its composition matches that of the other image, calculates the difference in image brightness between the deformed image and the other image, performs image processing to binarize each image based on the calculated image brightness difference, and extracts areas that change over time from the binarized images. The configuration allows for the deformation operation of one of the images while checking the amount of deformation of the deformed image. The feature point matching described above involves generating multiple sets of different images, selecting an image set with a high feature point matching rate from the multiple image sets, and performing feature point matching using the images in the selected image set. After selecting an image set with a high feature point matching rate, if the selected feature point matching rate is below a predetermined threshold, the camera that captures the comparison image performs an operation to improve the feature point matching rate, and the comparison image is captured by the camera that has performed the operation. The aforementioned binarization image processing employs an image processing method that generates a mask image by binarizing the image and hiding a portion of it. This is a method for extracting areas of change over time using background subtraction, which utilizes images acquired by imaging equipment mounted on mobile device 1. When acquiring multiple images at different acquisition times, the system detects matching feature points in the different acquired images, matches the matching feature points to deform one of the images so that its composition matches that of the other image, calculates the difference in image brightness between the deformed image and the other image, generates a mask image by binarizing each image based on the calculated image brightness difference and hiding a portion of the image, performs image processing by overlaying the generated mask image 5 with the deformed image and / or the other image, and extracts areas that have changed over time from the processed image.

[0020] Specific embodiments of the present invention will be described below with reference to the figures.

[0021] First, a mobile device 1, such as a drone, is used to photograph the target object / region at a later time, i.e., after the change has occurred, in order to extract areas that change over time using the background subtraction method (see Figure 1(b)). The image taken at the later time is called comparison image 2. The camera mounted on mobile device 1 may be a camera that takes still images or a video camera that takes videos. Furthermore, mobile device 1 is not limited to cameras mounted on mobile devices such as drones or vehicles, but also includes cameras carried by people. In other words, images acquired by a mobile device may be images taken and acquired by a person carrying a camera.

[0022] However, when taking comparison image 2 at a later time, it is necessary to take the image so that its composition or composition / angle of view is roughly the same as the reference image 3 (Figure 1(a)) taken at the previous time.

[0023] However, when capturing comparison image 2 using a camera mounted on mobile body 1, the camera is constantly moving, making it extremely difficult to capture the image in a way that roughly matches the composition or composition / angle of view of the reference image 3.

[0024] The present invention is characterized by an operation to capture the comparison image 2 using a photographic device mounted on a mobile body 1, such that the comparison image 2 is captured in a manner that is roughly the same as the reference image 3 in terms of composition or composition and field of view.

[0025] When taking comparison image 2, the composition or composition / angle of view will be roughly the same as reference image 3. Therefore, comparison image 2 will be taken to include as many of the features found in reference image 3 as possible, such as houses or transmission towers. Then, the image sets for reference image 3 and comparison image 2 will be determined.

[0026] Here, the aforementioned image set refers to, for example, a set formed by selecting two images from a group of reference images 3 and three images from a group of comparison images 2. In this case, there are six image sets.

[0027] The selection and configuration of such image sets can be done manually by selecting from multiple images, or it can be done automatically from multiple image data based on, for example, the degree of agreement between multiple feature points in the two aforementioned images. Alternatively, images can be extracted and selected from a series of frames, such as in a video.

[0028] In the image set configured as described above, feature points are detected for each image, such as the corners of the roof of the house or the boundary lines of the roof tiles.

[0029] Furthermore, in the feature point detection described above, well-known algorithms such as edge detection, HoG (Histogram of Oriented Gradients), SIFT (Scale Invariant Feature Transform), and AKAZE are used, and the detection is performed by a controller such as a CP.

[0030] Next, feature point matching is performed using the algorithm described above (see Figure 2). That is, the coordinate points of each feature point detected in reference image 3 are matched with those of each feature point detected in comparison image 2. Therefore, if there are many feature points, many feature point matches can be performed, thereby obtaining a comparison image 2 with a high probability of matching the composition or composition / angle of view of reference image 3. Note that the feature point matching operation can also be performed by a controller such as a CP.

[0031] Next, a deformation of the reference image 3 (deformed reference image 4) is formed based on the feature point matching results (see Figure 3). Here, for the deformation method of reference image 3, a linear deformation method that deforms the four corners of reference image 3, such as projection transformation (homography), or geometric correction such as a nonlinear method can be applied. Alternatively, reference image 3 can be meshed and corrections can be applied to each mesh. The deformation process of reference image 3 is also performed by a controller such as a CP. Note that the deformation is not limited to the reference image, but the comparison image may also be deformed.

[0032] The deformation of the reference image 3 shown in Figure 3 is an example of a deformed reference image 4, which is deformed using a projective transformation, a linear deformation method that deforms the four corners of the reference image 3. The square black frame shown around the outer edge of the deformed reference image 4 in Figure 3 represents the reference image 3 before deformation. In the deformed reference image 4, the white area formed in a roughly concave shape on the lower outer edge of the reference image 3 is recognized as a padding region 8, and it can be seen that the reference image has been deformed by compressing the image inward, resulting in a deformed reference image 4 that has been deformed into a so-called inverted trapezoid shape.

[0033] Here, the less the degree of deformation, that is, the smaller the padding area 8 of the white area, the more likely it is that the comparison image 2 was taken at a location where the composition or composition / angle of view matches that of the reference image 3.

[0034] However, in the case of the deformed reference image 4 in Figure 3, the padding area 8 in the white area is relatively small, and it can be recognized that the comparison image 2 was taken at a position where the composition or composition / angle of view is roughly the same as when the reference image 3 was taken.

[0035] Furthermore, during the deformation operation of the aforementioned reference image, the amount of deformation can be checked on a display such as a PC, thereby allowing for confirmation of the amount of deformation and calculation of the image matching rate.

[0036] Next, the brightness difference between the deformed reference image 4 and the comparison image 2 is calculated. This brightness difference calculation can also be performed by a controller such as a CP.

[0037] The calculation may be performed by first converting both images to grayscale and then calculating the difference in brightness at each pixel of the image, or by calculating the difference in brightness for each of the RGB values ​​at each pixel of both images.

[0038] The calculation results may be binarized by setting a threshold. For example, for each pixel in the image, if it is white, its brightness can be recognized as 255, and if it is black, its brightness can be recognized as 0. For a given pixel in the image, if it is gray, its brightness can be recognized as approximately 100. In this way, the brightness of all pixels in the image is determined, and the brightness of comparison image 2 is subtracted from the brightness of deformation reference image 4 to find the difference. When this value exceeds, for example, a threshold of 80, that area is recognized as white, and areas that do not exceed 80 are recognized as black. The image generated by this image processing is called mask image 5 (see Figure 5). Alternatively, mask image 5 can be generated by subtracting the brightness of deformation reference image 4 from the brightness of comparison image 2 and finding the difference (see Figure 4).

[0039] What can be determined from these generated mask images 5 is that when the brightness difference between the two images is calculated, if that brightness difference exceeds, for example, a threshold value of 80, that area is recognized as white, and the state of that area is recognized as having changed significantly.

[0040] The present invention allows for the extraction and confirmation of areas that change over time by directly using the image of the luminance difference obtained from the calculation results of the luminance difference, i.e., the mask image 5. Alternatively, as shown in Figure 6, by performing image processing to superimpose both mask images 5 onto the comparison image 2 and generating a superimposed output image 6, areas that have changed over time can be clearly recognized as a color image. In Figure 6, the image on the right is the superimposed output image 6. In this output image 6, the areas and how they changed, based on the luminance difference between the deformed reference image 5 and the comparison image 2, are shown in dark gray. Furthermore, the areas and how they changed, based on the luminance difference between the comparison image 2 and the deformed reference image 5, are shown in light gray.

[0041] In other words, the areas shown in dark gray represent trees that grew and extended during the summer, and these trees are gone by autumn when comparison image 2 was taken.

[0042] Additionally, the areas shown in light gray indicate that fallen leaves were piled up on the ground in the autumn when comparison image 2 was taken.

[0043] Furthermore, as mentioned above, these images are originally represented as color images. Areas shown in dark gray are represented in blue, and areas shown in light gray are represented in orange.

[0044] Figures 7 to 13 show a second embodiment of the present invention, illustrating an example of implementing the invention indoors. As can be seen from Figures 7 and 8, there are three points of change 7 in the books placed on a bookshelf indoors. However, these are extremely difficult to confirm with the naked eye. Therefore, by extracting the points of change over time using the present invention, the points of change can be clearly recognized in the resulting image 6 shown in Figure 13. This resulting image 6 is also originally represented as a color image.

[0045] Next, the general configuration of the present invention will be described with reference to Figures 14 to 17.

[0046] As shown in Figure 14, the camera mounted on the mobile body 1 captures a reference image 3 at the previous time and a comparison image at the later time. This invention also applies when the photographer holds the camera by hand and performs mobile photography.

[0047] Then, feature points are detected in both images, and these feature points are matched. Subsequently, the reference image is deformed to generate a deformed reference image 4 in order to match the composition of the reference image and the comparison image. After that, the brightness difference between the deformed reference image 4 and the comparison image is determined, and the difference is calculated. Based on this calculation result, it becomes easy to extract the parts that have changed over time in the reference image 3 from the previous time and the comparison image 2 from the later time.

[0048] Figure 15 demonstrates that increasing the degree of agreement between the compositions of both images will make the composition of the comparison image even more identical to that of the reference image, thereby enabling the application of a more accurate background subtraction method to images with less distortion. One way to improve the degree of agreement between compositions is to detect many feature points common to both images and match these numerous feature points.

[0049] In Figure 16, multiple image sets are generated as described above. Feature points are detected in both images for each generated image set, and feature point matching is performed. The image set with the highest feature point matching rate is then used to generate a deformation reference image 4. The difference between this deformation reference image 4 and comparison image 2 is calculated to extract areas of change over time.

[0050] In Figure 17, after determining the image set with the highest feature point agreement rate, it is determined whether the agreement rate is above a predetermined threshold. If it is not above the predetermined threshold, the deviation in the shooting position and orientation of the shooting device that takes comparison image 2 is calculated, and the calculated shooting position and orientation data is fed back to the shooting device mounted on the mobile body 1 to correct the deviation in shooting position and orientation, and the image is taken again. Then, when the agreement rate is above the threshold, the deformation reference image 4 is generated. [Industrial applicability]

[0051] In this invention, the image acquisition device can be a color camera, as well as image data acquired by the following types of imaging devices.

[0052] Examples include monochrome cameras, infrared cameras, spectral cameras, hemispherical and 360-degree cameras, medical imaging equipment (X-ray, MRI, CT), and microscopes.

[0053] Furthermore, the present invention is also applicable to image data acquired by the following types of mobile objects 1: namely, mobile objects 1 such as drones, aircraft, balloons, satellites, vehicles, UGVs (unmanned ground vehicles), trains, quadruped robots, people (with handheld cameras or wearable cameras), ships, submarines, and lawnmowers.

[0054] While the present invention is important for acquiring images using a camera mounted on a mobile device 1, it is not limited to this and can also be applied to correcting the shaking and misalignment of a fixed camera.

[0055] Furthermore, the present invention can also be applied to comparing videos. For example, the following processing may be performed.

[0056] For example, when applying this method to two video data from the same route, for a given frame in the comparison video, frames within a 5-second range before and after the same frame in the reference video are searched, and the extracted frames are determined based on the feature point matching rate, after which this method is applied. Furthermore, this method can be combined with machine learning. Specifically, machine learning can be used for geometric correction of the reference image 3 and for extracting the padding region 8.

[0057] Furthermore, by using a control device such as a PC, it is possible to easily correct the shooting position when taking comparison images to match the composition of reference image 3, based on the proportion of padding area 8 (the white area at the periphery added due to image correction).

[0058] Thus, while it is generally difficult to estimate the position of an aerial moving object 1 such as a drone, the method of the present invention makes it possible to quantitatively evaluate the relative position of aerial photography in reverse calculation. [Explanation of symbols]

[0059] 1 Mobile Unit 2. Comparison Images 3 Reference Image 4. Transformation reference image 5 Mask image 6. Deliverable Images 7 Changes 8 Padding Areas

Claims

1. This is a method for extracting areas of change over time using background subtraction with images acquired by a moving object. When acquiring multiple images at different acquisition times, the system detects matching feature points in the different acquired images, matches these matching feature points, deforms one of the images so that its composition is approximately the same as the other, calculates the difference in image brightness between the deformed image and the other image, and extracts areas that have changed over time based on the calculated image brightness difference. A method for extracting areas of change over time using background subtraction, characterized by the use of images acquired by a camera mounted on a mobile device.

2. The feature point matching process involves generating multiple sets of different images, selecting an image set with a high feature point matching rate from the multiple image sets, and performing feature point matching on the images in the selected image set. A method for extracting locations that change over time using a background subtraction method with respect to images acquired by a camera mounted on a mobile body according to claim 1.

3. After selecting the image set with a high feature point matching rate, if the selected feature point matching rate is below a predetermined threshold, the camera used to capture the comparison image performs an operation to improve the feature point matching rate, and the comparison image is captured by the camera that has performed the operation. A method for extracting locations that change over time using a background subtraction method with respect to images acquired by a camera mounted on a mobile body according to the feature described in claim 3.

4. Based on the calculation results of the image brightness difference, image processing is performed to binarize each image. A portion of the binarized image is processed using image processing to generate a superimposed image. A method for extracting locations that change over time using a background subtraction method with respect to images acquired by a camera mounted on a mobile body according to claim 1.

5. This is a method for extracting areas of change over time using background subtraction with images acquired by a moving object. When acquiring multiple images at different acquisition times, the system detects matching feature points in the different acquired images, matches the matching feature points to deform one of the images so that its composition matches that of the other image, calculates the difference in image brightness between the deformed image and the other image, generates a mask image by binarizing each image based on the calculated image brightness difference and hiding a portion of the image, performs image processing by overlaying the generated mask image with the deformed image and / or the other image, and extracts areas that have changed over time from the processed image. A method for extracting areas of change over time using background subtraction, characterized by the use of images acquired by a camera mounted on a mobile device.

Images