Single image analysis system

Abstract

It is an object of the present invention to solve the problems of the prior art, i.e. to provide a single image analysis system which can determine the distance from the camera to the object by means of a single photograph only.SOLUTION: The single image analysis system of the present invention is a system for analyzing a single image, and includes relative depth providing means, photographing distance estimation formula setting means, and photographing distance providing means. Here, the single photograph is a single photograph taken by a fixed-point camera installed at one point, and is an image including two or more reference objects whose photographing distances from the fixed-point camera are known. The photographing distance estimation equation setting means is means for setting a photographing distance estimation equation indicating a relationship between a photographing distance and a relative depth based on a photographing distance related to a reference object and a relative depth related to a pixel corresponding to the reference object. The shooting distance assigning means is means for assigning a shooting distance to a pixel based on the shooting distance estimation formula and the relative depth.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to a technology for determining the distance from a camera to a subject, and more specifically to a single-image analysis system that can determine the distance to a subject from a single image (a so-called single photograph) without using a stereo pair of images. [Background technology]

[0002] It is said that two-thirds of Japan's land area is mountainous, and as a result, many homes are built on land backed by slopes, and roads and railways almost always pass along slopes. Slopes are prone to disasters such as collapses and landslides, and Japan has suffered extensive damage from slope collapses and other disasters in the past.

[0003] Measurements are sometimes carried out to monitor the movement of slopes at risk of collapse (including natural slopes and artificial slopes) or slopes showing signs of landslides. For example, on slopes showing signs of landslides, measurements using extensometers and punching plates, measurements using borehole inclinometers, and measurements of ground surface displacement have been carried out. However, measurements using extensometers and punching plates are only effective if they are installed along the entire landslide boundary (especially the head), and borehole extensometers are only effective if they accurately estimate the landslide depth, and installing them in multiple locations is costly.

[0004] Ground surface displacement measurement involves determining the coordinates of numerous observation points installed on a slope and monitoring slope movement by detecting displacement over time, making it possible to directly identify abnormalities, and has the advantage that its effectiveness does not depend on the location of the instrument, as does the case with extensometers and borehole inclinometers.However, installing observation points involves risk, as people must enter the slope, and furthermore, the observation points must be positioned manually using a total station or similar device, which requires a great deal of effort and cost.

[0005] Therefore, various monitoring technologies have been proposed to replace the conventional measurements described above. For example, Patent Document 1 proposes an invention that periodically takes single photographs using a so-called fixed camera fixed at one location, and extracts areas of topographical change by comparing the single photographs taken at two different times. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Publication No. 2019-201275 Summary of the Invention [Problem to be solved by the invention]

[0007] The invention disclosed in Patent Document 1 is a technology that extracts changed areas after removing "temporal noise," which refers to areas that have changed in a short period of time, and "directional noise," which refers to areas that have changed in a physically impossible direction. This makes it possible to remove things that are not actual changes, such as the shaking of vegetation (temporal noise) or the rising of rock masses (directional noise), and to extract more appropriate changed areas.

[0008] On the other hand, in the invention of Patent Document 1, although it is possible to identify the changed area, the degree of the change is grasped in pixel units (for example, five pixels), and it is not possible to grasp the actual distance (i.e., the amount of displacement). Although stereo image analysis could be used to obtain the actual amount of displacement, a camera would need to be permanently installed to constantly observe changes in the target, and permanently installing two cameras would impose operational restrictions, making this method unrealistic.

[0009] To calculate the actual displacement from a single photograph, it is necessary to know the size of the subject represented by one pixel (i.e., the actual size). Naturally, the closer the pixel is to the camera, the smaller the actual size represented by the pixel, and the farther it is from the camera, the larger the actual size. Therefore, to determine the actual size represented by one pixel, it is necessary to know the distance from the camera to the subject. However, no technology has been proposed to date that can calculate the distance from the camera to the subject using only a single photograph.

[0010] The object of the present invention is to solve the conventional problems, that is, to provide a single image analysis system that can determine the distance from the camera to the subject using only a single photograph. [Means for solving the problem]

[0011] The present invention focuses on the fact that it uses a monocular depth estimation model (e.g., MiDaS (registered trademark)) that has been put into practical use in recent years to determine "relative depth," and calculates the distance from the camera to the subject (hereinafter referred to as "shooting distance") based on this relative depth, and is an invention based on an idea that has not been seen before.

[0012] The single image analysis system of the present invention is a system for analyzing a "single image" and includes a relative depth assignment means, a shooting distance estimation formula setting means, and a shooting distance assignment means. Here, a single image is a single image taken by a fixed camera installed at a single location, and includes two or more "reference objects" whose shooting distances from the fixed camera are known. The relative depth assignment means assigns a "relative depth" to each pixel constituting the single image using a trained model generated by machine learning. The shooting distance estimation formula setting means sets a "shooting distance estimation formula" that indicates the relationship between shooting distance and relative depth based on the shooting distances of two or more reference objects and the relative depths of the pixels corresponding to the reference objects. The shooting distance assignment means assigns a shooting distance to each pixel based on the shooting distance estimation formula and the relative depth of the pixel. The relative depth is a relative value that represents the magnitude of the shooting distance in a single image.

[0013] The single image analysis system of the present invention may further include a pixel actual size assigning means. This pixel actual size assigning means assigns pixel actual size (the size of the object corresponding to the pixel) to each pixel based on a "pixel actual size estimation formula" and the shooting distance associated with the pixel. Here, the pixel actual size estimation formula is a mathematical formula that shows the relationship between pixel actual size and shooting distance. The pixel actual size and shooting distance associated with a pixel can be determined from the specifications of a fixed camera, or can be empirically determined from the dimensions of a single image captured in advance by a fixed camera and the actual object.

[0014] The single image analysis system of the present invention may further include a fixed camera that periodically (or intermittently) photographs an object, an image movement distance calculation means, and an actual displacement amount calculation means. The image movement distance calculation means performs pattern matching between a "first single image (a single image acquired by the fixed camera at a first time period)" and a "second single image (a single image acquired by the fixed camera at a second time period)" to detect a "partially common image" that is common to both images, and calculates the "movement distance" of the partially common image between the two time periods. The actual displacement amount calculation means calculates the "actual displacement amount" of the partially common image based on the movement distance of the partially common image and the actual pixel size. The movement distance is a distance set based on the pixel size (pixel size on the image). [Effects of the Invention]

[0015] The single image analysis system of the present invention has the following advantages. (1) By continuously capturing and monitoring images using a fixed camera, it is possible to realize real-time and constant monitoring of the object, and also to grasp the shooting distance to the subject. (2) Furthermore, since the actual amount of displacement can be grasped, the observer can intuitively grasp the degree of change in the topography, etc. (3) Since it is sufficient to install one fixed camera, the costs of equipment and installation can be reduced compared to conventional methods. [Brief explanation of the drawings]

[0016] [Figure 1] A model diagram to explain "shooting distance." [Figure 2] A model diagram to explain "pixel size" and "actual pixel size." [Figure 3] 1 is a block diagram showing the main configuration of a single image analysis system according to the present invention. [Figure 4] 10 is a graph showing a pixel actual size estimation formula obtained based on coordinate points consisting of pixel actual size and shooting distance. [Figure 5] FIG. 10 is a graph showing a shooting distance estimation formula obtained based on coordinate points formed by relative depth and shooting distance. [Figure 6] (a) is a model diagram that schematically shows the "partial common image" that is common to the first image and the second image, and (b) is a model diagram that schematically shows the "movement distance" obtained based on the position of the partial common image of the first image and the partial common image of the second image. [Figure 7] FIG. 1 is a flowchart showing the main processing flow of the single image analysis system of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of the single image analysis system of the present invention will be described with reference to the drawings. Although the single image analysis system of the present invention can be implemented with a variety of objects as the object, for convenience, the following description will be given assuming that the object is a slope (such as a natural slope or a slope face).

[0018] 1.Definition Before describing the embodiments of the present invention, definitions of terms used herein will be provided.

[0019] (Shooting distance and single image) FIG. 1 is a model diagram illustrating the "shooting distance." The single-image analysis system of the present invention analyzes a single photograph of an object OJ taken by a fixed camera 108 (described later), and one of its technical features is the use of the distance from the fixed camera 108 to the object OJ. For convenience, the distance from the fixed camera 108 to the object OJ in real space is referred to as the "shooting distance." The present invention also utilizes a single photograph containing a reference object RO whose shooting distance is known. For convenience, the shooting distance related to the reference object RO is specifically referred to as the "reference distance," and a single photograph containing two or more reference objects RO is referred to as a "single image." Note that FIG. 1 illustrates a QR marker with known actual dimensions as an example of the reference object RO. However, various objects with known actual dimensions, such as pipes and stakes, can also be used as the reference object RO.

[0020] (Pixel size and actual pixel size) FIG. 2 is a model diagram for explaining "pixel size" and "actual pixel size." All pixels constituting a single image have the same size (e.g., side length). In contrast, the dimensions of the object contained in a pixel vary from pixel to pixel. Specifically, the closer an object is to the fixed camera 108, the smaller the dimensions represented by the pixel, and the farther an object is from the fixed camera 108, the larger the dimensions. For convenience, as shown in FIG. 2, the dimensions of the pixels constituting a single image will be referred to as "pixel size," and the dimensions of the object contained in the pixel will be referred to as "actual pixel size." In other words, the pixel size is the same for all pixels, but the actual pixel size varies from pixel to pixel. For example, a pixel corresponding to a nearby object OJ will have a small actual pixel size, and a pixel corresponding to a distant object OJ will have a large actual pixel size.

[0021] 2. Single image analysis system Next, a detailed description will be given of the single image analysis system 100 of the present invention. Figure 3 is a block diagram showing the main components of the single image analysis system 100 of the present invention. As shown in this figure, the single image analysis system 100 of the present invention is configured to include a relative depth assigning means 101, a shooting distance estimation formula setting means 102, and a shooting distance assigning means 103, and can also be configured to include a pixel actual size estimation formula setting means 104, a pixel actual size assigning means 105, an image movement distance calculation means 106, an actual displacement amount calculation means 107, a fixed camera 108, a single image storage means 109, and a pixel actual size estimation formula storage means 110.

[0022] Of the components of the single-image analysis system 100, the relative depth assigning means 101, shooting distance estimation formula setting means 102, shooting distance assigning means 103, pixel actual size estimation formula setting means 104, pixel actual size assigning means 105, image movement distance calculation means 106, and actual displacement amount calculation means 107 can be manufactured as dedicated components or can be made using general-purpose computers. That is, the computer executes arithmetic processing according to a predetermined program, thereby performing processing specific to each component. The computer can be configured as a personal computer (PC), a tablet PC such as an iPad (registered trademark), a mobile device such as a smartphone, or a personal data assistant (PDA). The computer includes a processor such as a CPU, memory such as ROM and RAM, and some also include input means such as a mouse and keyboard, and a display.

[0023] The single image storage means 109 and pixel actual size estimation formula storage means 110 can be implemented as a storage device of a general-purpose computer (for example, a personal computer), or can be implemented as a database server. When implemented as a database server, the database server can be placed on a local network (LAN: Local Area Network), or can be implemented as a cloud server that stores data via the Internet.

[0024] Below, each of the main elements that make up the single image analysis system 100 of the present invention will be described in detail.

[0025] (Fixed camera and single image storage means) The fixed camera 108 constituting the single image analysis system 100 can be a conventional camera, such as a trail camera for photographing wild animals. This fixed camera 108 is preferably one that takes pictures automatically without human operation, and can be set to take pictures periodically (or irregularly or intermittently), for example, every hour or 30 minutes. The fixed camera 108 is installed in a position (one point) that allows it to take pictures of the slope, which is the object OJ, from a bird's-eye view, and is fixed so that it takes pictures in the same direction from the same position, so that it basically acquires single images of the same area of ​​the slope. Note that if automatic photography is to be performed over a relatively long period of time (several months to a year), a sufficient battery is provided.

[0026] Since the fixed camera 108 takes pictures periodically over a long period of time, the number of acquired single images will be enormous. The single image storage means 109 stores these single images, and as described above, it can be placed in the vicinity of the fixed camera 108 and connected to the fixed camera 108 via a local network (image data communication), or it can be a cloud server that stores images via the Internet (wireless communication in this case).

[0027] (Means for setting the pixel actual size estimation formula) The pixel actual size estimation formula setting means 104 is a means for setting a mathematical formula (hereinafter referred to as the "pixel actual size estimation formula") that indicates the relationship between "pixel actual size" and "shooting distance." This pixel actual size estimation formula can be set based on the relationship between pixel actual size and shooting distance, which is determined by performing calculations using the specifications of the fixed camera 108 (such as focal length and angle of view). Alternatively, a pixel actual size estimation formula can be set based on the actual pixel actual size and shooting distance using an actually acquired single photograph. Specifically, as shown in FIG. 4, coordinate points formed by combinations of pixel actual size and shooting distance are plotted, and a regression curve or regression line determined based on these multiple coordinate points is set as the pixel actual size estimation formula. The pixel actual size estimation formula thus obtained is stored in pixel actual size estimation formula storage means 110 (FIG. 3).

[0028] (Relative depth imparting means) The relative depth assigning means 101 assigns a "relative depth" to each pixel constituting a single image. Here, relative depth is a value representing the degree of perspective (depth) from the fixed camera 108 to the object OJ, and is a relative value within a single image. In recent years, monocular depth estimation models such as MiDaS (registered trademark) have been put into practical use, making it possible to calculate relative depth for each pixel even in a single photograph. Note that monocular depth estimation models use "trained models" generated by machine learning such as Deep Learning. The relative depth assigning means 101 constituting the single image analysis system 100 can also use a monocular depth estimation model such as MiDaS. The relative depth assigning means 101 can also generate an image representing relative depth (hereinafter referred to as a "relative depth image") by assigning color information (e.g., RGB or grayscale) corresponding to the relative depth to each pixel.

[0029] (Shooting distance estimation formula setting means) The shooting distance estimation formula setting means 102 is a means for setting a mathematical formula (hereinafter referred to as the "shooting distance estimation formula") that indicates the relationship between "relative depth" and "shooting distance." This shooting distance estimation formula is set using an actually acquired single photograph and the relative depth assigned by the relative depth assigning means 101, and more specifically, is set based on the "reference distances (FIG. 1)" associated with two or more reference objects RO included in the single image and the relative depths associated with the pixels corresponding to the reference objects RO. Specifically, as shown in FIG. 5, coordinate points formed by combinations of relative depths and reference distances (shooting distances) are plotted, and a regression curve or regression line obtained based on these multiple coordinate points is set as the shooting distance estimation formula.

[0030] (Shooting distance providing means) The shooting distance assigning means 103 assigns a shooting distance to each pixel based on the "shooting distance estimation formula" and "relative depth." As described above, the shooting distance estimation formula indicates the relationship between relative depth and shooting distance, and the relative depth can be used as an explanatory variable and the shooting distance as a target variable. In other words, the shooting distance can be determined by inputting the relative depth into the shooting distance estimation formula. The shooting distance assigning means 103 assigns a shooting distance to every pixel. The shooting distance assigning means 103 can also generate an image representing the shooting distance (hereinafter referred to as a "shooting distance image") by assigning color information according to the shooting distance to each pixel.

[0031] (Means for providing actual pixel size) The actual pixel size assigning means 105 assigns actual pixel sizes to pixels based on the "actual pixel size estimation formula" and "shooting distance." As described above, the actual pixel size estimation formula indicates the relationship between the shooting distance and the actual pixel size, and the shooting distance can be used as an explanatory variable and the actual pixel size as a target variable. In other words, the actual pixel size can be determined by inputting the shooting distance into the actual pixel size estimation formula. The shooting distance assigning means 103 can be configured to assign the calculated actual pixel size to all pixels, or to assign actual pixel sizes to pixels that make up a "partial common image," which will be described later.

[0032] (Image movement distance calculation means) The image movement distance calculation means 106 is a means for comparing single images acquired at two different times and calculating the distance (hereinafter referred to as "movement distance") that a partial area common to both images (hereinafter referred to as "partial common image") has moved. The process by which the image movement distance calculation means 106 calculates the movement distance will be described below with reference to FIG. 6. The single image acquired at the first time period will be referred to as the "first single image," and the single image acquired at the second time period will be referred to as the "second single image."

[0033] First, the image movement distance calculation means 106 compares the first single image with the second single image and detects the partial common image by performing pattern matching. For example, in FIG. 6(a), pattern matching using the first single image and the second single image detects a convex partial common image area consisting of seven pixels. Furthermore, the position of the partial common image in the first single image is different from the position of the partial common image in the second single image, which means that it can be inferred that the partial common image has moved. Therefore, the image movement distance calculation means 106 aligns the planar positions of both images, overlays the first single image with the second single image, and extracts a vector of apparent movement of the partial common image (hereinafter referred to as a "movement vector"). For example, in FIG. 6(b), a movement vector is extracted that indicates that the partial common image has moved one pixel to the right and three pixels downward. The magnitude (length) of this movement vector is then set as the "movement distance." In other words, the movement distance calculated by the image movement distance calculation means 106 is a magnitude determined solely by the pixel size, not a size (distance) in real space.

[0034] The image movement distance calculation means 106 can be configured to calculate the movement distance for all detected partial common images, or to calculate the movement distance for a specific partial common image. For example, a size threshold based on pixel size (hereinafter referred to as "movement threshold") can be set in advance, and when the movement distance of a partial common image exceeds (or becomes equal to or greater than) the movement threshold, the movement distance for that partial common image can be calculated.

[0035] (Actual displacement calculation means) The actual displacement amount calculation means 107 is a means for calculating the displacement amount of the partial common image in real space (hereinafter simply referred to as "actual displacement amount") based on the movement distance and actual pixel size of the partial common image. Specifically, the actual displacement amount can be calculated by multiplying the movement distance by the actual pixel size (actual displacement amount = movement distance × actual pixel size). At this time, when the partial common image is composed of a plurality of pixels (seven pixels in FIG. 6), it is preferable to use an actual pixel size that represents a value obtained by statistically processing the actual pixel sizes of the plurality of pixels (such as an average, median, or mode) and then multiply the movement distance by this actual pixel size. Furthermore, the actual pixel size used here can be the actual pixel size assigned based on the first single image, the actual pixel size assigned based on the second single image, or a statistical value based on the actual pixel size of the first single image and the actual pixel size of the second single image.

[0036] (Processing flow) The main processing of the single image analysis system 100 of the present invention will be described in detail below with reference to Figure 7. Figure 7 is a flow diagram showing an example of the flow of the main processing of the single image analysis system 100 of the present invention, with the central column showing the processing to be performed, the left column showing the input information required for that processing, and the right column showing the output information resulting from that processing.

[0037] The pixel actual size estimation formula is set in advance using pixel actual size estimation formula setting means 104 (Step 201 in FIG. 7). As described above, the pixel actual size estimation formula is set after determining the relationship between pixel actual size and shooting distance based on the specifications of fixed camera 108 or the relationship between pixel actual size and shooting distance based on an actually acquired single photograph. The pixel actual size estimation formula obtained here is stored in pixel actual size estimation formula storage means 110.

[0038] Once the pixel actual size estimation formula is set, a fixed camera 108 is set on-site and begins taking images using the fixed camera 108. When the fixed camera 108 acquires single images periodically (or irregularly or intermittently), for example, at intervals of one hour or 30 minutes (Step 202 in FIG. 7), the single images are stored in single image storage means 109.

[0039] When the single images are obtained, the first single image acquired in the first period and the second single image acquired in the second period are selected, and a relative depth is assigned to each pixel using the relative depth assigning means 101 (Step 203 in FIG. 7), and a relative depth image is generated. At this time, the relative depth can be assigned to only one of the single images (for example, the second single image), or the relative depth can be assigned to both single images.

[0040] Once a relative depth has been assigned to each pixel, the shooting distance estimation equation is set using the shooting distance estimation equation setting means 102 (Step 204 in FIG. 7). As described above, the shooting distance estimation equation is set based on the "reference distances" associated with two or more reference objects RO included in the single image and the relative depths associated with the pixels corresponding to the reference objects RO. It is preferable to set the shooting distance estimation equation using either the first or second single image.

[0041] Once the shooting distance estimation formula is set, the shooting distance is assigned to each pixel using shooting distance assigning means 103 (Step 205 in FIG. 7). Specifically, the shooting distance is calculated by inputting the relative depth into the shooting distance estimation formula, and this is assigned to each pixel to generate a shooting distance image. At this time, the shooting distance can be assigned to only one of the single images (for example, the second single image), or it can be assigned to both single images.

[0042] Once the shooting distance is assigned to each pixel, the image movement distance calculation means 106 is used to detect partial common images that are common to the first and second single images (Step 206 in FIG. 7), and calculate the movement distance by which the partial common image has moved (Step 207 in FIG. 7). At this time, as described above, it is possible to calculate the movement distance for all detected partial common images, or to calculate the movement distance for a specific partial common image (a partial common image whose movement distance exceeds the movement threshold).

[0043] Once the movement distance of the partial common image is obtained, the pixel is assigned the shooting distance using the pixel actual size assigning means 105 (Step 208 in FIG. 7). Specifically, the pixel actual size is calculated by inputting the shooting distance into a pixel actual size estimation formula, and this is assigned to the image. At this time, it is preferable to assign the pixel actual size to a single image to which the shooting distance has been assigned. As described above, the pixel actual size can be assigned to all pixels, or it can be assigned to pixels that make up the partial common image.

[0044] When the movement distance and actual pixel size of the partial common image are obtained, the actual movement amount of the partial common image is calculated using the actual movement amount calculation means 107 (Step 209 in FIG. 7). Specifically, the actual movement amount is calculated by multiplying the movement distance by the actual pixel size. [Industrial Applicability]

[0045] The single-image analysis system of the present invention can be used to determine changes in natural slopes, cut slopes, and embankment slopes, as well as concrete structures such as concrete dams, reclaimed land, and soft ground. [Explanation of symbols]

[0046] 100 Single image analysis system of the present invention 101 Relative depth assignment means (for single image analysis systems) 102 (Single image analysis system) shooting distance estimation formula setting means 103 (Single image analysis system) shooting distance assignment means 104 (Single Image Analysis System) Pixel Actual Size Estimation Formula Setting Method 105 (Single image analysis system) pixel actual size assignment means 106 (Single image analysis system) Image movement distance calculation means 107 (Single image analysis system) Actual displacement calculation means 108 Fixed camera (single image analysis system) 109 Single image storage means (for single image analysis systems) 110 (Single image analysis system) pixel actual size estimation formula storage means OJ Object RO Reference Material

Claims

1. A system for analyzing a single image captured by a fixed camera installed at one location, the image including two or more reference objects whose shooting distances from the fixed camera are known, comprising: a relative depth assigning means for assigning a relative depth to each pixel constituting the single image using a trained model generated by machine learning; a shooting distance estimation equation setting means for setting a shooting distance estimation equation that indicates the relationship between the shooting distance and the relative depth based on the shooting distances related to two or more of the reference objects and the relative depths related to the pixels corresponding to the reference objects; a shooting distance assigning unit that assigns the shooting distance to each pixel based on the shooting distance estimation formula and the relative depth associated with the pixel, The relative depth is a relative value representing the magnitude of the shooting distance in the single image. A single image analysis system characterized by:

2. further comprising a pixel actual size assigning means for assigning a pixel actual size to each pixel based on the pixel actual size estimation formula and the shooting distance associated with the pixel; the pixel actual size is the size of an object corresponding to the pixel, the pixel actual size estimation formula is a mathematical formula that indicates the relationship between the pixel actual size and the shooting distance, The actual pixel size and the shooting distance of the pixel are calculated based on the specifications of the fixed camera, or are obtained in advance from the single image captured by the fixed camera and the dimensions of an actual object.

2. The single image analysis system according to claim 1.

3. the fixed camera that periodically or intermittently photographs the object; an image movement distance calculation means for performing pattern matching between a first single image, which is the single image acquired by the fixed camera at a first time period, and a second single image, which is the single image acquired by the fixed camera at a second time period, to detect a partial common image that is common to both images, and for calculating a movement distance that the partial common image has moved between the two time periods; an actual displacement amount calculation means for calculating an actual displacement amount of the partial common image based on the movement distance related to the partial common image and the actual pixel size, The movement distance is a distance set based on the size of the pixel.

3. The single image analysis system according to claim 2.

Images